who, me? Welcome denizens of The Reg to another Monday morn, which means an instalment of Who, Me? - the column in which readers share tales of times their undoubted technical prowess fell just a little short.

This week our hero is someone we'll Regomize as "Paton" who, several decades ago, worked for a prominent ISP in South Africa. Indeed it was a very prominent ISP, as its DNS servers were authoritative for thousands of domains including country code top-level domains (ccTLDs) in South Africa and elsewhere in the region. This will be important later on.

Paton worked as a "backbone engineer" which, particularly in the context of such a crucial ISP, sounds like a seriously important job. Not the sort of thing one does in a hurry, or while distracted. This, too, will be important later on.

One of Paton’s responsibilities was maintaining Access Control Lists (ACLs) specifying which users and domains had access to certain network resources. The ACLs in Paton's organization were extensive and complex, as you might imagine. They controlled not only the netblocks for the ISP's customers, but also for its internal infrastructure - including those authoritative DNS servers.

One fine afternoon, Paton was handed the task of updating the netblocks and propagating them via BGP to peers and transit providers on the network. This meant editing the ACLs with the new information - not a difficult task.

However, he was under pressure from a group of his colleagues, who were keen for him to join them outside for a bit of fresh air and a cigarette (somewhat negating the benefit of the fresh air). So Paton hurried - which, as noted above, he should not have done - and got the job done quickly so he could go enjoy that rich tobacco flavor.

When he got back, not very long afterwards, he found what he described as "pandemonium and chaos." The Network Operations Center was being flooded with calls from irate customers as a result of what was, to that date, the largest internet outage to hit the African continent.

To make matters worse, someone pretending to be a hacker contacted a local tech news service and claimed that it was their l33t skillz that had caused the outage. The news outlet ran with it, meaning Paton's bosses had both the technical problem and a potential security headache to deal with - on a multi-country scale.

It transpired, of course, that there was no hack. The ISP's security had not been breached. But that was hardly of comfort from a PR perspective, given the actual cause.

In adding to the ACLs in his haste, Paton had inadvertently replaced them. All of them. So the new netblocks were fine, but none of the other complex and interlocking control lists directing internet traffic for much of sub-Saharan Africa were in place.

Paton of course had to restore the ACLs, as well as updating the netblocks. Then, for his sins, he got to write the organization's very first change-management protocol. No doubt it included a bit about checking that you haven't wiped half a continent off the internet before you get a smoke.

OK, not everyone's little mistakes end up blowing up to nationwide scale. But if you've ever done one little thing wrong and it ended up being a really big thing, we'd like to hear about it. Click here to send an email to Who, Me? and we may share your story on some other manic Monday. ®

https://www.theregister.com//2024/08/12/who_me/