Who, Me? Greetings once again, gentle reader, and welcome to another instalment of Who, Me? in which Reg readers like yourselves soften the start of the work week with reminders that we all sometimes make mistakes.

This week's mere mortal is someone we'll Regomize as "Christopher" who worked, back in the halcyon days of the 1990s, as a "gopher/fix-it" type guy at a stockbroking firm - the kind of place that operated basically 24 hours a day whenever a financial market was open somewhere.

Part of Christopher's task was to archive each day's trades and ensure backups were made onto Digital Audio Tape (DAT) - a technology we're sure few remember fondly. It's important to note that both the trading software and the backup software were customized, Christopher was a vital cog in the wheel, and very diligent in his duties.

You may also recall that towards the end of the 1990s there was this whole big Y2K thing going around that prompted a lot of orgs to update their systems lest disaster strike at the dawn of the millennium. Christopher's employer was one such and took the opportunity to update its hardware and software. It also switched from the DAT backups to "a Commvault setup with a new network attached tape library using DLT tapes" according to Christopher.

Longish story shortish, almost everything was changing. That "almost" matters, so remember it.

Complicating matters, the upgrade had to be done without losing trading time. That meant setting up the new server in advance, getting the software vendor to log in and transfer the system over, then finding a brief window after the old server had finished its archiving run to test the new server before trading started.

The other trick was that the existing workstations were staying in place - thus the "almost." To avoid delays connecting workstations to the new server, the shiny box was set up as a clone of the old one. Even the server’s name and IP address were replicated.

Thus, the two servers could obviously not both be running on the network at the same time.

The server cloning scheme made it possible to cutover to the new infrastructure in the very short window between trading days. But it didn't allow enough time for the masses of historical data stored on the RAID within the old server to be transferred to the new one.

And because the two servers could not coexist on the network, moving the data would be accomplished with a portable USB hard drive. You'll no doubt recall that at this time neither the capacities nor the speeds of such devices were impressive.

By the time the new server was up and running and tested, Christopher had been working 20 hours straight, and was understandably tired. He decided to get a few hours' sleep before any of the stockbrokers started calling in with support questions (as was inevitable given the new system).

The software vendor assured him that the copy of the historical data, while important, was essentially a direct copy process and not urgent. It could be done any time. So Christopher put it on his to-do list.

Oops.

Two weeks passed, and some bright spark in IT decided that that old server that had been sitting around doing nothing ought to be put to some good use. It was therefore reformatted, wiped clean and repurposed elsewhere in the organization.

Had Christopher got around to copying the data from it to the new server? You know the answer.

But what about those DATs, we hear you ask? They were archiving the data from each day - much of what was in the RAID existed nowhere else.

That's another oops right there.

It all would have been just a lesson learned the hard way except that one of the brokers turned out to have been doing some questionable things, and law enforcement requested a look at some historic data on trades in which they'd been involved. Data that no longer existed, because Christopher didn't get around to it.

As it transpired there was sufficient other information available on said stockbroker's naughtiness that Christopher's oversight did not mean an escape from justice. Sometimes you just can't catch a break.

If you've been thinking about maybe sending a story of your own tech misadventures to Who, Me? one of these days, click here to do it right now - don't put it off any longer! We need your anecdotes as the Who, Me mailbox has seen better days. ®

https://www.theregister.com//2024/07/29/who_me/