How To Look Stupid (Part #612)
December 22, 2020 4 Comments
Now is the winter of our discontent. But rather than dwell on what a terrible year 2020 has been, I thought I’d make my final post of the year something more positive… so I am going to look back on one of the (many) times I made a fool of myself, in the hope that 2021 will give me the chance to do so again.
When Computers Go Bad
In the late 1990s, I was fresh out of university and working in my first job, for a small company (5 people!) at London’s Heathrow Airport, as a developer and database admin. We provided cargo handling software for all of the big airlines and freight companies. And on this particular day, “Dave”* at Air Canada had a problem with his system.
My company’s software managed the customs clearance of all inbound air freight for most of the airport. In order for inbound freight to leave the secure warehouses on a truck, this software (which, for Air Canada, ran on their main HPUX server) would send a message to the central HM Customs computer and then, upon receiving clearance, print out an official “air waybill” document. The waybill was legal proof that goods had clearance to leave the warehouse: no waybill = no clearance = no freight.
An hour ago, Dave had called in with a major problem: goods were being cleared by customs, but no waybills were bring printed. Air Canada now had a queue of lorries backed up at the warehouse and a crew that couldn’t do any work. There was nothing wrong with the printers, it was our software. Fix it, Dave begged us. Fix it now!
When DBAs Go Rogue
A senior colleague of mine, Denis**, was working on the problem and trying to test a fix on our lab system. He was also dialled in to Air Canada’s production system, on which our software ran – a crucial fact which turned out to be very important.
So when he called through to me from the server room to say, “Hey could you reboot the lab box?” I wondered over to his desktop and typed the magic reboot command on the first root window I found. Hey, one terminal session looks like another, right?
“Are you going to reboot it?” called Denis.
“I already have,” I yelled back, mildly irritated.
Denis stuck his head out of the door and stared at me, puzzled. I was then able to watch a whole range of emotions pass over his face: confusion changed to comprehension which in turn became outright horror.
I had just hard rebooted Air Canada’s entire UNIX platform with no warning to them at all.
Knowing When To Own Up
It took them a little while for Air Canada to realise what (or who) had happened to them. Remember, this was the 1990s, so big iron UNIX systems took about 15-30 mins to restart – and everybody was connected via dumb terminals which would have just suddenly gone blank.
Fred was a DBA until he accidentally truncated the wrong table
I mainly spent this time in purgatory, thinking about alternative careers, planning my new life in a Tibetan monastery or hoping for a natural disaster to divert attention.
But eventually, my desk phone rang and our receptionist said, “Dave from Air Canada wants to speak to you”.
I can vividly remember the dry mouth, my sweaty palms holding the phone, my voice about three octaves too high.
“Yes?” I stammered.
“I don’t know what you’ve done,” said Dave, “but all the waybills are coming out again now. Thanks very much!”
It’s important, I think, to be honest in these situations. But not that honest. So I let Dave get back to his busy job and made a mental note to confess to what had really happened some time within the next 25 years. And then I filed that next to the other mental note – the one about never, ever typing reboot without triple checking which system you are connected to.
Aspirations for 2021
When I look back at this story – and the many other times in my career when I made myself look stupid – I am grateful for the fact that things turned out ok. 
The whole year 2020 has felt like an elongated version of the purgatory I experienced above. But, as anybody who has ever rebooted a 1990’s-era big iron UNIX server will attest, the login window only appears about ten seconds after you’ve finally admitted to yourself that it’s never coming back.
So let’s hope that 2021, like Dave and his waybill printouts, gets us back on track fast.
* The names of innocent parties have been changed to protect their identities
** Denis really was called Denis though
flashdba 
Thanks for that! A bit of light relief from 2020. I wonder if there is a correlation between “experiencing” a potentially career-limiting screw up in the first two years of your working life in I.T. and general success. Once we are all allowed to meet up in person, I’ll let you buy me one of those salesmen lunches and tell you about how I destroyed several hundred thousand hospital patient records…
This is a deal. Although you’ll also need to commit to a minim order of at least three million dollars
while working for oracle I had a customer swear that the db crashed on its own, despite nothing in any log indicating such was the case. It became clear that someone had fat fingered a kill -9 $pmon_pid but pc dictated that I could never say that outright. Spent the better part of 3 wks trying to psycho-analyse customer to admit what happened. Have a great holiday and new year.
And to you!