So you’re a busy backup administrator and you’re getting ready to go on leave. It’s 4pm on your final day before the holiday, you’ve finally got everything off your plate, and you think to yourself, “Now I’ve finally got the time, I’ll just quickly upgrade NetWorker before I leave.”
This unfortunately is an alternative of that Friday change rule violation known as POETS.
There’s three distinctly wrong things with this scenario:
- Infrastructure upgrade done without change control.
- Infrastructure upgrade done at the last minute.
- Infrastructure upgrade done without follow-up monitoring.
Any one of those scenarios is enough to cause a nightmare situation – either for yourself, getting call-outs when you’re meant to be on holidays, or for your colleagues, left in the lurch after you switch your phone off for two weeks and go on a holiday to the East Islands.
All three though? That’s just asking for trouble.
(This lesson doesn’t actually just apply to NetWorker – it applies across the board for system, application and storage administration. Don’t modify the system just before going away for a while.)
Just before this holiday season, I had a customer upgrade* their NetWorker server from 7.3.x to 7.5 before going on leave. Not 7.5.1, not 7.5.1.8, 7.5. This didn’t go so well, and a few days later when the fill-in administrators noticed the issue**, there was a bit of work to rectify the various issues and some backups during that time didn’t work.
This however is by no means unique. Following Twitter I noticed one on-call person suffer a hideous xmas day and following day working on a call-out from what appeared to be an untested change done by someone else before that other person went on holiday.
And non-betting man that I am, I’d bet a considerable wad of money (and win) that this fellow’s experience wasn’t unique for IT workers over xmas 2009.
In short: choosing to do an untested/uncontrolled upgrade just before going on holidays can be either self-destructive or selfish (or even both) – it may lose your your holiday, depending on the level of the fail and the backup (or lack thereof) within your company, or it may cause a colleague to have an insufferably unpleasant time. (Alternatively, if you can be reached, it may result in you having a bad time on your holiday in order to help out a colleague having a bad time as well.)
The problem with rushing through upgrades at the last minute is that they tend to be poorly done, even if they seem simple enough. Even if change control is being followed, if that change control has been rushed through (as it can sometimes be done as a “last minute” activity), then it provides no guarantee that the change will work smoothly. And don’t forget: Murphy’s Law works in the datacentre as well. Something that looks easy, that you should be able to do with your eyes closed, when done as a rush job at the last minute can come unstuck quite easily.
So please – for your sake, for your colleagues sake, for NetWorkers’ sake and for the sake of your company: please don’t upgrade just before you go on holidays.
—
* upgrade = “update” in NetWorker speak
** Which should serve as a reminder that you should never only have one backup administrator.
I have a variation on that. Purely unintentional of course.
I upgraded a global organisation’s 3rd and 4th biggest networker zones from 7.2.2 to 7.4.3, it was under change control, the upgrade plan had a heap of checks built in before committing to the new version, they were almost the last two 7.2 series zones left out of more than 10 that had been upgraded carefully over the previous 18 months. There was of course an expectation that further tuning would be needed over the next week, and a reasonable certainty that some new buglet special to these zones would pop up. I was ready for this, but on my own basically as no one else in the organisation knew all that much about networker. I had all sorts of monitoring switched on, heaps of info flowing into my email box, not going on holidays for at least 3 more weeks… I was ready for anything.
The very next day I was retrenched (as part of a departmental decimation) no notice, no handover to anyone, out the door at 10am.
I didn’t have a plan for that eventuality.
Hi Peter,
I’m sorry to hear about how your upgrade process was broken – certainly not your responsibility in that case, that’s for sure – and you obviously had planned the upgrade exactly how an upgrade should be planned.
Cheers,
Preston.