Should POETS day be special?

 Aside, Backup theory  5 Responses »
Sep 162011
 

You all know about POETS day, don’t you? It’s a great acronym:

P-ss Off Early, Tomorrow’s Saturday

It’s a pretty good summation of a lot of the IT industry – we’re reluctant to kick off major changes on a Friday because … well, the weekend follows, and if something goes wrong, it could be disruptive.

But a day or so ago, Matt Stace (@matstace) tweeted:

If it’s not good enough to deploy on a Friday, what makes it good enough to deploy any other day of the week?

Some might think this is a little trite, but there’s actually good wisdom in Mat’s comment – if we lack the confidence that something we’re working on can be deployed safely on Friday, why should we be any more confident that it can be deployed safely at another time? In fact, when you stop and think about it, in the light of cold logic, there’s only two explanations:

  1. You aren’t sufficiently certain that what you’re going to deploy is ready, or
  2. You’re superstitious.

Now, I’m as willing as the next person to claim that Murphy’s Law takes a perverse delight in visiting computer rooms, but realistically, that’s just a tendency to catastrophise* things when they come up unexpectedly.

So, if you’re sitting back and saying that you or the company should hold off doing something on a Friday because, well, it’s Friday, it’s time to sit back and ask yourself – is it because you’re being superstitious, or is it because it’s just simply not ready to be done, regardless of what day it is?

I know I will be.

–
* Thanks to my good friend Christopher Banks (aka @bipolarbearnz) for introducing me to that word last night. I’ll be using it daily for months, I think.

 Posted by Preston de Guise at 16:35  Tagged with: , , ,

Going on holiday? Don’t upgrade first

 General Technology, General thoughts, NetWorker, Policies  2 Responses »
Dec 282009
 

So you’re a busy backup administrator and you’re getting ready to go on leave. It’s 4pm on your final day before the holiday, you’ve finally got everything off your plate, and you think to yourself, “Now I’ve finally got the time, I’ll just quickly upgrade NetWorker before I leave.”

This unfortunately is an alternative of that Friday change rule violation known as POETS.

There’s three distinctly wrong things with this scenario:

  • Infrastructure upgrade done without change control.
  • Infrastructure upgrade done at the last minute.
  • Infrastructure upgrade done without follow-up monitoring.

Any one of those scenarios is enough to cause a nightmare situation – either for yourself, getting call-outs when you’re meant to be on holidays, or for your colleagues, left in the lurch after you switch your phone off for two weeks and go on a holiday to the East Islands.

All three though? That’s just asking for trouble.

(This lesson doesn’t actually just apply to NetWorker – it applies across the board for system, application and storage administration. Don’t modify the system just before going away for a while.)

Just before this holiday season, I had a customer upgrade* their NetWorker server from 7.3.x to 7.5 before going on leave. Not 7.5.1, not 7.5.1.8, 7.5. This didn’t go so well, and a few days later when the fill-in administrators noticed the issue**, there was a bit of work to rectify the various issues and some backups during that time didn’t work.

This however is by no means unique. Following Twitter I noticed one on-call person suffer a hideous xmas day and following day working on a call-out from what appeared to be an untested change done by someone else before that other person went on holiday.

And non-betting man that I am, I’d bet a considerable wad of money (and win) that this fellow’s experience wasn’t unique for IT workers over xmas 2009.

In short: choosing to do an untested/uncontrolled upgrade just before going on holidays can be either self-destructive or selfish (or even both) – it may lose your your holiday, depending on the level of the fail and the backup (or lack thereof) within your company, or it may cause a colleague to have an insufferably unpleasant time. (Alternatively, if you can be reached, it may result in you having a bad time on your holiday in order to help out a colleague having a bad time as well.)

The problem with rushing through upgrades at the last minute is that they tend to be poorly done, even if they seem simple enough. Even if change control is being followed, if that change control has been rushed through (as it can sometimes be done as a “last minute” activity), then it provides no guarantee that the change will work smoothly. And don’t forget: Murphy’s Law works in the datacentre as well. Something that looks easy, that you should be able to do with your eyes closed, when done as a rush job at the last minute can come unstuck quite easily.

So please – for your sake, for your colleagues sake, for NetWorkers’ sake and for the sake of your company: please don’t upgrade just before you go on holidays.

–
* upgrade = “update” in NetWorker speak

** Which should serve as a reminder that you should never only have one backup administrator.

 Posted by Preston de Guise at 18:58  Tagged with: , , , , , ,
© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha