You all know about POETS day, don’t you? It’s a great acronym:
P-ss Off Early, Tomorrow’s Saturday
It’s a pretty good summation of a lot of the IT industry – we’re reluctant to kick off major changes on a Friday because … well, the weekend follows, and if something goes wrong, it could be disruptive.
But a day or so ago, Matt Stace (@matstace) tweeted:
If it’s not good enough to deploy on a Friday, what makes it good enough to deploy any other day of the week?
Some might think this is a little trite, but there’s actually good wisdom in Mat’s comment – if we lack the confidence that something we’re working on can be deployed safely on Friday, why should we be any more confident that it can be deployed safely at another time? In fact, when you stop and think about it, in the light of cold logic, there’s only two explanations:
- You aren’t sufficiently certain that what you’re going to deploy is ready, or
- You’re superstitious.
Now, I’m as willing as the next person to claim that Murphy’s Law takes a perverse delight in visiting computer rooms, but realistically, that’s just a tendency to catastrophise* things when they come up unexpectedly.
So, if you’re sitting back and saying that you or the company should hold off doing something on a Friday because, well, it’s Friday, it’s time to sit back and ask yourself – is it because you’re being superstitious, or is it because it’s just simply not ready to be done, regardless of what day it is?
I know I will be.
—
* Thanks to my good friend Christopher Banks (aka @bipolarbearnz) for introducing me to that word last night. I’ll be using it daily for months, I think.
In defense of the “no changes on Friday” rule – upgrades, changes, and install are completely different from day-to-day operations.
1. Many products that are stable once installed can be an adventure to install or upgrade. How many times have you been brought up short during an install because you don’t have a driver you didn’t know you needed?
2. Test environments almost never replicate production in every way, and never have the same workload. No matter how diligently testing was done (and too often it isn’t) even a small change can have unintended consequences when it goes live.
3. People like to take Fridays off, and when they take a long weekend, they may even go away. That means that if something DOES go wrong, it may be difficult to get the resources needed to fix it, especially if they had no idea they’d be needed. The best example is the change that “won’t impact anything”, that breaks a downstream system – on a Friday afternoon.
All valid points Mike 🙂
But if u ask those who deploy, that question, and the answer is no, maybe they should go back and test again 🙂
Look at it as part of your quality assurance, instead of a rule 🙂
I will try it here, oh my, they will hate me 😀
Enjoy the weekend everybody 🙂
Why not to deploy something on Friday? Because things break and we all want to have a free weekend.
Actually, that sort of avoids the point I was trying to make – if we think there’s a risk of what we’re doing is going to break something if we deploy it on a Friday, what makes it any better to deploy it on a Tuesday, or a Thursday, or a Wednesday, etc.?
Or in other words, what I’m trying to say is that we should be unsatisfied with the notion of failure regardless of what day it occurs.
It isn’t the notion of failure that makes you avoid deploying on a Friday – it is the support and recovery mechanisms around the deployment process which usually require people which ultimately has a cost associated with it (the currency of which is money, time, family etc).
Working through an example:
Application or service has a 50% change of failure in the _deployment_ process but 99% chance of sla once it is installed but also has some (unquantifiable?) risk of affecting other systems. Recovery time to deal with the failure is approximately 24 hours (95% chance) through standard processes (all resources available) or 48 hours otherwise.
So if you deploy on a Friday at 5pm (because heck, if we’re picking on Friday lets not worry about whether we deploy at 9am or 5pm? 🙂 and something goes wrong
1) What resources are available to recover ? none since everyone has gone so it either involves extra expense to deal with it in the same time frame or a longer recovery time. Yes you can have resources available to deal with it but who wants to pay for that ? Sometimes the business does and othertimes it’s willing to make the tradeoff to say it is cheaper to deploy on Tuesday than Friday so do that.
2) What are the cascading effects of the deployment failure and their recovery processes
Despite many businesses increasingly being 24-7 in operation i am not convinced that they have all adapted the business processes (some have!) to also deal with change in a 24-7 manner and that’s also acceptable – you might specify operations are 24-7 but change is 9-5 as your cost tradeoff?
Similarly you might have a operational process of 7 days a week but limit your change to a subset of that as it simply is more efficient.
That’s one theory of mine anyway..
cheers,
-jason