In the debut episode of Doctor Who for Matt Smith (“The Eleventh Hour”), the Doctor, having only just regenerated is faced with needing to work out what sort of food suits him. He churns through apples, yoghurt (to great comedic effect), sandwiches, bacon (much to the amusement of a young Scottish girl sourcing each demand) before settling on fish fingers dipped in custard.
The day to day operations of a backup administrator are fairly typical – diagnose any overnight issues, manage media and systems capacity, facilitate recoveries, add any new hosts, and ensure that maintenance/reporting functions are still in place. That’s the apples and the yoghurt, the sandwiches and the bacon of normal operations.
As I’ve said in the past though, enterprise backup is one of those rare domains where it not only touches on almost all other areas of IT, but is impacted by all other areas of IT. You don’t have the luxury of ignoring storage, or networking, or at least a basic understanding of all operating systems and databases being backed up – you have to be jack of all trades and master of some.
Because of that, backup administrators must always be ready for fishy custard. What’s fishy custard? That’s the unexpected, the completely novel situation – when your manager comes out of a meeting saying that you need to cater for another 20% of clients by the end of the weekend, or that you need to come up with a strategy for backing up a ten terabyte MySQL database running on SCO Unix connected by an ISDN line via a double-NAT firewall. Or when as a result of automated overnight patching, suddenly half of the clients failed their backups overnight.
We can’t literally prepare for fishy custard in that as much as anything it’s about the unknown and the completely unexpected. A lecturer I had at Uni once described a rather OCD colleague when he taught in the UK who carried around a large book with him. In that book he recorded contingency plans for all manner of emergency situations – such as what he might do if a meteorite struck in a field near to where he was riding a bike and lit a grassfire, etc. That’s not how you prepare for the unexpected. It’s how you get an odd-ball reputation.
Preparing for the unexpected (the fishy custard) is about entering each day willing to accept that the daily routine might have to be thrown out the window. Among the worst backup sites I’ve dealt with are those where the daily routine is maintained with such rigidity by the administrators and the managers that new and unexpected requirements get pushed out to “sometime later”, where “sometime later” equals “an entirely inappropriate time”. A balance has to be struck, and it has to be accepted that unexpected things will happen in the backup realm, regardless of whether you want them to or not. Sometimes it’s something unexpected that you need to file away and action later, and sometimes it’s something unexpected that you need to drop everything and action straight away.
So remember, like The Doctor, you need to start each day prepared for the possibility of fishy custard. It’ll make your job less stressful, and your backups more successful.