Feb 112010

As evidenced by the title of my book (Enterprise Systems Backup and Recovery: A corporate insurance policy), I’m a firm believer that the only way to conceptualise the purpose of backup is to describe it as insurance. The way I describe this is to compare the way in which we take out insurance, but hope not to use it, and to make backups, and similarly hope not to use them. This can be easiest described through a couple of Venn diagrams.

First, let’s look at insurance:

Backup and Insurance: Insurance Venn DiagramNo-one wants to claim on their insurance. We take it out on a yearly basis, and any year that we don’t have to use it is good. (Particularly in countries where insurance companies run rough-shod over morality, decency and legal restraint.) I personally have home insurance, contents insurance, car insurance, travel insurance (whenever I travel) and health insurance. Any time I don’t have to make a claim on any of these types of insurance is good – because in order to make a claim, something bad needs to have happened. So I’m much happier paying the fees each year and hoping that I don’t have any more involvement than that with my insurance agencies. Do I resent paying these fees? Hell no – because I’m well aware that if I don’t, and something bad happens, I’ll be up the creek without a paddle. (Or to use the Australian vernacular, I’d be up s––t creek.)

So let’s see the Venn diagram for backup:

Venn Diagram for BackupAs you can see, it’s spookily similar to the diagram for insurance. Now, one of the first things that I tend to hear when I roll out my “backup = insurance” argument is that occasionally, people will want to recover from backups – e.g., to migrate between systems, refresh Q/A systems from production, etc. Well, this isn’t really using backup for the primary purpose – recovery, but instead using it as a data migration/retrieval system. It’s a fine distinction, but it’s an important distinction. The primary reason backup systems are deployed is to recover data when there’s been a failure – any secondary benefit from a backup and recovery system is just that – a secondary benefit.

Your next question may be – so what point is there in classifying backup as a type of insurance?

This is the absolute core of why companies need to think of backup as being a type of insurance – it’s all about the budget.

Look at an example company. Let’s say there’s 5 departments:

  • IT
  • Finance and Human Resources
  • Sales
  • Warehousing and Operations
  • Solutions Delivery

In a standard company, each department will have it’s own budget, but there’s also the corporate budget. That’s the budget that covers costs which affect all departments and have to be met regardless of the size or capacity of each department – it’s for the core business costs. One of those “core” costs is usually the various insurance policies that companies take out. This will definitely include some sort of standard business insurance, but will then cover other types of insurance – professional indemnity, building insurance, contents insurance, car insurance, etc. Few businesses would argue that each department needs to individually seek out and/or pay for its own insurance on each of those matters.

The mistake then made by many businesses is to fail to think of backup as insurance, and therefore work on the basis that IT will manage data and systems backup out of its own budget. This sort of thinking leads to the most common disasters where:

  • Backup systems budget is cut to meet the budget requirements of “production” systems. (See my points here about why it’s a fallacy to think of backup systems as anything other than production systems.)
  • “Make do” data protection systems are deployed that require significant time to complete recovery – e.g., to “save” money, some IT departments will decide to only backup actual data, and leave operating systems and applications at the mercy of being re-installed from the ground up.
  • Backup retention is cut to reduce operational expenditure (i.e., limit the purchase of new media).
  • SLAs, if established, are silently ignored – or even railed against by IT.

None of these processes or decisions are conducive to sensible or useful business systems management – yet they’re the inevitable consequence of asking one department to meet costs that are shared between all departments. It would be like demanding that the sales department pay for all company insurance out of their budget: it just doesn’t make sense.

Where does this discussion leave us? There’s a lesson any business can take out of this: backup, being insurance, is something that’s funded by the corporate operational and capital budget, not the budgets of any individual department.

Chances are if your business isn’t thinking of backup as insurance, it’s not handling or funding backup properly either.

Dec 162009

There is a profoundly important backup adage that should be front and centre in the mind of any backup administrator, operator, manager, etc. This is:

It is always better to backup a little more than not quite enough.

This isn’t an invitation to wildly waste backup capacity on useless copies that can never be recovered from – or needlessly generate unnecessary backups. However, it should serve as a constant reminder that if you keep shaving important stuff out of your backups, you’ll eventually suffer a Titanic issue.

Now, people at this point often tell me either that (a) they’re being told to reduce the amount of data being backed up or (b) it makes their manager happy to be able to report less OpEx budget required or (c) some combination of the two, or (d) they’re reluctant to ask for additional budget for storage media.

The best way to counter these oppressive and dangerous memes is to draw up a graph of how happy your manager(s) will be over saving a few hundred dollars here and there on media versus potential recovery issues. To get you started, I’ve drawn up one which covers a lot of sites I’ve encountered over the years:

Manager happiness vs state of environment and needless cost savings on backupYou see, it’s easy to be happy about saving a few dollars here and there on backup media in the here and now, when your backups are running and you don’t need to recover.

However, as soon as the need for a recovery starts to creep in, previous happiness over saving a few hundred dollars rapidly evaporates in direct proportion to the level of the data loss. There might be minimal issues to a single lost document or email, but past that things start to get rather hairy. In fact, it’s very easy to switch from 100% management happiness to 100% management disgruntlement within the space of 24 hours in extreme situations.

You, as a backup administrator, may already be convinced of this. (I would hope you are.) Sometimes though, other staff or managers may need reminding that they too may be judged by more senior management on recoverability of systems under their supervision, so this graph equally applies to them. That continues right up the chain, further reinforcing the fact that backups are an activity which belong to the entire company, not just IT, and therefore they are a financial concern that need to be budgeted for by the entire company.

Media, CapEx and OpEx

 Backup theory, Policies  Comments Off on Media, CapEx and OpEx
Jul 022009

A common mistake I often see made, particularly when planning new system implementations, is to try to calculate out media costs over the entire planned growth period. That is, assuming a new backup system is going to be installed, accounting/management types will want to plan full tape requirements for the projected growth period the system was planned for (e.g., 3 years) from the outset.

The flaw of this approach is attempting to account for media costs as capital, rather than operational expenditure. This approach often results in unnecessary cost savings being made by cutting out other aspects of the system budget – software and hardware needed now are excluded from budget in order to make way for media that will be needed later.

This CapEx vs OpEx approach becomes most flawed when a system is being put in place making use of the “latest and greatest” media type. Let’s assume for the moment that LTO-5 has just been released, and a system with 4 x LTO-5 drives is installed, with planned capacity requirements for the next 3 years suggesting that 4,000 units of media will be required.

However, at just-released prices, media will be prohibitively expensive. Assume if you will that the RRP for each LTO-5 tape may be around $180 AU. Even with a bulk purchase discount bringing the price down to say, $100 AU per unit of media, that’s $400,000 of media if it is being purchased from the outset.

However, in the backup industry, we know that media gets progressively cheaper as it has been out for a while. Just look at all the LTO series of media. In Australia, each new format came out at around $180 RRP per cartridge. Now a simple search shows that I could pick up, on RRP alone, LTO-4 media for as little as $85 AU. (That’s just from one search, and for individual unit pricing.)

So going back to our not-yet-released LTO-5, assuming 4,000 units of media will be required across 3 years, the operational expenditure for that would be cheaper than the capital expenditure, and the media would only be purchased on an “as needs” or “near as needs” basis, ensuring media doesn’t sit on a shelf for lengthy periods of time before use.

Let’s say that media is purchased every 6 months for such a system, and an equal amount of media is purchased each time. So, every 6 months or so, one would need to order another 666 units of media for the system. Let’s round that up to 670 so we’re talking about packs of ten. We’ll assume an 11% decrease in the cost of the media every 6 months.

We’ll also assume that any bulk order (say, 300 units of media or more) will result in a 40% discount from the RRP. Let’s run a simple numbers game here then:

  • First purchase – 670 @ RRP $180 / Discount $108, $72,360.00
  • Second purchase – 670 @ RRP $160.20 / Discount $96.00, $64,400
  • Third purchase – 670 @ RRP $142.58 / Discount $86.00, $57,316
  • Fourth purchase – 670 @ RRP $126.89 / Discount $76, $51,012
  • Fifth purchase – 670 @ RRP $112.94 / Discount $68, $45,400
  • Sixth purchase – 670 @ RRP $100.51 / Discount $60, $40,406

That’s a total media OpEx budget over 3 years of just under $331,000, as opposed to $400,000 CapEx at the commencement of an implementation.

What’s more, because the media purchases are spread out over the course of the three years, rather than having to find $400,000 or even $331,000 up front, which would seriously put a dent in other budget activities, the most in any one financial year that would be required under OpEx for media would be in the first year, at a much lower $136,760.

Further, because backup is something that logically and operationally should source budget from the entire company, rather than this being OpEx out of the IT budget, it would be OpEx shared from all departmental budgets, or, if there’s a corporate overheads/OpEx budget, from that budget instead.

Contrary to popular belief, media purchases don’t have to be a nightmare or exorbitantly high.

%d bloggers like this: