Backup to Disk and Busy State Staging

Backup to disk has well and truly become entrenched as a core backup strategy in most companies. By “backup to disk” I’m referring to either of ADV_FILE devices or VTLs – i.e., the general notion of backing up first to disk. For the rest of the article, since I’m feeling a little lazy today, I’ll follow industry norm and call backup to disk by the generic “B2D”.

Now, in most companies, there’ll still be physical tape involved. Long-term backups held on sufficiently replicated storage – even with deduplication – is going to remain costly for some time to come; but once B2D appears within an organisation, one of two architecture decisions will typically occur:

1. B2D region designed to hold a “significant” nearline capacity, where “significant” refers to a business-appropriate amount of recent backups.
2. B2D region designed as a “staging” region to have just enough capacity, where “just enough” means that if data isn’t staged daily (or near-daily), staging areas will become full and backups will stop.

Having observed B2D regions designed as staging-only on several occasions now, I’m even more firmly convinced that B2D as staging is a false economy that fails to take into consideration a few key metrics. Sure, buying say, 5TB or 10TB of disk is cheaper than buying 40TB with deduplication, but the cost of storage doesn’t end with the purchase. In fact, since the actual dollar cost of storage is typically amortised out over its expected deployment time, that cost often ends up being pretty minimal.

There are three distinct costs that I see as evident when using B2D purely as a staging region. These are:

• Staff time.
• Physical wear and tear.
• Increased risk of recovery failure.

Before I go further, I want to cover a term I used in the title of this post; “busy state staging” – it refers to environments where a significant portion of each day is spent with the B2D region being used to stage out from disk to physical tape, so as to free up room. There’s probably four key activities a backup system can be doing at any one time. These are:

• Backup
• Recovery
• Duplication/Cloning
• Maintenance

Backup, recovery and cloning are all givens; maintenance functions encompass media import/export/labelling, configuration activities, and most definitely includes staging. That’s right – staging is not any of backup, recovery or cloning; it falls into the category of moving data around in order to keep the system running. It’s effectively an overhead function for the environment, and as we know, the aim in any environment is to keep overheads to a minimum.

Over the expected deployment period of the B2D region in a backup system, I’d argue that those three costs previously cited add up to enough to demonstrate that the vast majority of businesses should not deploy B2D in a staging-only configuration. Let’s consider each of them individually.

Staff Time

This is the easiest to factor in. Let’s say your backup administrator has to spend roughly an hour a day between monitoring and maintaining free capacity on a staging-only B2D region. Now add up those hours per day, per week, per year across the lifetime of a deployment, and see how much it represents based on the hourly rate of the backup administrator. Assume $40 per hour, 4 weeks annual leave a year. So that leaves 48 weeks, 5 hours per week at $40 an hour. That’s $9,600 per year of staff costs through managing a poorly provisioned B2D region.

Usually that’s not the final cost though in staff time – my personal experience is that there’s a higher tendency in environments that use B2D for staging to need to engage temporary contractors, etc., to help fill in on projects where systems administration staff don’t have available time to do other projects in the company. So let’s assume that as a result of the backup administrator having to focus on B2D staging an hour a day the organisation has to engage a contractor one week a year to make up the short-fall. Assuming a contracting rate of $80 per hour, that’s $3,200 per year.

Now, assuming B2D storage has been provisioned over a 3 year period, we’re adding $38,400 to the maintenance impact of a staging-only region.

My gut feel, by the way, is that in an appropriately provisioned B2D architecture, the backup administrator will spend at most one fifth of the time in B2D storage administration; and there won’t be a need to engage contractors for that reason. So that $38,400 cost would shrink to say, $5,760 of time. In anyone’s books, that’s a good percentage saving.

Physical Wear and Tear

We’d count ourselves lucky if the only impact of using B2D in a staging configuration were staff costs. There’s more though. The wear and tear on both physical media and physical tape drives will be significantly increased, as these units will be running more frequently. Not only that, rather than having a reduced priority, the service time on physical tape is almost as critical in a tape-only environment. The net consequence is that rather than being able to say, work with a next-day service contract for the physical tape libraries, organisations are forced to stick with a 4-hour same-day response contract. As we know, there’s usually a pretty significant price difference between these types of contracts!

Increased Risk of Recovery Failure

We’d equally count ourselves lucky if the only impacts of using B2D in a staging-only configuration were just staff time and increased maintenance costs. The real insidious cost though is the risk of a recovery failure. In this, I’m not referring to any limitations that may exist around simultaneously recovering to while staging/cloning from B2D media. What I’m referring to is the risk that a backup may not actually run in the first place because a staging region becomes full, blocking new sessions starting. When considered from a backup perspective, that may not sound a lot. Turning it around to the purpose of a backup: imagine the consequence though of that data that was never backed up being needed for a recovery. While it may be logical to say “if it can’t be backed up, then we can’t factor it into recovery requirements”, but disasters, emergencies and auditors do not come when it’s convenient for us.

With this in mind, any backup that fails to run because a staging area is full should be considered from the full impact of a recovery SLA being breached for that data. That may sound harsh, but I’d actually suggest it’s a more business-focused rather than IT-focused approach to backup.

How’s that busy-state staging sounding now?

Enterprise data protection is one of those areas where businesses are most tempted to do cost cutting. We see it with Icarus support contracts, with inappropriate coupling of services, and we see it with B2D staging areas. We can intuit with almost no effort that busy state staging isn’t the best backup model. If your system is busy 20 hours a day between backup, cloning and maintenance functions, then it’s obvious that there’s at least an increased risk of parts failure; but the cost of the architecture is also magnified by wasted staff time, increased maintenance contract costs, and the potential failure to facilitate business-required recoveries.

When we take all those things into consideration, architecting B2D for significant or at least appropriate nearline recovery purposes rather than just staging becomes the cheaper option.