Backup is a production activity

I routinely check (via the handy WordPress dashboard) what searches lead people to my blog. Often it’s for content that already exists on my site, but it also routinely helps me think of new topics to cover. (Occasionally it also provides some wry humour – for instance, someone a few weeks ago searched for “after the sun freezes”, which led them, I believe, to my posting on when I’d get around to running a search using Wolfram|Alpha.)

Interesting one today though was “why backups should not be on a production server”, and I thought that in this case, there’s a couple of distinct responses. These are:

1. Backups should not run on an existing production server (when configuring a new environment), because they should not be provisioned to share resources with existing services. Or more importantly, backups are a sufficiently important activity that one should not have to interrupt them to generate an outage for another system that shares the same system, or vice versa.
2. Backups are a production activity and they must be run on a production server.

There are obviously different levels of “production”; I’d suggest at bare minimum there are two styles of production systems for any enterprise:

• Operational production systems – Those systems that the business uses on a day to day business to fulfill standard business operations.
• Infrastructure support production systems – Those systems that the business uses at the “back end” to facilitate the success of the operational production systems.

Unless you’re a backup services provider, your backup server will never be an operational production system. However, in all other instances, your backup server will be part of the infrastructure support production systems.

You may consider that to be splitting hairs, but there are very simple yet important reasons why we need to consider backup systems as production systems. These include, but are not necessarily limited to the following:

• In many companies, non-production systems have a tendency to be “borrowed from” whenever there’s infrastructure overruns. For example:
  • A little bit of disk space here and there may be taken away for large image storage;
  • Redundancy on the system may be reduced if “production” systems need more storage;
  • New services may be “temporarily” placed on the server because there’s no other place for them.
• Outages or failures may be considered “acceptable” or not as closely monitored for non-production systems – thus backup systems that experience hardware faults overnight may not be suitably looked at;
• Systems profiles/allocation may be unsuitable for the performance requirements for enterprise production backups (in one extreme instance, I saw a desktop PC, years older than existing servers, used as a backup server!)
• CapEx/OpEx is improperly seen as something that should come from the IT budget rather than the operational budget of the company.

Let there be no uncertainty here – when it comes to production infrastructure support systems, your backup server, providing protection for your operational production systems, is equally as critical as all of the operational production systems it services.