The NetWorker Blog » errors

No Zero Error Policy? No backup system

Preston de Guise — Thu, 14 Apr 2011 22:02:48 +0000

In the past I’ve talked about the importance of having zero error policies.

In “What is a zero error policy?“, I said:

Having a zero error policy requires the following three rules:

1. All errors shall be known.

2. All errors shall be resolved.

3. No error shall be allowed to continue to occur indefinitely.

If you’ve not read that article, I suggest you go read it, as well as the follow-up article, “Zero error policy management“.

I’m going to make, and stand by, with fervid determination, the following assertion:

If you do not have a zero-error policy for your backups, you do not have a valid backup system.

No ifs, no buts, no maybes, no exceptions.

Why? Because why. Because across all the sites I’ve seen, regardless of size, regardless of complexity, the only ones that actually work properly are those where every error is captured, identified, and dealt with. Only those sites would I point at and say “They have every chance of meeting their SLAs”.

In my book, I introduce the notion that just deploying software and thinking you have a backup system is like making a sacrifice to a volcano. So, without a zero error policy, what does a network diagram of your IT environment look like?

It looks like this:

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Error lifecycle management

Preston de Guise — Mon, 07 Jun 2010 10:05:55 +0000

In previous articles I’ve discussed the need for zero error policies. This was covered first in What is a Zero Error Policy?, and followed up in Zero Error Policy Management. (If you’ve not read those articles, you really should before continuing.)

Key to ensuring a zero error policy is not only adopted, but also achieved, is a good understanding of the error lifecycle. That’s right – errors have a lifecycle, which is not only well defined, but actually helps us to keep them under control. An error lifecycle will resemble the following:

The start of the lifecycle is our Test and Detect loop:

Detect – An error is determined to have happened either as a result of a significant fault, or as a result of routine monitoring and analysis.
Test – An error is determined to have happened as a result of actual testing (formal or informal).

Once it’s determined that an error has happened, we then move into the resolution cycle, which consists of:

Diagnose – Determine the nature of the error – i.e., the root cause. If you don’t understand the actual cause, you can’t be certain that any solution you come up with is complete.
Rectify – Having understood the error, it’s time to resolve it. There’s two standard resolution techniques: complete resolution or workaround. Either are acceptable, so long as the resolution technique chosen is acceptable to the business and appropriate to the error.
Document – Once an error is solved, it needs to be documented. As has been said on numerous occasions, “Those who don’t learn from history are doomed to repeat it.” One of the worst possible error situations for instance is one where you’ve solved it in the past, but you can’t remember what you did and thus have to repeat the entire process. At minimum, documentation requires 3 components: (a) what lead to the error, (b) how the error manifests/is detected, and (c) how the error was resolved.

The error lifecycle doesn’t stop there though, as indicated by the diagram; instead, we add that error into a test and detection register – having encountered it, we should be able to more easily be on the look out for another instance. This is hopefully where the error finishes: being monitored for, but never again recurring. In the event though that it does reoccur, the diagnosis, rectification and documentation process should be simpler.

There you have it – the error lifecycle. Knowing it allows you to manage errors, rather than errors managing you.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The A-Z of Backup and Recovery

Preston de Guise — Thu, 07 Jan 2010 04:02:49 +0000

I’ve debated for a while whether to do this or not, since it might come across as somewhat twee. I think though that in the same way that “My Very Eager Mate Just Sat Up Near Pluto” works for planets, having an A-Z for backups might help to point out the most important aspects to a backup and recovery system.

So, here goes:

A	A is for Audit. Your backup system should be able to stand in front of an audit as complete and trustworthy.
B	B is for Backup. Without backup, you can't have recovery, and without recovery, your business is uninsured.
C	C is for Change Control. If your backup system isn't integrated into the change control process, neither your backup system nor your change control process works.
D	D is for DeDupe. You'll be seeing a lot more of it in Backup and Recovery moving forward. My money is on target dedupe being considerably more popular than source dedupe. Why? For the same reason that VTLs are around. Target dedupe = easier dedupe, both for vendors, and for companies with existing solutions to integrate.
E	E is for Errors, User. The most common reason you'll need to recover is from user errors. Use this to help plan how your backup system will work.
F	F is for Fast. Every person and their dog seems to have a story about making backups faster. Look instead for the stories about making recovery faster – they're the more important ones.
G	G is for Growth. Your backup environment should be scoped to handle at least 2 years growth upon implementation. If it isn't, budgets haven't been established correctly.
H	H is for Help. Don't try to solve backup/recovery problems in isolation; they're too important to let stew.
I	I is for Insurance. It's the central purpose of backup, and if you think of it any other way, chances are you're wrong.
J	J is for Jeckyll, not Hyde. When it comes to recovery situations, people should be able to work through them as calmly and cleanly as Dr Jeckyll might – not storm through them like Mr Hyde, flying apart.
K	K is for Knowledge. Know your system. Know your errors. Know where to look for information. Know your support hotline numbers. Know your averages. Know your performance peaks and your troughs. Know at a glance whether your system is running smoothly or having problems.
L	L is for Logs. Treasure your logs. Don't throw them away too quickly, make sure they're backed up too. With access to your logs, you can answer in 3 years time why a backup from yesterday is proving problematic to recover from.
M	M is for Magnetic Tape. It's not going away any time soon. Don't kid yourself, you'll still be using it in backup and recovery systems for some time to come.
N	N is for Napkin. If you can't summarise your backup system on the back of a napkin, it's too complicated. There are no exceptions to this rule.
O	O is for Order. Backups bring Order to Chaos. Hence, your backup system must be an ordered process, rather than a chaotic and haphazard arrangement of scripts and non-processes.
P	P is for Procedures; without them, you don't have a backup system at all.
Q	Q is for Query. If you're the backup administrator, you should be constantly prepared for a query about backup success. If you're a manager or system owner, you should feel confident you can get a positive response at any time to a query about backup success.
R	R is for Recovery, the most important facet of data protection.
S	S is for SLAs. (Service Level Agreements). Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs) form the heart of SLAs, and contrary to popular opinion in many circles, SLAs are vital to good design. Having SLAs is the first, most critical step to getting the correct budget for the correct system. Without defined recovery requirements, you can't prioritise activities properly; i.e., you'll have a reactionary environment rather than a proactive environment.
T	T is for Testing. In fact, T is for Testing, Testing, Testing. If your backup system doesn't include test planning, test procedures and test results, it's not a system at all.
U	U is for Ululate. It's that sound you make when your only copy of a backup is destroyed by a failing tape drive or failing tape because you didn't clone it, and you know that recovery failure is not an option.
V	V is for VTL. Whether you like the need for them or not, they're not going away any time soon.
W	W is for Windows. No, not that Windows. Backup Windows. Clone Windows. Recovery Windows. Design your system first to meet you recovery windows, then your clone windows, then and only then, your backup windows. If you don't do it in that order, your system isn't designed for recovery.
X	X is for X-Ray. If you can't X-Ray your backup status, drill down and see how happened, you should assume the worst. (OK, I'm grasping there, but what do you eXpect?)
Y	Y is for Yes. Yes you should be backing up. Yes you should be checking the backup status. Yes you should be able to recover.
Z	Z is for Zero Error Policy. If you don't run your backup system with a zero error policy, you're not running it properly, and it's not actually a system.

And there we have it. Maybe neither short, nor succinct, yet hopefully useful none-the-less.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

And I had thought it was some marketing stunt…

Preston de Guise — Fri, 13 Nov 2009 20:49:05 +0000

So yesterday I got an excited email from Virgin’s Velocity Rewards Program*.

It enthusiastically read:

Preston, surprise! You’ve turned gold!

This was news to me. I looked at my skin and indeed, I was my normal complexion – I had not suddenly morphed into a victim in a bad James Bond movie.

Reading on, I found out:

We’ve got a treat for you – a free upgrade to Velocity Gold! Given you came so close to making it on your own, we wanted to say thanks so much for your ongoing commitment to the Virgin Blue Group, we really love having you around.

I initially thought that Virgin had lowered their standards somewhat on their Velocity rewards program – after all, I hadn’t flown with them since February 2006. However, I decided after reading the rather spartan programme of goodies that one got with Gold Velocity rewards that it was just some marketing stunt to get me to go fly Virgin again after such a long absence. “Aww….”, I thought to myself, “Some programmer has developed an algorithm that puts me into the ‘needs to fly with us again’ bucket.”

About 3 hours later though, the explanation came through – it was just an error. Fortunately though, there was an explanation … it was caused by Friday the 13th:

friday the 13th strikes

Oops! Due to an error you’ve received our previous email by mistake. Please disregard the free upgrade communication as unfortunately you do not qualify for that upgrade.

We apologise for any inconvenience caused.

Blaming it on Friday the 13th explains everything. Way to go, Virgin. A pity really, I was half considering booking a flight with the Gold Rewards for a Friday afternoon sometime, just so I could hear the cabin crew doing their funny safety messages. (The best I ever heard was on a return trip from Melbourne, “In the event of your aeroplane becoming a cruise ship, there is a life jacket under your seat … your life jacket has a light, and a whistle for attracting sailors.”)

–
* Note – I said the email was excited. It didn’t really excite me all that much.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.