We frequently encourage this sort of setup because it takes two items which NetWorker can run disparately – start time, and level, and effectively merges the two – something a lot of other backup products just do as the one configuration item. Perhaps even more importantly, in small to mid size businesses with modest data levels, this makes more sense anyway – it allows you to readily construct “classic” backup scenarios, such as “full on Friday, incrementals the rest of the week”. So from the perspective of level and amount of data backed up, your backup week would look similar to the following:

Now, as I said, this works well for businesses with modest data sizes. However, as the image graphically demonstrates, this creates scenarios where there is a significant disparity between the amount of data backed up on regular days and the amount of data backed up for the fulls. Remembering that it’s the full backups that frequently end up straining backup architectures, companies will often end up revisiting their architecture when the amount of data backed up on the “full day” becomes unmanageable.

For some companies, the full day is chosen for sound business reasons – finance companies for instance may have to do weekly full backups starting close of business Friday, and full monthly backups on the last Friday every month. In these scenarios, where there are important business reasons for keeping full backups on a single day of the week/month, the backup architecture must remain constantly configured to handle the massive spike that full backups create.

However, in other companies where there are no strong business reasons for running all the fulls on the same day, it’s worth remembering that there is an alternate configuration – ironically enough, it’s very much the “default” NetWorker configuration, it’s just one most sites tend not to use. This configuration sees the group control only the start time/collection of clients, and does not have a schedule/level override assigned. Instead, the schedule of each client defines what level backup will be done for that client. This sort of configuration resembles the following:

As you can imagine, this does require a slight change of administrative policies in relation to setting the correct schedule at the client level, and potentially needing additional client instances to handle the daily and monthly backups, but the advantage of this is that you can then start having groups where both incremental and non-incremental backups are done concurrently, spreading out the load of the full backups to create a significantly lower spike in resource requirements. So from the perspective of level and amount of data backed up, your backup week would instead look like the following:

This style of schedule isn’t for everyone – as I said, if you have a strong business need to restrict all full backups to a particular day, it’s very unlikely to work. I’d suggest as well that it may not be a good strategy if you happen to have a high staff turnover, as it does realistically add a little more complexity into the environment. (While your environment should be as simple as possible, that doesn’t always mean “as simple as conceivable”.)

In larger environments though with significantly higher amounts of data requiring backup, this style of configuration can be a real boon. Compare weekly fulls of say, 10TB (effectively tiny) with weekly fulls of say, 500TB, and you can instantly see the attraction of this programme. Instead of having to design a system capable of handling 500TB in 24 hours, you might instead be able to limit your design to a system that at most has to handle 100TB over a 24 hour period (factoring in incrementals + fulls on any given night). That’s not an insignificant difference.

[Edit, 2010-05-11]

What’s this got to do with large groups? It occurred to me overnight that while the title of the post was originally “Large group backups”, I diverged somewhat between the original intent of the post and the actual resulting post.

So, the other area where this can be useful is in situations where you have groups with large numbers of clients. For example, in environments with 500+ clients, where a single group may have hundreds of clients in it, switching to mixed levels in the one group has the same effect as for an entire large environment, but at a single, localised group.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Now, to start with, the architecture of having a client inactivity timeout setting is, I believe, flawed, and should be addressed by adding heartbeat functionality between the NetWorker server and the client backup process*.

There are a plethora of situations that don’t fall into client inactivity. These include:

• Blocking IO call failing (can happen to just about any product)
• Saveset initiation request sent but not responded to. (This is a tricky one to define – that seems to be the point where the failure happens, but it’s almost impossible to diagnose.)
• Backup server’s bootstrap/index:server saveset waiting for media on the backup server.

There have been various attempts to fix these situations over the years – for instance, most recently there were patches introduced into the 7.5 service pack stream to try to prevent a situation where a group would hang on startup probe. As is always the case with hanging situations, it’s difficult to say for sure whether those potential issues were well and truly dealt with.

What it remains clear though, and it’s really important to remember, is that just because you’ve set a “client inactivity” timeout within a group doesn’t guarantee that the group will timeout after a certain period of inactivity. I.e., it doesn’t excuse you from confirming on a daily basis whether your groups have finished or not.

Monitoring can be achieved a few different ways:

• Literally checking each group that is still running in NMC at a certain point in the day and determining whether it should be running or if it is hung.
• Paying special attention to savegroup completion reports that tell the group is “aborted, already running” (though that means missing a hung group for around 24 hours).
• Scripting a check and alert for still-running groups – like the NMC option, but automated.

It would be great to say that there should never be a case where a group hangs and doesn’t complete, but I recognise this is one of those things that’s difficult to program, and in actual fact is almost impossible to guarantee. Could it be handled better? Undoubtedly; it’s just I’m enough of a pragmatist to know that it’s never going to be perfect.

The catch-cry of the backup administrator should be “constant vigilance!” As I’ve discussed previously in posts about enacting zero error policies, it’s not about trying to configure a “set and forget” system where there’ll never be an issue, it’s about always having your finger on the pulse and never, ever accepting that there will be regular alerts for “events-that-look-like-errors-but-you-know-they’re-not”.

So while the client inactivity timeout in a group will save you from some mundane aspects of group administration, it won’t let you ignore monitoring your groups for unexpected states.

__________
* By flawed, I mean:

Currently the backup process works as follows:

1. Server instructs client to start backing up
2. Client starts sending data to appropriate storage node/nsrmmd process
3. If client fails to send any data for ‘inactivity timeout’ minutes, backup is considered to have failed, and restart is run if necessary.

This doesn’t suit situations where there’s a dense filesystem walk taking place, and in fact it really, really should work as follows:

1. Server instructs client to start backing up
2. Client starts sending data to appropriate storage node/nsrmmd process
3. Every X (e.g., 90) seconds or so when no data has been sent, the storage node/nsrmmd process asks the client if the save is still running.
4. If the client responds within X seconds, keep waiting.

That’s the sort of heartbeat mechanism that should be used…

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

However, sometimes snapshots and other high-availability options aren’t really what we want – we just want to be able to run a backup more frequently than 24 hours, and have it run automatically. (For instance, on particularly busy Oracle systems, you might want archived redo logs backed up every 4 hours, with logs deleted after 2 backups.)

Thankfully, NetWorker supports this (and has done for quite some time), via the interval setting in groups. By default, this is set to “24:00″ – 24 hours. It can however be set to a smaller value, which will trigger the group to run more frequently.

Before we consider smaller intervals, lets first revisit the key timing settings involved in a traditional group:

• Start Time – The time the group is configured to run. (Defaults to 03:33*).
• Interval – How often the group is configured to run. (Defaults to 24 hours).
• Restart Window – How many hours after the start time will the group, if restarted, only re-run those savesets that failed or never ran, instead of re-running the entire group. (Defaults to 12 hours.)

Now, all these options are still used (and required) under higher frequency backups, with their meaning as follows:

• Start Time – When the group is first run. This can be anything within a standard 24 hour window.
• Interval – How often the group will re-run. This is not affected by when the group finishes.
• Restart Window – Same as for standard interval backups.

So, let’s go back to that sample requirement – Oracle archived redo log backups run every 4 hours. Let’s consider setting up a new group that does this, with the backups starting at 00:01 initially, then running every 4 hours after that – i.e.,

• 00:01
• 04:01
• 08:01
• 12:01
• etc

Here’s what this group configuration would look like in NMC:

Group settings in NMC (1 of 2)

In the first pane, it looks fairly standard – setting a start time of 00:01, and enabling autostart. It’s the second pane where things are a little different:

Group settings in NMC (2 of 2)

Here, we set the interval to 4 hours, and the restart window to 2 hours.

–
* I’m told that there were some ‘fun’ numbers used by early NetWorker programmers. E.g., one of the original index checks used to run every ? weeks (or more correctly, every 22/7 weeks). It’s possible that the critical situation engineer who told me this may have been pulling my leg however. I do think though that given how so many people dislike backups, 03:33 may have been chosen as a start time as a play on 6:66!

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

• Daily Servers
• Monthly Servers
• Yearly Servers

All start at the same time. A common response when I express concern over this is “even though they all start at once, only one group will ever be backing up”. (I.e., skips are deployed appropriately.)

This isn’t sufficient.

Starting multiple groups simultaneously cause what I like to refer to as server spikes. That is, sudden, sharp increases in server resource usage. By ‘resource usage’, I’m not necessarily referring to memory and CPU, though that can occur, but by internal NetWorker communications and resource usage.

When server spikes occur, odd things can happen – albeit randomly and often intermittently, but they can still happen. Savesets might unexpectedly drop communications with the server and need to be restarted (or worse, hang, then continue once a second saveset for the same client/data is started by the server, creating load on the client); a single media load instruction might fail, or a single nsrmmd process might get timed out and restarted.

There’s an easy solution for this, and one which everyone should follow:

Never, ever, have more than one group start at the same time.

You don’t have to have a big gap. I’ve typically found that 5-10 minutes is an ample gap. If each group starts on its own, then the server behaves considerably more smoothly, and less weird intermittent/random failures occur. (If your response is that your backup windows don’t allow a five minute gap between groups, I’d reasonably confidently argue, even having not seen your site, that your backup configuration needs to be re-evaluated.)

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.