Basics – Skip and Forget

 Basics, NetWorker  Comments Off on Basics – Skip and Forget
Apr 192010

For quite a while I worked under the assumption that you could do the following with directives in NetWorker:

<< /path >>
+skip: *.mp3
<< /path/subpath/criticalpath >>

The logic of this is that it should be possible to skip files in one directory, but forget that directive in a lower directory and thus be able to still backup files matching a particular criteria in a subpath.

Recent discussions on the NetWorker mailing list left me questioning whether I was correct in my assumption. I thought I’d tested it long ago, but the discussions on the list (and the tests that I did) seemed to indicate this wasn’t the way NetWorker worked.

It turns out I was testing incorrectly. Instead of testing with an exact specification such as the above, I was testing “lazily”:

<< /path >>
+skip: *
<< /path/subpath/criticalpath >>

The mistake that I made was in the “*” vs the “*.mp3” I should have been testing my use case scenario. In short:

  • Obviously skipping “*” will result in NetWorker determining that everything is being skipped, at which point there is no need to continue to traverse any directory path beneath the point in which the “skip *” is encountered.
  • However, if just skipping a particular pattern, then NetWorker will have to continue to traverse all subdirectories from the path it encounters the skip command, meaning that the forget directive will still be honoured at a deeper directory path.

So I wasn’t wrong about my long-term belief, I just tested incorrectly.

This does mean that you can use skip, followed by forget, so long as your skip isn’t too open in its selection criteria.

Oct 202009

A while ago, I made a posting about a long-running annoyance I have with directive management in NetWorker.

This time I want to expand slightly upon it, thanks mainly to some recent discussions with customers that pointed out an obvious and annoying additional lack of flexibility in directive management.

That’s to do with the complete inability to apply directives – particularly the skip or the null directive, against “special” savesets. By “special” I’m referring to savesets that aren’t part of standard filesystem backups yet are still effectively just a bunch of files.

Such as say:

  • ASR:

(And so on.)

In short, NetWorker provides no way of skipping these savesets while still using the “All” special saveset for Windows clients. You can’t do any of the following:

  • Hand craft a server-side directive
  • Hand craft a client-side directive
  • Use the directive management option in the client GUI (winworkr) to create a directive to skip these styles of savesets.

OK, the last point is just slightly inaccurate. Yes, you can create the directive using this method – but:

  • The created directive is not honoured, either when left as is, or by transferring to a more standard directive;
  • The created directive is “lost” when you next load winworkr’s directive management option. Clearly it lets you create directives that aren’t valid and it subsequently won’t deal with.

Why does this suck? For a very important reason – in some situations you don’t want to have to back these up, or you can’t back them up. For instance, on certain OS levels and bitness using clusters, you will get an error if you try to backup the ASR: saveset.

This creates a requirement to either:

  1. Accept that you’ll get an error every day in your backup report (completely unacceptable)
  2. Switch from exclusionary backups to inclusionary backups (highly unpalatable and risky)

Clearly then the option is the second, not the first. This though is effectively removing an error by introducing poor backup systems management into the environment.

It would be nice if this problem “went away”.

Sep 032009

I’m a big fan of careful management of directives – for instance, I always go by the axiom that it’s better to backup a little bit too much and waste some tape than it is to not backup enough and not be able to recover.

That being said, I’m also a big fan of correct use of directives within NetWorker – skipping files that 100% are not required, adjusting preferences for the way files are backed up (e.g., logasm), etc., are quite important to getting well running backups.

So needless to say it bugs the hell out of me that after all this time, you still can’t include a directive within a directive.

Or rather, you can, but it’s through a method called “copy and paste”, which as we know, doesn’t lend itself too well to auto updating functionality.

So the current directive format is:

<< path >>
[+]asm: criteria

For example, you might want directives for a system such as:

<< / >>
+skip: *.mp3

<< /home/academics >>

<< /home/students >>
+skip: *.mov *.m4v *.wma *.dv

Now, it could be that for every Unix system, you also want to include Unix Standard Directives. Currently the only way to do this is to create a new directive where you’ve copied and pasted in all the Unix Standard Directives then added in your above criteria.

This, to use the appropriate technical term, is a dogs breakfast.

The only logical way, the way which obviously hasn’t been developed yet for NetWorker but falls into the category of “why the hell not?” would be support for include statements. That way, it could be embedded into the directive itself.

For example, what I’m talking about is that we should be able to do the following:

<< / >>
include: Unix Standard Directives

<< / >>
+skip: *.mp3

<< /home/academics >>

<< /home/students >>
+skip: *.mov *.m4v *.wma *.dv

Now wouldn’t that be nice? Honestly, how hard could it be?

NB: The correct answer to “how hard could it be?” is actually “I don’t care.” That is, there’s some things that should be done regardless of whether they’re easy to do.

Directives and change control

 NetWorker, Policies  Comments Off on Directives and change control
Jul 152009

It’s easy to change NetWorker directives. A few clicks here and there if you use NMC, then a couple of lines of text rattled off into the right fields, and suddenly you’ve made anywhere from small, precise changes to massive changes to a backup.

It’s for this reason that I think that modifying directives within the backup configuration should be considered important enough that they warrant their own change control processes. (I’ve previously talked about the backup administrator needing to be part of the change control authorisation process – this is another aspect however.)

Now, don’t get me wrong – despite what former employees may think, I’m not keen on excessive levels of red tape. In fact, I think a smart system should be designed at all times to minimise administrative overheads while ensuring that all accounting is still correctly done.

That being said, directives are, for want of a better term, dangerous. Mis-used, they can result in recovered systems being unusable – in data loss.

With this in mind, like other aspects of the backup system (adding clients, removing clients, adjusting savesets etc.), adjusting directives or applying directives to clients should also form part of change control.

Whenever directives are being changed, or applied, the following questions should be asked:

  • What is not working as desired?
  • What is the solution required?
  • What are the minimal steps required to make those changes?
  • How can system recoverability following the changes be tested?

It’s that final point that often goes missing with directives. Once, a long time ago (long enough to be NetWorker 5.5.3), a customer providing backup services to a host of companies setup a “zero error policy” but due to budget and time constraints merely kept on adjusting directives to remove any file from the backup that couldn’t be opened/read during the backup process. The end result was unrecoverable systems.

By placing directive maintenance into the realm of change control, we don’t seek to add more red tape to the backup system, but more thought, and more consideration of the consequences of changes that may adversely affect data and systems recovery.