[root@tara ~]# nsrjb
1:	VTL1 	[enabled]
2:	VTL2 	[enabled]
No jukebox selected.
Please select a jukebox to use:? [1] _

(Slight aside: never assume the numbered list is the same; NetWorker doesn’t guarantee the order being the same between executions – in fact, I actually only put in an RFE about this a couple of days ago, as I’m hoping it could at least be alphabetically ordered at all times…)

If you want to avoid the jukebox-prompt from nsrjb, one of the easiest ways is to specify the jukebox name as part of the command – e.g.,

[root@tara ~]# nsrjb -j VTL1

That’s fine of course, but if the vast majority of the time you perform operations on a single jukebox, you can specify a default jukebox as an environment variable (NSR_JUKEBOX) and streamline your processes. For example, on Linux, using the bash, this might look as follows:

[root@tara ~]# export NSR_JUKEBOX=VTL1
[root@tara ~]# nsrjb

Jukebox VTL1: (Ready to accept commands)
slot  volume         pool           barcode   volume id        recyclable
1: 800840L4       ClientTesting  800840L4  3814088325       no
2: 800841L4       ClientTesting  800841L4  3797311146       no
3: 800842L4       ClientTesting  800842L4  3847642669       no
4: 800843L4       ClientTesting  800843L4  3780533937       no
5: 800844L4       ClientTesting  800844L4  3763756765       yes
6: 800845L4       ClientTesting  800845L4  3864419885       yes
<snip>

Being an environment variable, this is something you can choose to set locally – say, on a per storage-node basis, when you have multiple storage nodes. It’s relatively common for instance to have a tape library on one or more storage nodes, so for the appropriate logins (or even at a system level) on each storage node it would be possible to set the local jukebox as the default, thereby streamlining usage of the units.

As an example, here’s a lab storage node with the setting in use:

[root@fawn ~]# export NSR_JUKEBOX="rd=fawn:VTL3"
[root@fawn ~]# nsrjb -s tara

Jukebox rd=fawn:VTL3: (Ready to accept commands)
<snip>

For something that can take you less than 30 seconds to set, the environment variable NSR_JUKEBOX can certainly be a big time saver if you have multiple jukeboxes in your environment and (like me) you’re a command line junkie.

Share this using...

Related posts:

1. Basics – Peeking inside your jukebox without leaving your desk In order to speed up jukebox operations, NetWorker maintains a...
2. Staging and Connectivity Loss For a while now I’ve been working with EMC support...
3. Basics – VTLs and default media sizes While this is pertinent to all versions of NetWorker, it...

Related posts brought to you by Yet Another Related Posts Plugin.

The way you do this is via the save command, invoking with the switches “-n” and “-v”. Of course, these can be used in the same argument (-nv or if you prefer, -vn). The “-n” option tells NetWorker not to actually perform the backup, while the “-v” option tells NetWorker to be verbose as it walks the filesystem. This verbosity will tell you what application specific module (ASM) NetWorker is going to invoke against each file/directory encountered.

For instance, let’s consider a save against the /root folder on a backup server where I’ve got a directive that’s configured to skip everything in there. The start of the output looks like the following:

[root@tara ~]# save -b Default -nv /root
3817:save: Using tara.pmdg.lab as server
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /dev/
70342:save: save: got prototype for /
70342:save: save: got prototype for /dev/
70342:save: save: got prototype for /
70342:save: save: got prototype for /opt/
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /media/
70342:save: save: got prototype for /proc/sys/fs/
70342:save: save: got prototype for /var/lib/nfs/
chdir(/root)
Name=`/root/', name=`/root/', fname=`./'
32240:save: found protofile spec for /:
mntasm : backup4
70342:save: save: got prototype for /
66135:save: NSR directive file (/.nsr) parsed
70342:save: save: got prototype for /root/
66135:save: NSR directive file (/root/.nsr) parsed
walk(/root/, ./)
walk(/root/.nsr, .nsr)
uasm -s /root/.nsr
walk(/root/.odbc.ini, .odbc.ini)
uasm -s /root/.odbc.ini
walk(/root/backup.log, backup.log)
32246:save: matched internal `skip' on `backup.log' for `/root/backup.log'
walk(/root/micromanual-nsradmin, micromanual-nsradmin)
32246:save: matched internal `skip' on `micromanual-nsradmin' for `/root/micromanual-nsradmin'
walk(/root/lose-files.sh, lose-files.sh)
32246:save: matched internal `skip' on `licenses.sh' for `/root/lose-files.sh'

As you can see by the above, we get a file-by-file analysis of the walk process, and that includes the ASM that will be invoked against that file. For instance, looking at the ‘backup.log’ file, we see:

walk(/root/backup.log, backup.log)

32246:save: matched internal `skip’ on `backup.log’ for `/root/backup.log’

So the first line tells us that it’s encountered a file called ‘backup.log’, with a full path of ‘/root/backup.log’. The second line tells us that it’s matched the (internal) ASM, ‘skip’, on the file ‘backup.log’ for the path ‘/root/backup.log’ – i.e., it’s effectively telling us that it will skip that file.

While it may be messy, it is a valid and usable technique to see what your directives are going to do.

Share this using...

Related posts:

1. Using NetWorker for Live Search There was recently a discussion on the NetWorker mailing list...
2. NetWorker and Incremental Manual Backups There was a recent posting on the NetWorker mailing list...
3. January’s Top Post I had hoped that the NetWorker Power User’s Guide to...

Related posts brought to you by Yet Another Related Posts Plugin.

Using ADV_FILE devices can play havoc with conventional tape rotation strategies; if you aren’t aware of these implications, it could cause operational challenges when it comes time to do recovery from tape. Let’s look at the lifecycle of a saveset in a disk backup environment where a conventional setup is used. It typically runs like this:

1. Backup to disk
2. Clone to tape
3. (Later) Stage to tape
4. (At rest) 2 copies on tape

Looking at each stage of this, we have:

The saveset, once written to an ADV_FILE volume, has two instances. The instance recorded as being on the read-read only part of the volume will have an SSID/CloneID of X/Y. The instance recorded as being on the read-write part of the volume will have an SSID/CloneID of X/Y+1. This higher CloneID is what causes NetWorker, upon a recovery request, to seek the “instance” on the read-only volume. Of course, there’s only one actual instance (hence why I object so strongly to the ‘validcopies’ field introduced in 7.6 reporting 2) – the two instances reported are “smoke and mirrors” to allow simultaneous backup to and recovery from an ADV_FILE volume.

The next stage sees the saveset cloned:

This leaves us with 3 ‘instances’ – 2 physical, one virtual. Our SSID/CloneIDs are:

• ADV_FILE read-only: X/Y
• ADV_FILE read-write: X/Y+1
• Tape: X/Y+n, where n > 1.

At this point, any recovery request will still call for the instance on the read-only part of the ADV_FILE volume, so as to help ensure the fastest recovery initiation.

At some future point, as disk capacity starts to run out on the ADV_FILE device, the saveset will typically be staged out:

At the conclusion of the staging operation, the physical + virtual instances of the saveset on the ADV_FILE device are removed, leaving us with:

So, at this point, we end up with:

• A saveset instance on a clone volume with SSID/CloneID of: X/Y+n.
• A saveset instance on (typically) a non-clone volume with SSID/CloneID of: X/Y+n+m, where m > 0.

So, where does this leave us? (Or if you’re not sure where I’ve been heading yet, you may be wondering what point I’m actually trying to make.)

Note what I’ve been saying each time – NetWorker, when it needs to read from a saveset for recovery purposes, will want to pick the saveset instance with the lowest CloneID. At the point where we’ve got a clone copy and a staged copy, both on tape, the clone copy will have the lowest CloneID.

The net result is that NetWorker will, in these circumstances, when both tapes aren’t online, request the clone volume for recovery – even though in an extreme number of cases, this will be the volume that’s offsite.

For NetWorker versions 7.3.1 and lower, there was only one solution to this – you had to hunt down the actual clone saveset instances NetWorker was asking for, mark them as suspect, and reattempt the recovery. If you managed to mark them all as suspect, then you’d be able to ‘force’ NetWorker into facilitating the recovery from the volume(s) that had been staged to. However, after the recovery you had to make sure you backed out of those changes, so that both the clones and the staged copies would be considered not-suspect.

Some companies, in this situation, would instigate a tape rotation policy such that clone volumes would be brought back from off-site before savesets were likely to be staged out, with subsequently staged media sent offsite. This has a dangerous side-effect of temporarily leaving all copies of backups on-site, jeapordising disaster recovery situations, and hence it’s something that I couldn’t in any way recommend.

The solution introduced around 7.3.2 however is far simpler – a mminfo flag called offsite. This isn’t to be confused with the convention of setting a volume location field to ‘offsite’ when the media is removed from site. Annoyingly, this remains unqueryable; you can set it, and NetWorker will use it, but you can’t say, search for volumes with the ‘offsite’ flag set.

The offsite flag has to be manually set, using the command:

# nsrmm -o offsite volumeName

(where volumeName typically equals the barcode).

Once this is set, then NetWorker’s standard saveset (and therefore volume) selection criteria is subtly adjusted. Normally if there are no online instances of a saveset, NetWorker will request the saveset with the lowest CloneID. However, saveset instances that are on volumes with the offsite flag set will be deemed ineligible and NetWorker will look for a saveset instance that isn’t flagged as being offsite.

The net result is that when following a traditional backup model with ADV_FILE disk backup (backup to disk, clone to tape, stage to tape), it’s very important that tape offsiting procedures be adjusted to set the offsite flag on clone volumes as they’re removed from the system.

The good news is that you don’t normally have to do anything when it’s time to pull the tape back onsite. The flag is automatically cleared* for a volume as soon as it’s put back into an autochanger and detected by NetWorker. So when the media is recycled, the flag will be cleared.

If you come from a long-term NetWorker site and the convention is still to mark savesets as suspect in this sort of recovery scenario, I’d suggest that you update your tape rotation policies to instead use the offsite flag. If on the other hand, you’re about to implement an ADV_FILE based backup to disk policy, I’d strongly recommend you plan in advance to configure a tape rotation policy that uses the offsite flag as cloned media is sent away from the primary site.

–
* If you did need to explicitly clear the flag, you can run:

# nsrmm -o notoffsite volumeName

Which would turn the flag back off for the given volumeName.

Share this using...

Related posts:

1. Basics – Need a different volume from pool … The scenario: A clone or stage operation has aborted (or...
2. Validcopies hazardous to your sanity While much of NetWorker 7.6′s enhancements have been surrounding updates...
3. Manually Staging? Don’t forget the Clone ID! Something that continues to periodically come up is the need...

Related posts brought to you by Yet Another Related Posts Plugin.

It doesn’t matter whether you’re on Windows, Linux or Unix. Like Batman’s utility belt, having some tools will help you go beyond  a standard NetWorker install.

What I’ll do is outline what my NetWorker utility belt would look like, and then let others comment on what they’d declare as the essentials for themselves. Here’s what I advocate as “must haves” when installing NetWorker:

• An advanced scripting language – in my case, Perl.
• SMTP mail (outgoing) from the backup server.
• SSH (outgoing) from the backup server. (On Windows, this implies use of a bare cygwin install, etc.)
• IDATA Tools – I kid you not, I’m saying it just “for sales”, I’ve been working on these tools for years and they’re such second nature for certain operations that unless I’m running up a lab server for only a single test, it even gets installed on all my test systems too.
• The “tail” command; whether it’s installed by default on Unix, or added as a single command on Windows or added as part of a cygwin install on Windows, I can’t go without tail.
• A web browser – I know that sounds like a given, but on headless enterprise Unix systems, that means ensuring that at least elinks is installed on the NetWorker server itself.
• A tool for viewing potentially large log files. My tool of choice is usually vi, but I’m a grouchy old Unix user.

So, they’re my “absolutes” – or to be more correct, they’re the tools I’ll either (a) want to automatically install or (b) automatically miss if they’re not installed when I step up to a NetWorker server.

Does this somehow detract from NetWorker? Of course not. Most of those, as you’ll see, are about useful situations around the backup product rather than direct modification of it. I.e., they’re about system process tools. Those that are to do about scripting should be welcomed – I’d take any backup framework product over any monolithic backup product any day!

So, what’s in your utility belt? Or what do you wish was in your utility belt for NetWorker?

Share this using...

Related posts:

1. nsrnmo[291]: -l: not found Yesterday I experienced one of those weird NetWorker issues that...
2. NetWorker 7.6 Enhanced Monitoring One of the features most missed by NetWorker users since...
3. Jobquery finally gets the duct tape removed Some time ago, I posted that EMC had added a...

Related posts brought to you by Yet Another Related Posts Plugin.

That high number of downloads has well and truly been reflected in the fact that the article introducing the micromanual was the top viewed article for January.

If you’ve not already checked out the micromanual, please feel free to download it. Don’t be afraid of the request for a name and email address – I’m not harvesting this information for any nefarious purposes. As I state quite clearly on the download page, it’s only to let you know if there are any updates to the manual. Any person who has already downloaded the manual will attest to the fact that I’ve not contacted them – and that’s because there’s been no updates yet.

As a side note, this blog is now officially a year old, and the readership continues to grow – a big thank-you to everyone for taking the time to read what I have to say!

Share this using...

Related posts:

1. micromanual: NetWorker Power User Guide to nsradmin I’m pleased to say that the first micromanual for the...
2. Validcopies hazardous to your sanity While much of NetWorker 7.6′s enhancements have been surrounding updates...
3. NetWorker and Incremental Manual Backups There was a recent posting on the NetWorker mailing list...

Related posts brought to you by Yet Another Related Posts Plugin.

Here’s the scenario. A customer was:

• Previously running NetWorker 7.4.2 on their backup server.
• Upgraded the server to 7.5.1.
• Had a bunch of Windows clients and one Unix client.
• The Unix client was configured for filesystem backups and Oracle backups.
• All clients were running 7.4.2(ish). The Oracle module was 4.5.
• Once the upgrade was done, Unix filesystem backups continued to work but the Oracle backups would fail with:

client:RMAN:/path/to/script.rman 1 retry attempted
client:RMAN:/path/to/script.rman off
client:RMAN:/path/to/script.rman /path/to/nsrnmo[291]: -l:  not found
client:RMAN:/path/to/script.rman nsrnmostart returned status of 127
client:RMAN:/path/to/script.rman /path/to/nsrnmo exiting.

My first thought when a colleague asked me to have a look at it was that somehow there was enough of a slight enough incompatibility between 7.5.x and NMO 4.5 that some argument carried over from an earlier version of NMO was causing problems with talking to a 7.5.x server. This wasn’t the case. (Yes, I knew that the two versions are meant to be compatible, and when I’ve installed and used them they have been, but that doesn’t mean you can’t have one single setting somewhere that tickles a coding error across versions.)

I went back and forth with a few other checks with the customer, noting that there were various issues reported in the NMO applogs, but none specific enough to nail the problem. So since everything looked OK I agreed with the customer that a WebEx would probably help us solve the issue faster.

Even though the customer had given me the client resource, I hadn’t found anything wrong with the backup command or the save set name, so out of curiosity I’d asked the customer when we started the WebEx to show me the client details. The saveset looked fine, so we jumped across to the backup command, and that also looked fine. But then, underneath the backup command, there was the “save operations” field, and in that save operations field held:

VSS:*=off

It hadn’t been recently added. It had been there since before the upgrade, and before the upgrade the backups had been working. But as we know, on pre-VSS Windows systems invoking that will cause backup failures, so I asked the customer to remove that entry and start the backup. Neither of us really thought that this would solve the problem, given the filesystem backups were still working, but lo and behold, with that removed the Oracle RMAN backups started properly working.

In retrospect, this of course was definitely the problem, but working it out was a bit more challenging. The reason was that the configuration shouldn’t have worked under a NetWorker 7.4.x server either, but for some reason it did. The 7.4.x NetWorker server was likely not sending through the VSS directive to the Unix client and the Unix Oracle module, but having upgraded to 7.5.x, the new install stopped “filtering the error” and started causing the problem to manifest. Or alternatively, 7.4.x and 7.5.x both send the save operations setting, but just differently enough to be dangerous.

I wouldn’t exactly say this was NetWorker’s fault – those VSS options are only designed for use with Windows 2003 and higher clients, and I’d guess that the VSS:*=off was just applied to every single client on the customer site without considering the 1 x Unix client.

In retrospect, the following line now completely makes sense:

client:RMAN:/path/to/script.rman off

That was our only “hint” as to the cause of the problem in the savegroup completion. It wasn’t enough by a long stretch. Sometimes, and this is the challenging bit – sometimes you can have configuration errors even if you haven’t changed the actual resource configuration. Different versions of NetWorker will react differently to an incorrect configuration – so the upgrade didn’t cause the problem, it just allowed the problem to appear.

Share this using...

Related posts:

1. NetWorker and Incremental Manual Backups There was a recent posting on the NetWorker mailing list...
2. Gotchas for disparate versions of NMC and NetWorker A few days ago a customer was having a rather...

Related posts brought to you by Yet Another Related Posts Plugin.

You may think that as of 7.5 onwards, where the level is expressly ignored for manual backups, that this isn’t possible:

[root@tara ~]# save -l incr -b Default /tmp
Client initiated backup.Option '-l' is ignored and backup is performed at level adhoc

After all, in 7.4 and below, if you ran the above command anyway, you wouldn’t have actually got an incremental backup of /tmp anyway – sure, it would have been tagged as an incremental backup, but that’s not the way that non-complete backup is actually generated in NetWorker. You see, NetWorker needs a timestamp to base a non-full backup against. That timestamp is going to be the nsavetime of a previous backup. (For an incremental, it will be the nsavetime of whatever the most recent backup for the saveset was – for differentials, it may vary.)

I’ll walk through an example of getting an incremental manual backup. It will still be tagged in NetWorker as a manual backup (that just is unavoidable these days), but it will at least just be an incremental. To start with, I need a full backup of something. I’ve got a full backup of my /usr/share directory as its own saveset here:

[root@tara ~]# mminfo -q "name=/usr/share" -r volume,level,sumsize,nsavetime
 volume          lvl   size  save time
800803L4        full 1244 MB 1263844861

Now, in order to be able to run a ‘manual’ incremental backup against this, I need to run save with a -t (for time) option – and the time I use will be 1263844861, which will backup all changes to that directory since the last backup.

So the command becomes:

[root@tara ~]# save -q -LL -t 1263844861 /usr/share
66135:save: NSR directive file (/.nsr) parsed
save: /usr/share  251 KB 00:00:20    588 files
completed savetime=1263880379

Note there that I haven’t included a level. If I had, even with the “-t” option included, NetWorker would have still generated the warning/error about ignoring the level for client initiated backups. However, I can confirm that it’s effectively an incremental backup by checking mminfo and looking at the sumsize field again:

[root@tara ~]# mminfo -q "name=/usr/share" -r volume,level,sumsize,nsavetime
 volume          lvl   size  save time
800803L4        full 1244 MB 1263844861
800803L4      manual 251 KB 1263880379

As you can see, we’ve got a full backup, and a subsequent manual backup that is effectively an incremental against the full.

Where is this useful? I wouldn’t imagine that it’s something you should be making use of in normal operations. However, in an emergency, when there’s an upgrade about to be done and you need to walk someone through doing an incremental backup before the upgrade without giving them administrative access to the backup server, this would be the sort of technique that can come in handy.

Share this using...

Related posts:

1. Validcopies hazardous to your sanity While much of NetWorker 7.6′s enhancements have been surrounding updates...
2. Quibbles – Why can't you clone or stage incomplete savesets? NetWorker has an irritating quirk where it doesn’t allow you...
3. Manually Staging? Don’t forget the Clone ID! Something that continues to periodically come up is the need...

Related posts brought to you by Yet Another Related Posts Plugin.

Thankfully though, as of NetWorker 7.6, the duct tape has been fully removed and jobquery will happily give you that meta-information:

[root@nox ~]# jobquery
NetWorker jobs query utility.
Use the "help" command for help.
jobquery> help
Legal commands are:
          print [query] (set current query)
           show [attrlist]
          types
            all
           quit
           help [command]
              . [query]
              ? [command]
Where:
        query ::= attrlist
        attrlist ::= attribute [; attribute]*
        attribute ::= name [: [value [, value]* ]
jobquery> types
     Known types: job indication, save job, probe job,
                  savegroup job, session info, index save job,
                  savefs job, bootstrap save job, utility job,
                  active job db;

Now sure, in previous versions you could type commands in and get output, but not having the prompt to tell you when jobquery was waiting, or when it was working, or when (as I originally thought) it was hanging on startup, is a fairly critical part in having a useful user interface.

Having a responsive user interface makes jobquery a little nicer to work with. For instance, let’s look at the “save job” type and run a backup. There’s a lot of fields in the “save job” resource type, so I’m going to limit it as follows:

jobquery> show command:; group name:; host:; job state:; level:
jobquery> print type: save job
                     command: \
"save -s nox.anywebdb.com -g \"Staging Servers\" -LL -f - -m nox\
 -t 1262601936 -l incr -q -W 78 -N /d/nsr/01 /d/nsr/01";
                  group name: Staging Servers;
                        host: nox;
                   job state: COMPLETED;
                       level: incr;

Then, once the backup starts, if I run (and abbreviate the output, remembering it shows all jobs in the database!), I can see:

                     command: \
savepnpc -s nox.anywebdb.com -g archon -LL -f - -m archon -t 1262818820 -l in\
cr -q -W 78 -N / /;
                  group name: archon;
                        host: archon;
                   job state: ACTIVE;
                       level: incr;

However, the “ACTIVE” state simply means that the job is actively queued, not that it’s actually sending data. If you want to only see jobs that are actively backing up rather than just simply active, you’d look for a job state of “SESSION ACTIVE”:

jobquery> print type: save job; job state: SESSION ACTIVE
                     command: \
savepnpc -s nox.anywebdb.com -g archon -LL -f - -m archon -t 1262818820 -l in\
cr -q -W 78 -N / /;
                  group name: archon;
                        host: archon;
                   job state: SESSION ACTIVE;
                       level: incr;

                     command: \
savepnpc -s nox.anywebdb.com -g archon -LL -f - -m archon -t 1262818814 -l in\
cr -q -W 78 -N /Volumes/Yu /Volumes/Yu;
                  group name: archon;
                        host: archon;
                   job state: SESSION ACTIVE;
                       level: incr;

Share this using...

Related posts:

1. NetWorker 7.6 and nsr_render_log You may recall that in an earlier article, “(In)securing your...
2. NetWorker and Incremental Manual Backups There was a recent posting on the NetWorker mailing list...
3. Validcopies hazardous to your sanity While much of NetWorker 7.6′s enhancements have been surrounding updates...

Related posts brought to you by Yet Another Related Posts Plugin.

One of those new features is the validcopies flag, something I unfortunately failed to check out in beta testing. It looks like it could use some more work, but the theory is a good one. The idea behind validcopies is that we can use it in VTL style situations to determine not only whether we’ve got an appropriate number of copies, but they’re also valid – i.e., they’re usable by NetWorker for recovery purposes.

It’s a shame it’s too buggy to be used.

Here’s an example where I backup to an ADV_FILE type device:

[root@tara ~]# save -b Default -e "+3 weeks" -LL -q /usr/share
57777:save:Multiple client instances of tara.pmdg.lab, using the first entry
save: /usr/share  1244 MB 00:03:23  87843 files
completed savetime=1259366579

[root@tara ~]# mminfo -q "name=/usr/share,validcopies>1"
 volume        client       date      size   level  name
Default.001    tara.pmdg.lab 11/28/2009 1244 MB manual /usr/share
Default.001.RO tara.pmdg.lab 11/28/2009 1244 MB manual /usr/share

[root@tara ~]# mminfo -q "name=/usr/share,validcopies>1" -r validcopies
6095:mminfo: no matches found for the query

[root@tara ~]# mminfo -q "name=/usr/share,validcopies>1"
 volume        client       date      size   level  name
Default.001    tara.pmdg.lab 11/28/2009 1244 MB manual /usr/share
Default.001.RO tara.pmdg.lab 11/28/2009 1244 MB manual /usr/share

[root@tara ~]# mminfo -q "name=/usr/share,validcopies>1" -r validcopies
6095:mminfo: no matches found for the query

[root@tara ~]# mminfo -q "name=/usr/share,validcopies>1" -r validcopies,copies
 validcopies copies
 2     2
 2     2

I have a few problems with the above output, and am working through the bugs in validcopies with EMC. Let’s look at each of those items and see what I’m concerned about:

1. We don’t have more than one valid copy just because it’s sitting on an ADV_FILE device. If the purpose of the “validcopies” flag is to count the number of unique recoverable copies, we do not have 2 copies for each instance on ADV_FILE. There should be some logic there to not count copies on ADV_FILE devices twice for valid copy counts.
2. As you can see from the last two commands, the results found differ depending on report options. This is inappropriate, to say the least. We’re getting no validcopies reported at all if we only look for validcopies, or 2 validcopies reported if we search for both validcopies and copies.

Verdict from the above:

• Don’t use validcopies for disk backup units.
• Don’t report on validcopies only, or you’ll skew your results.

Let’s move on to VTLs though – we’ll clone the saveset I just generated to the ADV_FILE type over to the VTL:

[root@tara ~]# mminfo -q "volume=Default.001.RO" -r ssid,cloneid
 ssid         clone id
4279265459  1259366578

[root@tara ~]# nsrclone -b "Big Clone" -v -S 4279265459/1259366578
5874:nsrclone: Automatically copying save sets(s) to other volume(s)
6216:nsrclone:
Starting cloning operation...
Nov 28 11:29:42 tara logger: NetWorker media: (waiting) Waiting for 1 writable volume(s)
to backup pool 'Big Clone' tape(s) or disk(s) on tara.pmdg.lab
5884:nsrclone: Successfully cloned all requested save sets
5886:nsrclone: Clones were written to the following volume(s):
 BIG998S3

[root@tara ~]# mminfo -q "ssid=4279265459" -r validcopies
 0

[root@tara ~]# mminfo -q "ssid=4279265459" -r copies,validcopies
 copies validcopies
 3          3
 3          3
 3          3

In the above instance, if we query just by the saveset ID for the number of valid copies, NetWorker happily tells us “0″. If we query for copies and validcopies, we get 3 of each.

So, what does this say to me? Steer away from ‘validcopies’ until it’s fixed.

(On a side note, why does the offsite parameter remain Write Only? We can’t query it through mminfo, and I’ve had an RFE in since the day the offsite option was introduced into nsrmm. Why this is “hard” or taking so long is beyond me.)

Share this using...

Related posts:

1. Quibbles – Why can't you clone or stage incomplete savesets? NetWorker has an irritating quirk where it doesn’t allow you...
2. Manually Staging? Don’t forget the Clone ID! Something that continues to periodically come up is the need...
3. Client side compression and saveset sizes While it turned out to be unrelated, a recent customer...

Related posts brought to you by Yet Another Related Posts Plugin.

The short version of why is simple: If you stage by SSID alone, NetWorker will delete/purge all instances of the saveset other than the one you just created. This is Not A Good Thing for 99.999% of what we do within NetWorker.

So to demonstrate, here’s a session where I:

1. Generate a backup
2. Clone the backup to tape
3. Stage the saveset only to tape

In between each step, I’ll run mminfo to get a dump of what the media database says about saveset availability.

Part 1 – Generate the Backup

Here’s a very simple backup for the purposes of this demonstration, and the subsequent mminfo command to find out about the backup:

[root@tara ~]# save -b Default -LL -q /etc
save: /etc  106 MB 00:00:07   2122 files
completed savetime=1258093549

[root@tara ~]# mminfo -q "client=tara.pmdg.lab,name=/etc" -r volume,ssid,cloneid,
savetime
 volume        ssid          clone id  date
Default.001    2600270829  1258093549 11/13/2009
Default.001.RO 2600270829  1258093548 11/13/2009

There’s nothing out of the ordinary here, so we’ll move onto the next step.

Part 2 – Clone the Backup

We’ll just do a manual clone to the Default Clone pool. Here we’ll specify the saveset ID alone, which is fine for cloning – but is often what leads people to being in the habit of not specifying a particular saveset instance. I’m using very small VTL tapes, so don’t be worried that in this case I’ve got a clone of /etc spanning 3 volumes:

[root@tara ~]# nsrclone -b "Default Clone" -S 2600270829
[root@tara ~]# mminfo -q "client=tara.pmdg.lab,name=/etc" -r volume,ssid,cloneid,
savetime
 volume        ssid          clone id  date
800843S3       2600270829  1258094164 11/13/2009
800844S3       2600270829  1258094164 11/13/2009
800845S3       2600270829  1258094164 11/13/2009
Default.001    2600270829  1258093549 11/13/2009
Default.001.RO 2600270829  1258093548 11/13/2009

As you can see there, it’s all looking fairly ordinary at this point – nothing surprising is going on at all.

Part 3 – Stage by Saveset ID Only

In this next step, I’m going to stage by saveset ID alone rather than specifying the saveset ID/clone ID, which is the correct way of staging, so as to demonstrate what happens at the conclusion of the staging. I’ll be staging to a pool called “Big”:

[root@tara ~]# nsrstage -b Big -v -m -S 2600270829
Obtaining media database information on server tara.pmdg.lab
Parsing save set id(s)
Migrating the following save sets (ids):
 2600270829
5874:nsrstage: Automatically copying save sets(s) to other volume(s)

Starting migration operation...
Nov 13 17:34:00 tara logger: NetWorker media: (waiting) Waiting for 1 writable
volume(s) to backup pool 'Big' disk(s) or tape(s) on tara.pmdg.lab
5884:nsrstage: Successfully cloned all requested save sets
5886:nsrstage: Clones were written to the following volume(s):
 BIG991S3
6359:nsrstage: Deleting the successfully cloned save set 2600270829
Successfully deleted original clone 1258093548 of save set 2600270829
from media database.
Successfully deleted AFTD's companion clone 1258093549 of save set 2600270829
from media database with 0 retries.
Successfully deleted original clone 1258094164 of save set 2600270829
from media database.
Recovering space from volume 4294740163 failed with the error
'Cannot access volume 800844S3, please mount the volume or verify its label.'.
Refer to the NetWorker log for details.
6330:nsrstage: Cannot access volume 800844S3, please mount the volume
or verify its label.
Completed recover space operation for volume 4177299774
Refer to the NetWorker log for any failures.
Recovering space from volume 4277962971 failed with the error
'Cannot access volume 800845S3, please mount the volume or verify its label.'.
Refer to the NetWorker log for details.
6330:nsrstage: Cannot access volume 800845S3, please mount the volume
or verify its label.
Recovering space from volume 16550059 failed with the error
'Cannot access volume 800843S3, please mount the volume or verify its label.'.
Refer to the NetWorker log for details.
6330:nsrstage: Cannot access volume 800843S3, please mount the volume
or verify its label.

You’ll note there’s a bunch of output there about being unable to access the clone volumes the saveset was previously cloned to. When we then check mminfo, we see the consequences of the staging operation though:

[root@tara ~]# mminfo -q "client=tara.pmdg.lab,name=/etc" -r volume,ssid,cloneid,
savetime
 volume        ssid          clone id  date
BIG991S3       2600270829  1258095244 11/13/2009

As you can see – no reference to the clone volumes at all!

Now, has the clone data been erased? No, but it has been removed from the media database, meaning you’d have to manually scan the volumes back in order to be able to use them again. Worse, if those volumes only contained clone data that was subsequently removed from the media database, they may become eligible for recycling and get re-used before you notice what has gone wrong!

Wrapping Up

Hopefully the above session will have demonstrated the danger of staging by saveset ID alone. If instead of staging by saveset ID we staged by saveset ID and clone ID, we’d have had a much more desirable outcome. Here’s a (short) example of that:

[root@tara ~]# save -b Default -LL -q /tmp
save: /tmp  2352 KB 00:00:01     67 files
completed savetime=1258094378
[root@tara ~]# mminfo -q "name=/tmp" -r volume,ssid,cloneid
 volume        ssid          clone id
Default.001    2583494442  1258094378
Default.001.RO 2583494442  1258094377
[root@tara ~]# nsrclone -b "Default Clone" -S 2583494442

[root@tara ~]# mminfo -q "name=/tmp" -r volume,ssid,cloneid
 volume        ssid          clone id
800845S3       2583494442  1258095244
Default.001    2583494442  1258094378
Default.001.RO 2583494442  1258094377
[root@tara ~]# nsrstage -b Big -v -m -S 2583494442/1258094377
Obtaining media database information on server tara.pmdg.lab
Parsing save set id(s)
Migrating the following save sets (ids):
 2583494442
5874:nsrstage: Automatically copying save sets(s) to other volume(s)

Starting migration operation...

5886:nsrstage: Clones were written to the following volume(s):
 BIG991S3
6359:nsrstage: Deleting the successfully cloned save set 2583494442
Successfully deleted original clone 1258094377 of save set 2583494442 from
media database.
Successfully deleted AFTD's companion clone 1258094378 of save set 2583494442
from media database with 0 retries.
Completed recover space operation for volume 4177299774
Refer to the NetWorker log for any failures.

[root@tara ~]# mminfo -q "name=/tmp" -r volume,ssid,cloneid
 volume        ssid          clone id
800845S3       2583494442  1258095244
BIG991S3       2583494442  1258096324

The recommendation that I always make is that you forget about using saveset IDs alone unless you absolutely have to. Instead, get yourself into the habit of always specifying a particular instance of a saveset ID via the “ssid/cloneid” option. That way, if you do any manual staging, you won’t wipe out access to data!

Share this using...

Related posts:

1. Quibbles – Why can't you clone or stage incomplete savesets? NetWorker has an irritating quirk where it doesn’t allow you...
2. Staging and Connectivity Loss For a while now I’ve been working with EMC support...
3. Client side compression and saveset sizes While it turned out to be unrelated, a recent customer...

Related posts brought to you by Yet Another Related Posts Plugin.