Basics – Taking a turn about the filesystem

 Basics, NetWorker  Comments Off on Basics – Taking a turn about the filesystem
Apr 272015

“Miss Eliza Bennet, let me persuade you to follow my example, and take a turn about the room. — I assure you it is very refreshing after sitting so long in one attitude.”

Jane Austin: Pride and Prejudice.

The NetWorker savegrp command has a lot of different command line options, but one which falls into that useful-for-debugging category for me has always been the -n option. This allows you to invoke the save commands for a group (or a single client in the group) in walk/don’t do mode.

While filesystems have become considerably more capable at self-repair and resilient towards minor corruption, there was a time in the past where you could encounter an operating system crash as a result of attempting to access a particularly corrupt file or part of the filesystem. Backups, of course, want to walk all the filesystems (unless you direct them otherwise), and so being able to see what NetWorker might do during a backup was helpful to diagnose such issues. (Even if it meant one more crash.)

These days, if a host being backed up by NetWorker via a filesystem agent gets a lot of changes during a day, you might simply be interested in seeing just how many files are going to be backed up.

The command is pretty straight forward:

# savegrp -nv [-c client] groupName

For instance, consider the following execution:

[root@orilla ~]# savegrp -nv -c mondas Servers
90528:savegrp: mondas:All level=incr
7236:savegrp: Group will not limit job parallelism
83643:savegrp: mondas:All started
savefs -s orilla -c mondas -g Servers -p -n -l full -R -v
mondas:/ level=incr, vers=pools, p=4
mondas:/d/01 level=incr, vers=pools, p=4
mondas:/boot level=incr, vers=pools, p=4
mondas:/d/backup level=incr, vers=pools, p=4
90491:savegrp: mondas:All succeeded.
83647:savegrp: Servers mondas:All See the file /nsr/logs/sg/Servers/832077 for command output
83643:savegrp: mondas:/ started
save -s orilla -g Servers -n -LL -f - -m mondas -t 1430050510 -o MODIFIED_ASOF_TIME:timeval=1430050506;RENAMED_DIRECTORIES:index_lookup=on;BACKUPTIME:lookup_range=1429877707:1430050510; -l incr -W 78 -N / /
83643:savegrp: mondas:/d/01 started
save -s orilla -g Servers -n -LL -f - -m mondas -t 1430050508 -o MODIFIED_ASOF_TIME:timeval=1430050506;RENAMED_DIRECTORIES:index_lookup=on;BACKUPTIME:lookup_range=1429877710:1430050508; -l incr -W 78 -N /d/01 /d/01
83643:savegrp: mondas:/boot started
save -s orilla -g Servers -n -LL -f - -m mondas -t 1430050507 -o MODIFIED_ASOF_TIME:timeval=1430050506;RENAMED_DIRECTORIES:index_lookup=on;BACKUPTIME:lookup_range=1429877709:1430050507; -l incr -W 78 -N /boot /boot
83643:savegrp: mondas:/d/backup started
save -s orilla -g Servers -n -LL -f - -m mondas -t 1430050509 -o MODIFIED_ASOF_TIME:timeval=1430050506;RENAMED_DIRECTORIES:index_lookup=on;BACKUPTIME:lookup_range=1429877708:1430050509; -l incr -W 78 -N /d/backup /d/backup
77562:savegrp: job (832078) host: mondas savepoint: / had WARNING indication(s) at completion
90491:savegrp: mondas:/ succeeded.
83647:savegrp: Servers mondas:/ See the file /nsr/logs/sg/Servers/832078 for command output
90491:savegrp: mondas:/boot succeeded.
83647:savegrp: Servers mondas:/boot See the file /nsr/logs/sg/Servers/832080 for command output
90491:savegrp: mondas:/d/01 succeeded.
83647:savegrp: Servers mondas:/d/01 See the file /nsr/logs/sg/Servers/832079 for command output
90491:savegrp: mondas:/d/backup succeeded.
83647:savegrp: Servers mondas:/d/backup See the file /nsr/logs/sg/Servers/832081 for command output
83643:savegrp: mondas:index started
save -s orilla -S -g Servers -n -LL -f - -m orilla -V -t 1429878349 -l 9 -W 78 -N index:147f6a46-00000004-5457fce2-5457fce1-0016b3a0-02efe8cc /nsr/index/mondas
128137:savegrp: Group Servers waiting for 1 jobs (0 awaiting restart) to complete.
90491:savegrp: mondas:index succeeded.
83647:savegrp: Servers mondas:index See the file /nsr/logs/sg/Servers/832082 for command output
* mondas:All savefs mondas: succeeded.
* mondas:/ suppressed 2038 bytes of output.

You’ll see there the output reaches a point where NetWorker tells you “suppressed X bytes of output”. That’s a protection mechanism for NetWorker to prevent savegroup completion notifications growing to massive sizes. However, because we’ve used the verbose option, the output is captured – it’s just directed to the appropriate log file for the group. In this case, the output (underlined above) tells me I can check out the file /nsr/logs/sg/Servers/832078 to see the details of the root filesystem backup for the client mondas.

Checking that file, I can see what files would have been backed up:

[root@orilla Servers]# more /nsr/logs/sg/Servers/832078
96311:save: Ignoring Parallel savestreams per saveset setting due to incompatibl
e -n/-E option(s)
75146:save: Saving files modified since Sun Apr 26 22:15:06 2015

This command only works for filesystem backups performed by the core NetWorker agent. It’s not compatible for instance, with a database module or VBA – but regardless, it is the sort of debugging/analysis tool you should be aware of. (Forewarned is forearmed, and forearmed is a lot of arms… Ahem.)

Check out savegrp -n on a client/group when you have time to familiarise yourself with how it works. It’s reasonably straightforward and is a good addition to your NetWorker utility belt.

file walk

Basics – virtual machine names in VBA backups

 Basics, NetWorker, VBA  Comments Off on Basics – virtual machine names in VBA backups
Mar 262015

If you’ve been backing up your virtual machines with VBA, you’ve probably hit that moment when you’ve run an mminfo query and seen output looking like the following:


As you can see, that’s not the most effective way to see virtual machine names – vm:<id> doesn’t allow you to easily match it back to the virtual machine in question.

However, not all is lost. With VBA backups came a couple of new options. The first one is a “VBA backups” style report, using the command:

# mminfo -k

Using mminfo -k you’ll get a very tailored output focused entirely on your VBA backups, and it’ll resemble the following:


That’s a really good way of seeing a quick listing of all your VBA-based virtual machine backups, but if you’re wanting a way of reconciling in normal mminfo output, you can also make use of a new mminfo report field, vmname. For example:


(In the above command I could have used name and vmname in order to reconcile vm:<id> entries to virtual machine names, but elected not to for brevity.)

There you have it – a couple of quick and easy ways of quickly seeing details of your virtual machine backups via mminfo.


Sep 012014

A question I get asked from time to time is “How do I do X in NetWorker?”, and by how, I mean what’s the order of steps, rather than a general description.

Workflow for adding a new client

To me, the configuration steps in NetWorker are often quite minimal compared to the operational and organisational processes that typically should be followed to ensure an appropriately maintained system. Configuring a new client is a perfect example of this, so below is the procedure I normally recommend following:

  1. Determine if there are any databases or applications on the host that require module-based backups.
  2. Determine if there is anything on the host that should be excluded from backup.
  3. Determine any special retention requirements (vs ‘default’ retention requirements used in the business).
  4. Determine if any SLAs require integration between backup and other data protection processes (e.g., with snapshots, replication targets, etc.)
  5. Check OS and application versions against the compatibility guide if they’re not standard/already known versions.
  6. Ensure the backup system has sufficient capacity for bringing the client on-board.
  7. Determine what tests are to be applied to this client to confirm it’s successfully brought on-board.
  8. Determine whether any backup software to be installed will require an OS or application restart – for example:
    • NMM with GLR might require reboots (and if .Net needs to be installed, 2 reboots may be required).
    • Oracle and other databases may require restarting for library linking.
  9. Determine if any firewalls will need to be adjusted to allow for backup traffic.
  10. Confirm forward/reverse lookups between all appropriate hosts – for example:
    • New client and backup server
    • New client and storage node(s)
    • New client and IP backup storage (e.g., Data Domains)
  11. Confirm network connectivity between all appropriate hosts.
  12. File change requests or work plans as appropriate within the organisation, supplying appropriate installation/back-out plans, peripheral configuration activity (e.g., changing firewalls, etc.)
  13. Confirm change approval and schedule.
  14. Install filesystem client.
  15. Install database module (if required).
  16. Configure filesystem backups in NetWorker.
  17. Test filesystem backups in NetWorker and remediate.
  18. Configure database backups.
  19. Test database backups and remediate.
  20. Integrate client instances with appropriate retention policies and schedules.
  21. Confirm successful next-day operation of automated backups.
  22. Add client into any custom reporting (should fold automatically into standard reporting).
  23. Close off change as required.

Depending on your environment, those processes may change a bit – or they may even be less formal, but cutting corners in data protection can easily lead to a mishap, so if you’re looking for a procedure for adding a client, you could do a lot worse than the one above.

Jan 092012

Upgrading NetWorker

So a new version of NetWorker has come out, or is coming out, and it’s been decided that you’re going to upgrade, but you want a few tips for making that upgrade as painless as possible. Here’s my 5 rules for upgrading NetWorker:

  1. Read the release notes. If you’re not going to read the release notes, you are better off staying on your current version, no matter what issues you’re having. I can’t stress enough the importance of reading the release notes and having a thorough grasp of:
    • What has changed?
    • What are the known issues with the current release?
    • What were the resolved issues between the current release and the release you’re currently running?
  2. Do a bootstrap and index backup if upgrading between major or minor releases. If going between service packs on the same release, you can skip the index backup so long as your backups have been successful lately, but ensure you still do a bootstrap backup.
  3. Unload all tapes (physical or virtual) in jukeboxes before the upgrade. You’ll see why shortly.
  4. Upgrade in this order:
    • Storage node(s) on the day of the upgrade, before the NetWorker server
    • Server on the day of the upgrade, after the storage node(s)
    • Client(s) later, at suitable times
  5. After the upgrade but before the NetWorker services are restarted on the storage node(s) and server, delete the nsr/tmp directory on those hosts.

Obviously standard caveats, such as following any additional instructions in the release notes or upgrade notes should of course be followed, but sticking to the above rules as well can save a lot of hassle over time. I’ve noticed over the years that a odd, random problems following upgrades can be solved by clearing the nsr/tmp directory on the server and storage nodes. If there’s no tapes in the jukeboxes when the services first start after the upgrade, there’s less futzing for NetWorker to take care of before it’s fully up and running, too.

Basics – Specifying a default Jukebox

 Basics, NetWorker, Scripting  Comments Off on Basics – Specifying a default Jukebox
Aug 052010

If you’ve got multiple jukeboxes within a NetWorker environment, but primarily work with one of them, you may find ‘nsrjb’ to be a bit of a pain any time you forget to specify the jukebox name. If you’re not familiar with this, here’s how nsrjb reacts in this situation:

[root@tara ~]# nsrjb
1:	VTL1 	[enabled]
2:	VTL2 	[enabled]
No jukebox selected.
Please select a jukebox to use:? [1] _

(Slight aside: never assume the numbered list is the same; NetWorker doesn’t guarantee the order being the same between executions – in fact, I actually only put in an RFE about this a couple of days ago, as I’m hoping it could at least be alphabetically ordered at all times…)

If you want to avoid the jukebox-prompt from nsrjb, one of the easiest ways is to specify the jukebox name as part of the command – e.g.,

[root@tara ~]# nsrjb -j VTL1

That’s fine of course, but if the vast majority of the time you perform operations on a single jukebox, you can specify a default jukebox as an environment variable (NSR_JUKEBOX) and streamline your processes. For example, on Linux, using the bash, this might look as follows:

[root@tara ~]# export NSR_JUKEBOX=VTL1
[root@tara ~]# nsrjb
Jukebox VTL1: (Ready to accept commands)
slot  volume         pool           barcode   volume id        recyclable
1: 800840L4       ClientTesting  800840L4  3814088325       no
2: 800841L4       ClientTesting  800841L4  3797311146       no
3: 800842L4       ClientTesting  800842L4  3847642669       no
4: 800843L4       ClientTesting  800843L4  3780533937       no
5: 800844L4       ClientTesting  800844L4  3763756765       yes
6: 800845L4       ClientTesting  800845L4  3864419885       yes

Being an environment variable, this is something you can choose to set locally – say, on a per storage-node basis, when you have multiple storage nodes. It’s relatively common for instance to have a tape library on one or more storage nodes, so for the appropriate logins (or even at a system level) on each storage node it would be possible to set the local jukebox as the default, thereby streamlining usage of the units.

As an example, here’s a lab storage node with the setting in use:

[root@fawn ~]# export NSR_JUKEBOX="rd=fawn:VTL3"
[root@fawn ~]# nsrjb -s tara
Jukebox rd=fawn:VTL3: (Ready to accept commands)

For something that can take you less than 30 seconds to set, the environment variable NSR_JUKEBOX can certainly be a big time saver if you have multiple jukeboxes in your environment and (like me) you’re a command line junkie.

Basics – jbverify

 Basics  Comments Off on Basics – jbverify
Apr 082009

Now, some might say that I’m not the smartest card in the deck for what I’m about to note, but sometimes I don’t notice new commands when they appear in NetWorker, particularly if I skimmed through release notes.

I was pleasantly surprised today to find that a new command had slipped in at some point called “jbverify”. I can immediately see it however entering my stable of must-use commands, particularly in a support environment.

To quote the man page, jbverify:

[V]erifies the devices defined in the NetWorker database, making sure that each one of them is configured properly by checking them  for accessibility  and  usability.

This is the sort of diagnostic tool that support people live for, and sites suddenly experiencing strange jukebox issues should think of as a matter of course.

When run on my lab server this afternoon, I got the following badly formatted but still very useful output:

# jbverify
 Jbverify is running on host nox, Linux 2.6.18-128.1.6.el5

 Processing stand-alone devices...

 Processing /d/nsr/02/_AF_readonly
 Finished processing /d/nsr/02/_AF_readonly

 Processing /d/nsr/idata/backup/_AF_readonly
 Finished processing /d/nsr/idata/backup/_AF_readonly

 Processing /d/nsr/01/_AF_readonly
 Finished processing /d/nsr/01/_AF_readonly

 Processing /d/nsr/idata/backup
 Finished processing /d/nsr/idata/backup

 Processing /d/nsr/idata/clone/_AF_readonly
 Finished processing /d/nsr/idata/clone/_AF_readonly

 Processing /d/nsr/01
 Finished processing /d/nsr/01

 Processing /d/nsr/03
 Finished processing /d/nsr/03

 Processing /d/nsr/03/_AF_readonly
 Finished processing /d/nsr/03/_AF_readonly

 Processing /d/nsr/02
 Finished processing /d/nsr/02

 Processing /d/nsr/idata/clone
 Finished processing /d/nsr/idata/clone

 Finished processing stand-alone devices.

 Processing jukebox devices...

 Processing jukebox LTO1_LIB:

 Testing drive 1 (/dev/nst0) of JB LTO1_LIB


 Jukebox LTO1_LIB on nox successfully processed.


 Finished processing jukebox devices.


         Summary report of jbverify
         ======= ====== == ========

Hostname   Device Handle    Blocksize  Jukebox  Drv No. Status
--------   -------------    ---------  -------  ------- ------
nox        /d/nsr/02/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/idata/backup/_AF_readonly 131072 N/A N/A Pass
nox        /d/nsr/01/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/idata/backup 131072  N/A      N/A     Pass
nox        /d/nsr/idata/clone/_AF_readonly 131072 N/A N/A Pass
nox        /d/nsr/01        131072     N/A      N/A     Pass
nox        /d/nsr/03        131072     N/A      N/A     Pass
nox        /d/nsr/03/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/02        131072     N/A      N/A     Pass
nox        /d/nsr/idata/clone 131072   N/A      N/A     Pass
nox        /dev/nst0        65536      LTO1_LIB 1       Pass


If you’ve come from NetBackup, the nature of this program is somewhat reminsicent of the robtest utility. I don’t claim EMC are special for having introduced this tool, but I do applaud that it’s there (and lament that I didn’t notice it sooner).

(One thing to note: after running jbverify, make sure you reset your jukebox.)

Mar 222009

I’m not fond of software encryption (or compression, for that matter). Particularly in a 24×7 enterprise environment, clients (i.e., production servers) have better things to be doing than doing on-the-fly software encryption or compression. In these environments, hardware encryption routers should be the product of choice for achieving totally secure backups. Such devices also have advantages in terms of key management – much more flexible, scalable and appropriate for role based data access.

That being said, in smaller environments, or environments where servers are relatively idle overnight, NetWorker’s datazone encryption can be sufficient to achieve a reasonable modicum of backup protection with minimum effort – and most importantly, cost.

To get started using NetWorker datazone encryption, you first need to assign a pass phrase. This is done in the NetWorker server properties (typically accessed within NMC):

datazone-encryptionWith the pass phrase in place, you can then configure directives within NetWorker to make use of AES 256 bit encryption. However! As soon as you turn encryption on, you lose all potential for hardware based compression for your media. Why? Quite simply, compression is about finding patterns in data and reducing all the matching patterns to a single reference point; however, encryption is all about eliminating patterns, making the data appear completely random.

Thus, if you want to still get some measure of compression, you should, when using this method, employ software based compression in your directive as well.

Thus, a base directive might look like the following:

<< / >>
+compressasm: .
+aes: .

This will apply compression first to all files encountered, then once the file has been compressed, it will be encrypted. A side benefit of this is that by compressing first, you reduce the amount of data to be encrypted*.

So long as the datazone pass phrase is stored in the server, encryption will occur, and no password will be required to recover the data. Remember, this style of encryption, using a single pass-phrase, isn’t about being able to restrict whom within the datazone can recover the data, but instead it’s about keeping the data stored on-tape (which is potentially off-site, or otherwise at higher risk of theft), from being recovered.

[Edit, 2009-08-15]

It’s been pointed out to me that you can’t compress + encrypt at the client side. Indeed, I’ve now found the part in the administration guide that explicitly says this. What is extremely disappointing about this is that NetWorker actually doesn’t warn you that it’s not going to compress + encrypt! To me, that’s a security issue.

So, for the examples above, forget about enabling client side compression as well as encryption – you can have one or the other, but not both.

* In the same way that ice-cream that’s 99% fat free, but 87% sugar is a “benefit”.

Basics – Remote Control

 Basics, Scripting  Comments Off on Basics – Remote Control
Mar 032009

I learned this technique several years ago before NetWorker supported running an nsrjb command from the backup server to manipulate a jukebox on a storage node. (Previously it had required that actual nsrjb commands, when run from the command line, be run from the owner of the jukebox.)

While you don’t need it any more to control remote jukeboxes, it can still be useful to know how to do remote control operations in NetWorker – particularly if you’re say, debugging backups on a client but you don’t have console access to that client.

Note that this only applies to commands that obey the following restrictions:

  • Name starts with “nsr” or “save”
  • Resides in the same directory as the “save” command on the client.

Thus, this isn’t about remote hijacking of a NetWorker client. Such commands can only be executed by an authorised administrator on the backup server as specified in the nsr/servers file. Before you (quite rightly) point out that it would mean that a valid NetWorker administrator could in fact hijack a client by say, doing a directed recovery out to that client of an appropriately named file, they can do that already – that’s part of the trust relationship of the NetWorker administrator anyway.

So, with all those caveats out of the way, it’s remarkably simple. Going from a Unix host, you do the following:

# export RUSER=<user>
# export RCMD=<cmd>
# nsrexec -c <client>


  • user will typically be “root” or “administrator”, depending on whether your backup server is Unix or Windows.
  • cmd will be a NetWorker command you want executed.
  • clientName is the name of the client the command is to be executed from.

For instance, say I’ve got a NetWorker server called “nox”, and a client called “asgard” that I don’t have administrative login to, but I want to simulate a backup without firing up a savegroup. To do so, I could do the following:

# export RUSER=root
# export RCMD="save -e tomorrow -b Default -LL -q -s nox /tmp"
# nsrexec -c asgard
save: /tmp  36 MB 00:00:05     38 files
completed savetime=1235278395

Admittedly this is not a technique you should need to know often, but it’s useful to know about.

* You do populate your nsr/servers file, don’t you? If you don’t, go do it NOW. I mean it, stop what you’re doing, go and fix up the nsr/servers file on every client in your environment!

Feb 162009

Ever need to adjust the browse/retention time for a saveset, but you’ve not been sure how to do so? Here’s how.

To change the browse or retention time, you’ll need to find out the saveset ID (SSID) of the given saveset. This can be done with mminfo.

For instance, say you had a backup done last night of a machine called ‘archon’ that has now been rebuilt, but you want to keep the old backup for much longer than normal – e.g., ten years instead of the normal 3.

First, to find out what you need to change, get a list of the SSIDs:

# mminfo -q "client=archon,savetime>=24 hours ago" -r name,ssid
 name                          ssid
/                              4036558666
/Volumes/TARDIS/Yojimbo        4019781450
/Volumes/Yu                    4003004234

(If you’re confused about that savetime command, see my other post here.)

Now, for each of those SSIDs that are returned, we’ll run a nsrmm command to adjust the browse and retention time*.

The basic nsrmm command for adjusting the browse and retention time is:

# nsrmm -S ssid -w browse -e retent

or, for a single instance of a saveset:

# nsrmm -S ssid/cloneid -w browse -e retent

Where the ‘browse’ and ‘retent’ values can be either one of the two following:

  • A literal date in US date format ** – e.g., “12/31/2019” for 31 December 2019.
  • A ‘fuzzy’ english worded date – e.g., “+10 years” for 10 years from today.

Note that (rather obviously), your browse time cannot exceed your retention time, and generally its recommended that you set browse time to retention time.

So in this case, you’d run for each SSID or SSID/CloneID you want to affect:

# nsrmm -S ssid -w "+10 years" -e "+10 years"

Which will look like the following, based on my mminfo output:

# nsrmm -S 4036558666 -w "+10 years" -e "+10 years"
# nsrmm -S 4019781450 -w "+10 years" -e "+10 years"
# nsrmm -S 4003004234 -w "+10 years" -e "+10 years"

It’s that simple.

* You can also do this against an instance of a saveset by using the SSID/Clone ID; to do that variant, request “-r name,ssid,cloneid”, then use the two numbers in the nsrmm command separated by a forward slash – ssid/cloneid.

** The restriction on US date format may have eased in 7.5. I’m going to do some additional playing around with locales sometime soonish.

Feb 112009

I always like to know how to work with the command line, even on Windows. (As I say in my book, while a picture may be worth a thousand works, a GUI is not necessarily worth a thousand command line options.)

As such, when I want to stop and restart NetWorker on Windows, I find it faster on the command line than going to the services panel and scrolling through to the various processes.

Note: All of these options are documented in the NetWorker and License Manager administration manuals, but as always, there’s a difference between something that’s documented and something you can find, so I’m presenting these for those who are pressed for time.

There are lengthy names for each of the NetWorker client service, NetWorker server service, and NetWorker Management Console server service which you may think would make it inconvenient to stop and restart from the command line, but thankfully there’s shortcut names as well. These are:

  • NetWorker Remote Exec Service: nsrexecd
  • NetWorker Backup and Recover Server: nsrd
  • EMC GST Service: gstd
  • NetWorker License Manager: lgtolmd

Now, if you’re wanting to quickly stop everything, you can rely on process dependencies and run:

C:> net stop nsrexecd /y

That will shutdown everything, since everything relies on the nsrexecd (client) process in order to run.

If you want to start NetWorker on a client, all you need to run is:

C:> net start nsrexecd

If you’re wanting to start NetWorker on a server though, you should run:

C:> net start nsrd

There’s a simple reason for this – nsrd relies on nsrexecd, so Windows dependency checking will start nsrexecd first for you.

Similarly, if you want to start the management console on a Windows machine, you’d run:

C:> net start gstd

And if you’re running Legato License Manager, you’d run:

C:> net start lgtolmd