Windows block based backup with Linux AFTDs

 Basics, Linux, NetWorker, Recovery, Windows  Comments Off on Windows block based backup with Linux AFTDs
Jul 062014

One of the new features of NetWorker 8.2 is the expansion of Windows Block Based Backups (BBB) to support additional backup targets. When the feature was originally introduced into NetWorker 8.1, it supported only the following devices:

  • Data Domain Boost, and
  • Advanced File Type Devices (AFTDs) on Windows systems only.

However, there’s a lot of environments out there that can’t necessarily position a Windows storage node for such backups if Boost isn’t available, and so the logical extension to the solution was to support backing up to AFTDs on Unix and Linux systems, too.

That’s what has been added in 8.2. If you’re using Data Domain, you’ll almost certainly want to do these backups to Data Domain Boost devices, of course. However, if you don’t have Data Domain, then the option of backing up to any AFTD makes Windows BBB much more attractive.

The setup is surprisingly straight forward, but you will need to install and configure Samba on your Linux or Unix host in order to be able to present the AFTD as a CIFS share to the Windows host.

On my Linux lab server, I have several AFTDs – 2 x 150GB devices and 2 x 50GB devices. For the purposes of the setup, I decided to configure the 2 larger AFTDs for CIFS based BBB backups for a Windows 2012 host. The Samba configuration looks like the following:

Samba Device Configuration

That provides two Samba shares, one per device. The device at path /d/03 is shared as d_03, and the device at path /d/04 is shared as d_04.

To enable successful sharing, I used smbpasswd to first add, then enable the root user:

# smbpasswd -a root

Followed by:

# smbpasswd -e root

(I picked a reasonably secure password for the root user for Samba which was unrelated to the actual root user account.)

Next, it becomes necessary to edit the device access information for each device:

Device list

You’ll need to pick the devices out of the device list that match the paths you’ve configured for Samba access – in my case, they’re the devices ‘BIG-01’ and ‘BIG-02’. Editing the device properties for BIG-01:

Device access information

In all cases, make sure the owner storage node’s path is listed first in the device access information. In this case, that’s /d/03 for the Linux server itself. The CIFS path to the device is listed for Windows access. Note that using a drive mapping isn’t recommended (and in fact is usually quite painful to configure). So in this case, the CIFS share for /d/03 iis \\tara\d_03, and is listed second.

In addition to specifying the device access information, it’s important you specify the remote username and password that the NetWorker client software will use when accessing the CIFS share from the client. That’s done in the Configuration tab:

Remote user and password for the devices

With those settings in place, it’s time for the client configuration. This is actually very straight forward:

Enabling BBB on a client

In actual fact, it’s just a simple case of checking the Block Based Backup checkbox on the main configuration tab of the client. Well, almost. This is a lab environment so that’s all I had to do. There are some considerations in a production environment for BBB, however. For instance, the C:\ drive on a Windows system can get block based backups, but incremental backups will fail – the system is designed, after all, to be used on larger filesystems in specific scenarios (e.g., highly dense filesystems) rather than for every filesystem.

Once the backup has been kicked off, you’ll get reasonably good performance since you’re not working with the client filesystem. For example, even in my lab environment:

BBB Save

Once completed, the BBB save looks reasonably similar to a standard backup, viz:

BBB Savegroup Results

You’ll note one key exception of course – the BBB is reported as having a file count of 1, since it didn’t actually traverse the filesystem.

Recovery is a very straight-forward process via the NMC Recovery interface. First, select the client you’ll be recovering from, and having done so, choose the option to do a recovery from a Block Based Backup:

BBB Recovery Step 1

Clicking Next will allow you to select what you want to recover: file level, or image level. If file level, you can choose which files you want to recover from:

BBB Recovery Step 2

Having selected the data to recover, going to the Recovery Options allows you to choose to recover the data in place, or to a new directory:

BBB Recovery, Step 3

Next, you get to confirm what you’ll be doing and decide when the recovery will be run:

BBB Recovery, Step 4

Once you’ve named the recovery, you can click the “Run Recover” button (not shown above) to initiate the recovery. The results should be similar to the following:

BBB Recovery, Step 5

At the completion of the recovery, you can check the client to confirm the files have come back, but that’s about all there is to it.






 Posted by at 12:38 pm
Mar 132011

I have to say, I’m really liking NetWorker’s ability to work with persistent binding on tape libraries now.

If you’re not aware of persistent binding, it exists to resolve a problem whereby on some platforms (such as Windows and Linux), device re-ordering can happen across reboots. For a long time there’s been ways of preventing this from being an issue for filesystems/LUNs – for instance, on Linux most filesystem types support mounting a filesystem via a unique label or UUID. For example, this is a typical entry from /etc/fstab on a CentOS install:

LABEL=/boot     /boot       ext3    defaults        1 2

This allows the OS to mount the /boot filesystem regardless of whether it’s on /dev/sda, /dev/sdb, /dev/whatever.

Persistent binding allows us to do the same thing with tape as the above does with disk. The advantage of this is obvious: when NetWorker configures a tape library, it maps element order to device paths –

[root@linuxvtl ~]# sjisn 3.0.0
Serial Number data for 3.0.0 (SPECTRA  PYTHON):
		Serial Number: XYZZY_A   
		SCSI-3 Device Identifiers:
	Drive at element address 1:
		SCSI-3 Device Identifiers:
			ATNN=IBM     ULT3580-TD1  XYZZY_A1  
	Drive at element address 2:
		SCSI-3 Device Identifiers:
			ATNN=IBM     ULT3580-TD1  XYZZY_A2  
	Drive at element address 3:
		SCSI-3 Device Identifiers:
			ATNN=IBM     ULT3580-TD1  XYZZY_A3  
	Drive at element address 4:
		SCSI-3 Device Identifiers:
			ATNN=IBM     ULT3580-TD1  XYZZY_A4

If you can’t see the mappings there, don’t worry – we can see that via inquire – for example:

scsidev@3.1.0:IBM   ULT3580-TD1   550V|Tape, /dev/nst0
	S/N:	XYZZY_A1  
	ATNN=IBM     ULT3580-TD1     XYZZY_A1  

That information works well for conventional situations where there’s no risk of a device re-ordering.

When there’s a risk of re-ordering however, the above style of configuration doesn’t work – or if it does, it doesn’t work across reboots. To avoid it in its entirety, we instead do a configuration that uses more exact identification details. Typically, this is in a fibre-channel scenario, and that means WWNs.

We can access device details referencing WWNs via the persistent binding mode – inquire -p, and jbconfig -p.

Let’s look first at inquire -p:

[root@linuxvtl ~]# inquire -p
scsidev@3.0.0:SPECTRA PYTHON    |Autochanger (Jukebox),
		S/N:	XYZZY_A   
scsidev@3.1.0:IBM   ULT3580-TD1 550V|Tape,
		S/N:	XYZZY_A1  
		ATNN=IBM     ULT3580-TD1     XYZZY_A1  

If we run jbconfig -p, the output and run-scenario looks a little different, because it’s referencing WWN-based paths rather than standard /dev/nst* paths:

[root@linuxvtl ~]# jbconfig -p

Jbconfig is running on host linuxvtl (Linux 2.6.18-128.el5),
  and is using linuxvtl as the NetWorker server.

	 1) Configure an AlphaStor Library.
	 2) Configure an Autodetected SCSI Jukebox.
	 3) Configure an Autodetected NDMP SCSI Jukebox.
	 4) Configure an SJI Jukebox.
	 5) Configure an STL Silo.

What kind of Jukebox are you configuring? [1] 2
14484:jbconfig: Scanning SCSI buses; this may take a
while ... 
These are the SCSI Jukeboxes currently attached to your
  1) 350223344ab000000: Spectralogic
  2) 350223344ab000800: Spectralogic
Which one do you want to install? 1
Installing 'Spectralogic' jukebox - 350223344ab000000.

What name do you want to assign to this jukebox device? VTL1
15814:jbconfig: Attempting to detect serial numbers on the
jukebox and drives ...

15815:jbconfig: Will try to use SCSI information returned by
jukebox to configure drives.

Turn NetWorker auto-cleaning on (yes / no) [yes]? no

The following drive(s) can be auto-configured in this
 1> LTO Ultrium @ 3.1.0 ==>
 2> LTO Ultrium @ 3.2.0 ==>
 3> LTO Ultrium @ 3.3.0 ==>
 4> LTO Ultrium @ 3.4.0 ==>
These are all the drives that this jukebox has reported.

To change the drive model(s) or configure them as
shared or NDMP drives, 
 you need to bypass auto-configure. Bypass
auto-configure? (yes / no) [no] no

Jukebox has been added successfully

Once a library has been configured with persistent binding, the device access paths logically become different. On Windows, you’ll get device path names of \.TapeX where X starts at something along the lines of 2^31-X; on Linux, the paths will vary depending on the install – for instance, CentOS may give a different result than Oracle Unbreakable Linux, etc. The device paths on Linux as well will explicitly reference the WWNs:

[root@linuxvtl ~]# nsrjb -v
drive 1 (/dev/tape/by-id/scsi-350223344ab000100-nst) slot :   
drive 2 (/dev/tape/by-id/scsi-350223344ab000200-nst) slot :   
drive 3 (/dev/tape/by-id/scsi-350223344ab000300-nst) slot :   
drive 4 (/dev/tape/by-id/scsi-350223344ab000400-nst) slot :
While this makes referencing individual tape drives a little more fiddly, it has the distinct advantage that across multiple reboots, the library remains fully operable and all devices accessible – a very, very small price to pay. There is, indeed, virtue in persistence. If you want to read more about NetWorker and persistent binding, check out the whitepaper about it available on PowerLink.
Feb 222011

I’ve been involved with an increasing number of NetWorker 7.6 SP1 configurations on Windows 2008 R2, and I’m not sure whether what I’ve encountered is specific to Windows 2008 R2 or just a general deficiency in the NetWorker installer’s firewall configuration process. Either way, since it caused some challenges for me, I wanted to note down the issues I’ve observed.

First, the firewall configuration is only applied to the “Public” profile. This is OK for single-interface servers, but if your system has multiple interfaces, it isn’t sufficient – you need to edit the rules to apply to all three of “Domain”, “Private” and “Public”:

Firewall configuration 1

The next issues encountered were relating to tape libraries on storage nodes. In particular, it appeared that the default automatic NetWorker firewall configuration on at least Windows 2008 R2 didn’t add support for the nsrmmgd or nsrlcpd daemons to communicate.

To create these rules:

  • On the server:
    • Copied two of the existing rules – one for TCP, one for UDP – and updated the “Programs and Services” pane to reference X:pathtobinnsrmmgd.exe.
  • On each storage node:
    • Copied two of the existing rules – one for TCP, one for UDP – and updated the “Programs and Services” pane to reference X:pathtobinnsrlcpd.exe.

With these sets of changes in play, NetWorker has behaved a lot more normally.

(Obviously, any firewall changes you make in your environment should be considered against site requirements.)

Saveset sizes from 32-bit Windows

 Features, NetWorker, Windows  Comments Off on Saveset sizes from 32-bit Windows
Jan 172010

There’s currently a bug within NetWorker whereby if you’re using a 32-bit Windows client that has a filesystem large enough such that the savesets generated are larger than 2TB, you’ll get a massively truncated size reported in the savegroup completion. In fact, for a 2,510 GB saveset, the savegroup completion report will look like this:

Start time:   Sat Nov 14 17:42:52 2009
End time:     Sun Nov 15 06:58:57 2009

--- Successful Save Sets ---
* cyclops:Probe savefs cyclops: succeeded.
* cyclops:C:bigasms 66135:(pid 3308): NSR directive file (C:bigasmsnsr.dir) parsed
 cyclops: C:bigasms               level=full,   1742 MB 13:15:56    255 files
 trash.pmdg.lab: index:cyclops     level=full,     31 KB 00:00:00      7 files
 trash.pmdg.lab: bootstrap         level=full,    213 KB 00:00:00    198 files

However, when checked through NMC, nsrwatch or mminfo, you’ll find that that the correct size for the saveset is actually shown:

[root@trash ~]# mminfo
 volume        client       date      size   level  name
XFS.002        cyclops   11/14/2009 2510 GB   full  C:bigasms
XFS.002.RO     cyclops   11/14/2009 2510 GB   full  C:bigasms

The reporting doesn’t affect recoverability, but if you’re reviewing savegroup completion reports the data sizes will likely (a) be a cause for concern or (b) affect any auto parsing that you’re doing of the savegroup completion report.

I’ve managed to secure a fix for 7.4.4 for this, with requests in to get it ported to 7.5.1 as well, and to get it integrated into the main trees for permanent inclusion upon the next service packs, etc. If you’ve been putting up with this problem for a while or have just noticed it and want it fixed, the escalation patch number was NW110493.

(It’s possible that this problem affects more than just 32-bit Windows clients – i.e,. it could affect other 32-bit clients as well. I’d be interested in knowing if someone has spotted it on another operating system. I’d test, but my lab environment is currently otherwise occupied and generating 2+TB of data, even at 90MB/s, is a wee bit long.)

Jan 122010

Over at Backup Central, Curtis Preston has written a couple of excellent blog posts to do with VSS.

The first, What is Windows VSS and why should you care? is an excellent overview of how the VSS process works within Windows. Even if you’ve been using VSS within your environment, if you’re not quite sure how it works, this is a great piece to read.

The second delves into issues relating to VMware VCB’s (in)ability to perform consistent application backups – i.e., via VSS for say, an Exchange or Microsoft SQL guest. Titled Hyper-V ahead of VMware in the Backup Race, it’s a justifiable kick in the pants to VMware, and a pointed warning regarding VMware/VCB backups of applications.

(These two articles, Curtis mentions, came about from some posts by Scott Waterhouse on The Backup Blog, which talked about vSphere backups.)

Aug 052009

There’s an old saying, “Be careful what you wish for; you may get it.”

Many would say that’s an apt description of the NetWorker software distribution functionality. When it first appeared, it was inadequately documented and error prone. Fast forward to now, and it’s less error prone but still reasonably inadequately documented.

I would suggest that unless you’re currently running NetWorker 7.4.4 or NetWorker 7.5.1 (as of time of writing), you will probably find the software distribution functionality too problematic to work with. Any 7.4.x version higher than 7.4.4, and any version of NetWorker higher than 7.5.1 should continue to show improvements. Versions lower than these in either the 7.4.x tree or 7.5 tree should generally be avoided.

A few of the known problems with the software distribution functionality include:

  • Works very poorly in a heterogenous environment, where heterogenous means operating systems outside of what can be updated.
  • Works very poorly when large numbers of clients are included in an operation at once.
  • Gives inadequate progress updates during GUI operations.
  • Can fail in an undesirable state on clients, leaving NetWorker either uninstalled or shutdown.

That being said, you might think that it’s somewhat of an oxymoron for the title of this post to be “Using …”, but I do think it’s now at the point where it can be used, but just with more caution than you might otherwise expect.

(NB: For simplicity here on in, I’m going to call the “NetWorker Software Distribution Functionality” the “NetWorker repository”, since that’s the term for the storage area for the distribution system, and less cumbersome to type or read!)

Here’s some key tips on using the repository successfully:

  1. Do not use NMC. Forget about it. Maybe in another v7.x release, or a v8 release, NMC might catch up.
  2. Do not target multiple operating systems in any activity.
  3. Do not run software updates in parallel.

You might wonder then, what’s the point? If you can’t use the GUI, and you can’t operate against the entire environment at once, or indeed against more than one client at once, where is it better than manual operations?

If you’re prepared to spend the time to get the repository working, it’s better than entirely manual operations because of the following factors:

  1. Centralised storage of NetWorker software.
  2. Push updates are still very worthwhile, particularly in environments where you may not always have easy access to the clients of the datazone.
  3. Activities can be scripted with minimum effort, allowing for centralised yet granular operations.

Despite the limitations described previously, there are still some key activities that you can do with the repository. These are:

  1. Inventory the existing software install state of clients.
  2. Query the current state of repository software.
  3. Add software to the repository.
  4. Remove software from the repository.
  5. Update client software.

Here’s some examples of how to go about doing these operations, from the command line.

Inventorying clients

When inventorying clients, ensure that you don’t include in the client inventory list unsupported operating systems. For the average environment, this particularly applies to Mac OS X clients, NetWare clients and OpenVMS clients. The inventory operation has a long history of failure when these clients are thrown into the mix.

To inventory clients, you use one of two commands:

# nsrpush -i clients


# nsrpush -i -all

However, as I’ve pointed out, doing operations against all clients isn’t the best idea, so the second option there is really only presented for completeness.

Here’s an example of inventorying 3 clients, ‘nox’, ‘nimrod’ and ‘asgard’:

[root@nox ~]# nsrpush -i asgard nox nimrod
Starting Inventory Operation on selected clients
Creating client type job for client asgard.
Copying inventory scripts to client asgard.
Successfully copied inventory scripts to client asgard.
Starting inventory of client asgard.
Successfully ran inventory scripts on client asgard.
Cleaning up client asgard.
Successfully cleaned up client asgard.
Creating client type job for client nox.
Copying inventory scripts to client nox.
Successfully copied inventory scripts to client nox.
Starting inventory of client nox.
Successfully ran inventory scripts on client nox.
Cleaning up client nox.
Successfully cleaned up client nox.
Creating client type job for client nimrod.
Copying inventory scripts to client nimrod.
Successfully copied inventory scripts to client nimrod.
Starting inventory of client nimrod.
Successfully ran inventory scripts on client nimrod.
Cleaning up client nimrod.
Successfully cleaned up client nimrod.

Inventory status: succeeded

Perhaps one of the sillier parts of inventorying is that while it updates the database of client versions, it actually doesn’t report those versions to you in the same output – you have to run another command for that.

To show the versions, you want to invoke one of:

# nsrpush -s -all


# nsrpush -s clients

Now, in this case, the “show” option only runs against the local database, so it’s reasonably safe to use the ‘-all’ option. To reduce the amount of output though, I’ll show the output for just those three clients inventoried before:

[root@nox ~]# nsrpush -s asgard nimrod nox

 nox  linux_x86_64
     NetWorker 7.5.1
                Man Pages                      
                Storage Node                   

 nimrod  linux_x86
     NetWorker 7.5.1
                Management Console             
                Man Pages                      

 asgard  linux_x86
     NetWorker 7.4.2
                Man Pages                      

As you can see in the above, the output is reasonably comprehensive, covering for each host, all three of:

  • Version
  • Platform
  • Packages

Note that you can’t show before you inventory, so don’t start by trying to show the state of all systems – it just won’t work.

Querying software in the repository

To get a list of the software currently held in the repository, you’d run the command:

# nsrpush -l

For instance, here’s the (abridged) output from one of my lab servers:

[root@nox ~]# nsrpush -l

Products in the repository

   NetWorker 7.5.1
         Korean Language Pack           
         Chinese Language Pack          
         Japanese Language Pack         
         French Language Pack           
         License Manager                
         Storage Node                   
         Man Pages                      
         Management Console             
         Chinese Language Pack          
         Japanese Language Pack         
         French Language Pack           
         License Manager                
         Storage Node                   
         Man Pages                      
         Management Console             
         Korean Language Pack
   NetWorker 7.5.0
         Management Console             
         English Language Pack          
         License Manager                
         Storage Node                   
         Management Console             
         English Language Pack          
         Language Packs                 
         License Manager                
         Storage Node                   

Note that both Windows and Unix versions of NetWorker are stored in the same repository. (Due to restrictions with copy/paste in WordPress not all spacing has been preserved in the above.)

Adding software to the repository

Adding software to the repository is reasonably straight forward. There are two different techniques for this – one for adding the software designated as the same platform of the backup server, and one for adding software designated as the different platform of the backup server.

By “same” I mean:

  • Adding Unix/Linux software to a Unix/Linux backup server
  • Adding Windows software to a Windows backup server

By “different” I mean:

  • Adding Windows software to a Unix backup server
  • Adding Unix software to a Windows backup server

Adding software from the same platform

Let’s say I’ve got a Unix NetWorker server, and I want to add software from a supported Unix platform to the repository. Here’s the steps:

  1. Download the package from EMC PowerLink.
  2. Uncompress and untar the downloaded package into a directory.
  3. Run nsrpush against that directory with the appropriate arguments.

So, let’s assume that I want to add NetWorker 7.4.4 for Linux 32-bit to a software repository. I’d start with getting the software into a known directory.

[root@nox tmp]# mkdir /tmp/repo
[root@nox tmp]# cd repo
[root@nox repo]# gunzip -c /d/03/share-a/Downloads/NetWorker/v7_4/
NetWorker 7.4 SP4/nw74sp4_linux_x86.tar.gz | tar xvpf -

Now, the command we want to use in this case is:

# nsrpush -a -p product -v version -P platform -m path -U

(NB: The “-U” is for “Unix”; obviously if you’re running a like-for-like installation on Windows, using “-U” here isn’t going to be a good idea – instead, use “-W”.)

Now, to work out what we need to put into the product, version and platform strings, we want to check the LGTO_METAFILE.linuxx86 file that was included in the distribution. If you look at the start of the file, you’ll find a section detailing this, and for the file specified above on my lab server, this section looks like the following:

# Client Push metafile to be included in linux_x86 NetWorker distributions





DESCRIPTION=Networker 7.4.4

So, that means our nsrpush command will look like this:

# nsrpush -a -p NetWorker -v 7.4.4 -P linux_x86 -m /tmp/repo -U
Success adding product from:

Add to repository status: succeeded

That’s all there is to it! That software is now in the repository and ready to be rolled out to a client.

Adding software from the other platform

If you need to add Windows software to a Unix server, or Unix software to a Windows server, you’ll need a helper client. This is a client of the same platform as the software you are adding, with NetWorker already running, and a copy of the software extracted on the client. The procedure works along the lines of the following:

  1. Unpack the alternate platform software onto a ‘like’ client.
  2. Unpack the alternate platform software onto the backup server.
  3. Run nsrpush to add the software.

So in this case, I’m going to use a Windows client called ‘medusa’ as a helper/proxy to allow me to add NetWorker 7.5.1 64-bit for Windows to a Unix server’s repository. On my Windows client, I’ve got a directory called “C:temp751”:

Adding alternate platform software - folder where software can be found on client

Adding alternate platform software - folder where software can be found on client

Now, I also need the software unpacked on the backup server, so I flush out the contents of my /tmp/repo directory, and unzip the Windows 64-bit NetWorker 7.5.1 archive. This gives me a path to the LGTO_METAFILE.winx64 file of:


Now, I need to add – the command will be:

# nsrpush -a -p product -v version -P platform -m path -c client -C path -W


  • Product, version and platform will be extracted from the LGTO_METAFILE as per usual.
  • -m path will refer to the local copy, on the backup server, of the software.
  • -c client will refer to the helper/proxy client (in this case, ‘medusa’),
  • -C path will refer to the path on the helper/proxy client where the software has also been extracted on
  • -W tells NetWorker we need to add Windows software.

The relevant product/version/platform details of the LGTO_METAFILE.winx64 looks like the following:

# Product Definition

DESCRIPTION=Networker 7.5.1

So, our command will therefore be:

[root@nox repo]# nsrpush -a -p NetWorker -P win_x64 -v 7.5.1 -m /tmp/repo
-c medusa -C 'C:temp751' -W
Hostname and mount point recorded.
Success adding product from:

Add to repository status: succeeded

We can subsequently confirm successful addition to the repository with our trusty nsrpush -l command:

[root@nox repo]# nsrpush -l

Products in the repository

   NetWorker 7.5.1
                 Management Console             
                 Chinese Language Pack          
                 Korean Language Pack           
                 Japanese Language Pack         
                 French Language Pack           
                 English Language Pack          
                 Language Packs                 
                 License Manager                
                 Storage Node

There you have it – software for an alternate platform added to the repository, with very little work.

Updating client software

Finally, there wouldn’t be much use for the repository without being able to push out updates.

The upgrade command is reasonably straight forward:

# nsrpush -u -p product -v version clients

Where ‘clients’ is a space separated list of one or more clients. (Note that while you can in theory use the ‘-all’ option for all clients, I’d really, really strongly recommend against it.)

Now, our client ‘asgard’ from before was still running NetWorker 7.4.2 for Linux, and I’d like to upgrade it to 7.4.4. The command and upgrade process is as follows:

[root@nox tmp]# nsrpush -u -p NetWorker -v 7.4.4 asgard
Starting upgrade operation on selected clients.
Creating client type job for client asgard.
Copying upgrade scripts to client asgard.
Successfully copied upgrade scripts to client asgard.
Checking tmp space on client asgard.
Successfully issued a tmp space check command to client asgard.
Copying upgrade packages to client asgard.
Successfully copied upgrade packages to client asgard.
Starting upgrade of client asgard.
Successfully issued an upgrade command to client asgard.
Waiting for upgrade status for client asgard.
Still waiting for the upgrade to complete on client asgard.
Waiting for upgrade status for client asgard.
Successfully retrieved and processed the package upgrade status data for client asgard
Starting cleanup phase on client asgard.
Cleaning up client asgard.
Successfully issued the cleanup command to client asgard.

Upgrade status: succeeded

Pushing out software to alternate platforms (i.e., Windows from Unix or vice versa) is exactly the same command as above.

Cautions for upgrades

When you run upgrades, here’s what I’d suggest should be your cautions:

  1. Always capture the output and confirm you get an “Upgrade status: succeeded” message for each client that you upgrade.
  2. Don’t run the command against more than say, 5 clients at once.
  3. Don’t run the command for Windows and Unix clients at the same time.
  4. If the upgrade fails, confirm from the output whether NetWorker was able to restart on the client before attempting another push. If it doesn’t look like it did, you may have to install manually.

Many errors that can come out of nsrpush are currently undocumented. Thus, if you run an nsrpush operation and get an error that you can’t find in PowerLink from a casual search, you probably need to log a case with EMC. (For instance, in tests today I got an error “client X has products installed that prevent upgrade”. There’s no publically visible documentation on this error, and despite the seeming logic of it, after checking the client, I don’t understand why the error came about.)

[Edit – 2009-08-19 – While reviewing the 7.4.5 release notes I stumbled across the reason for the “client X has products installed that prevent upgrade” error; while I had been aware that you couldn’t use the repository to update hosts with the NetWorker server software installed, I wasn’t aware that you also can’t use it for hosts with NMC or Legato License Manager installed. The client I experienced an issue on was running LLM.]

Why would you use the repository?

It’s worth revisiting this – obviously I’ve couched all of the above in quite a few caveats and thrown in some curly comments (“many errors … are currently undocumented”). With this in mind, you may be wondering now that you’ve read to the bottom of the post why you would want to risk using the repository and running into problems.

To me the most obvious reason is that the software repository is going to continue to improve. For instance, between 7.4.4 and 7.5.1 many known issues were sorted out, making 7.5.1 much more stable to use. It’s already, say, useful now, if you’re careful with it, and thus keeping abreast of it is going to allow you to continue to save time as each successive version comes out. It is already in a position to save you time and to help you keep clients (and modules) up to date – so this in itself, I believe, is reason enough to use it.

[Edit 2009-08-21 Erroneous information left from a draft removed. Thanks Rolf for pointing this out.]

Finally, even as the GUI continues to improve, I have been and always will be a fan of command line functionality because it allows for scripting. So even when there are limitations with what you can do in NMC, using nsrpush on the command line isn’t rocket science once you get used to the syntax.

Much as there are parts of the repository that I find a bit curmudgeonly at the moment, I’m genuinely happy with where it has (finally) got to, and I’m equally genuinely looking forward to increased functionality and stability as newer versions of NetWorker come out. Over time it’s going to evolve from something which is a “partial time saver” to something that’s a real time saver.

Does NetWorker scan the indices for changed files when it does a backup?

 Basics, NetWorker, Windows  Comments Off on Does NetWorker scan the indices for changed files when it does a backup?
May 212009

This is a fairly common question to see asked – does NetWorker, when a non-full backup is run, scan the existing client indices to determine what files have changed from previous backups?

The short answer is: no.

The more in-depth answer is that NetWorker will use one of a few different mechanisms for determining what files should be backed up in a non-full backup scenario, and none of those mechanisms involve scanning the client indices. These mechanisms are:

  • Check for files that have changed since a certain date. Whenever a non-full backup is run, the NetWorker server includes in the backup command the last savetime. Thus, all changed files can be quickly calculated from this.
  • Check for changes according to the change journal (Windows only).
  • Check for changes based on the archive bit (Windows only).

Personally, I really dislike the use of the archive bit. Too many programmers on Windows take liberty with this odious little setting, and it’s become so bastardised and unreliable that my very firm recommendation is you follow the instructions in the NetWorker administration guide to turn off use of the archive bit in incremental backups. (Hint: search for NSR_AVOID_ARCHIVE*).

So, there’s 3 ways that NetWorker can be expected to use to determine what files should be backed up in a non-full backup – and none of those mechanisms are achieved through an index scan.

* [Updated 2009-06-18]

Expanding on this more fully – on the backup server itself, establish an environment variable called NSR_AVOID_ARCHIVE and set it to any value other than “No”. I prefer to set it to “YES” or 1 so it’s entirely clear what the desired result is.

On Unix, places to set this is in the /etc/profile or the NetWorker startup script; however, the problem with setting it in the NetWorker startup script is that you have to remember to re-create that setting every time you upgrade NetWorker, since the startup script is fully replaced each time.

In Windows, set it as a system environment variable under the properties for the system itself. These variables are established before programs are started, meaning that NetWorker will be aware of them when it starts.

May 042009

While researching an issue today I was reminded of a problem I had a few years ago that still remains quite valid.

Client operating systems must not be changed. To be more specific, client platforms must not be changed – at least while keeping the same client name.

The reason for this is that index data is incompatible between client platforms. That is, if you’ve been backing up the machine ‘cerberus’ which is a Linux machine, then the machine is rebuilt as a Windows machine, it must be renamed.

Failing to rename a client in these circumstances can result in index corruption. This may manifest in one of three different ways:

  1. Inability to backup at all.
  2. Backups may consistently fail at specific points.
  3. Backups may succeed but recoveries may fail.

Now, there may be some workarounds, and I’ll profess to having tried some of them out in lab environments, but let me be blunt: this causes so many problems (and is so unrecommended by EMC) that changing a client operating system (platform) should never be done without the client being renamed as well.

This may be done by either:

  • Creating a new client name entirely for the rebuilt machine


  • Using the standard procedure for first renaming the machine in NetWorker before the rebuild, so that when the new operating system is configured in NetWorker with the ‘previous’ name, there is no namespace clash or client ID similarities.

Trust me, if you don’t do one of the above when a client operating system is changed, at some point you will come a cropper.


I thought I’d come back and add output from a backup attempt to show what happens when you do this. Here’s the scenario – I created a client called ‘dopey’*, installed CentOS 5, ran a full + incremental backup, then reinstalled ‘dopey’ as a Windows 2003 machine. Here’s (some of) the output from that backup, using NetWorker 7.5.1 as the server and client in all 3 instances:

[root@nox ~]# savegrp -l full -c dopey idata
40473:savegrp: command ' save -s -g idata -LL -m dopey -l full -q -W 78 -N "VSS ASR DISK:\" "VSS ASR DISK:\"' for client dopey exited with return code 5.
7336:savegrp: Log file /nsr/tmp/sg/idata/sso.dopey.pSMS7s is empty.
68703:savegrp: savegrp:idata * dopey:VSS ASR DISK: Cannot determine status of backup process.  Use mminfo to determine job status.

7341:savegrp: dopey:VSS ASR DISK: unexpectedly exited.
(snip)40473:savegrp: command ' save -s -g idata -LL -m dopey -l full -q -W 78 -N "C:\" "C:\"' for client dopey exited with return code 5.
7336:savegrp: Log file /nsr/tmp/sg/idata/sso.dopey.BlOmrS is empty.
68703:savegrp: savegrp:idata * dopey:C: Cannot determine status of backup process.  Use mminfo to determine job status.

7341:savegrp: dopey:C: unexpectedly exited.

As you can see, we’re just not getting a successful backup at all, since NetWorker is unable to merge the index entries from the two incompatible platforms.

* Currently running through and naming test machines based on the Seven Dwarves from Snow White, that’s all.