Using the NetWorker Software Distribution Functionality

There’s an old saying, “Be careful what you wish for; you may get it.”

Many would say that’s an apt description of the NetWorker software distribution functionality. When it first appeared, it was inadequately documented and error prone. Fast forward to now, and it’s less error prone but still reasonably inadequately documented.

I would suggest that unless you’re currently running NetWorker 7.4.4 or NetWorker 7.5.1 (as of time of writing), you will probably find the software distribution functionality too problematic to work with. Any 7.4.x version higher than 7.4.4, and any version of NetWorker higher than 7.5.1 should continue to show improvements. Versions lower than these in either the 7.4.x tree or 7.5 tree should generally be avoided.

A few of the known problems with the software distribution functionality include:

• Works very poorly in a heterogenous environment, where heterogenous means operating systems outside of what can be updated.
• Works very poorly when large numbers of clients are included in an operation at once.
• Gives inadequate progress updates during GUI operations.
• Can fail in an undesirable state on clients, leaving NetWorker either uninstalled or shutdown.

That being said, you might think that it’s somewhat of an oxymoron for the title of this post to be “Using …”, but I do think it’s now at the point where it can be used, but just with more caution than you might otherwise expect.

(NB: For simplicity here on in, I’m going to call the “NetWorker Software Distribution Functionality” the “NetWorker repository”, since that’s the term for the storage area for the distribution system, and less cumbersome to type or read!)

Here’s some key tips on using the repository successfully:

1. Do not use NMC. Forget about it. Maybe in another v7.x release, or a v8 release, NMC might catch up.
2. Do not target multiple operating systems in any activity.
3. Do not run software updates in parallel.

You might wonder then, what’s the point? If you can’t use the GUI, and you can’t operate against the entire environment at once, or indeed against more than one client at once, where is it better than manual operations?

If you’re prepared to spend the time to get the repository working, it’s better than entirely manual operations because of the following factors:

1. Centralised storage of NetWorker software.
2. Push updates are still very worthwhile, particularly in environments where you may not always have easy access to the clients of the datazone.
3. Activities can be scripted with minimum effort, allowing for centralised yet granular operations.

Despite the limitations described previously, there are still some key activities that you can do with the repository. These are:

1. Inventory the existing software install state of clients.
2. Query the current state of repository software.
3. Add software to the repository.
4. Remove software from the repository.
5. Update client software.

Here’s some examples of how to go about doing these operations, from the command line.

Inventorying clients

When inventorying clients, ensure that you don’t include in the client inventory list unsupported operating systems. For the average environment, this particularly applies to Mac OS X clients, NetWare clients and OpenVMS clients. The inventory operation has a long history of failure when these clients are thrown into the mix.

To inventory clients, you use one of two commands:

# nsrpush -i clients

or

# nsrpush -i -all

However, as I’ve pointed out, doing operations against all clients isn’t the best idea, so the second option there is really only presented for completeness.

Here’s an example of inventorying 3 clients, ‘nox’, ‘nimrod’ and ‘asgard’:

[root@nox ~]# nsrpush -i asgard nox nimrod
Starting Inventory Operation on selected clients
Creating client type job for client asgard.
Copying inventory scripts to client asgard.
Successfully copied inventory scripts to client asgard.
Starting inventory of client asgard.
Successfully ran inventory scripts on client asgard.
Cleaning up client asgard.
Successfully cleaned up client asgard.
Creating client type job for client nox.
Copying inventory scripts to client nox.
Successfully copied inventory scripts to client nox.
Starting inventory of client nox.
Successfully ran inventory scripts on client nox.
Cleaning up client nox.
Successfully cleaned up client nox.
Creating client type job for client nimrod.
Copying inventory scripts to client nimrod.
Successfully copied inventory scripts to client nimrod.
Starting inventory of client nimrod.
Successfully ran inventory scripts on client nimrod.
Cleaning up client nimrod.
Successfully cleaned up client nimrod.

Inventory status: succeeded

Perhaps one of the sillier parts of inventorying is that while it updates the database of client versions, it actually doesn’t report those versions to you in the same output – you have to run another command for that.

To show the versions, you want to invoke one of:

# nsrpush -s -all

or

# nsrpush -s clients

Now, in this case, the “show” option only runs against the local database, so it’s reasonably safe to use the ‘-all’ option. To reduce the amount of output though, I’ll show the output for just those three clients inventoried before:

[root@nox ~]# nsrpush -s asgard nimrod nox

 nox  linux_x86_64
     NetWorker 7.5.1
                Client                         
                Man Pages                      
                Server                         
                Storage Node                   

 nimrod  linux_x86
     NetWorker 7.5.1
                Management Console             
                Man Pages                      
                Client                         

 asgard  linux_x86
     NetWorker 7.4.2
                Man Pages                      
                Client

As you can see in the above, the output is reasonably comprehensive, covering for each host, all three of:

• Version
• Platform
• Packages

Note that you can’t show before you inventory, so don’t start by trying to show the state of all systems – it just won’t work.

Querying software in the repository

To get a list of the software currently held in the repository, you’d run the command:

# nsrpush -l

For instance, here’s the (abridged) output from one of my lab servers:

[root@nox ~]# nsrpush -l

Products in the repository
================================

   NetWorker 7.5.1
     linux_x86                      
         Korean Language Pack           
         Chinese Language Pack          
         Japanese Language Pack         
         French Language Pack           
         Server                         
         License Manager                
         Storage Node                   
         Man Pages                      
         Client                         
         Management Console             
     linux_x86_64                   
         Chinese Language Pack          
         Japanese Language Pack         
         French Language Pack           
         Server                         
         License Manager                
         Storage Node                   
         Man Pages                      
         Client                         
         Management Console             
         Korean Language Pack
   NetWorker 7.5.0
     win_x64                        
         Management Console             
         English Language Pack          
         License Manager                
         Server                         
         Storage Node                   
         Client                         
     win_x86                        
         Management Console             
         English Language Pack          
         Language Packs                 
         License Manager                
         Server                         
         Storage Node                   
         Client

Note that both Windows and Unix versions of NetWorker are stored in the same repository. (Due to restrictions with copy/paste in WordPress not all spacing has been preserved in the above.)

Adding software to the repository

Adding software to the repository is reasonably straight forward. There are two different techniques for this – one for adding the software designated as the same platform of the backup server, and one for adding software designated as the different platform of the backup server.

By “same” I mean:

• Adding Unix/Linux software to a Unix/Linux backup server
• Adding Windows software to a Windows backup server

By “different” I mean:

• Adding Windows software to a Unix backup server
• Adding Unix software to a Windows backup server

Adding software from the same platform

Let’s say I’ve got a Unix NetWorker server, and I want to add software from a supported Unix platform to the repository. Here’s the steps:

1. Download the package from EMC PowerLink.
2. Uncompress and untar the downloaded package into a directory.
3. Run nsrpush against that directory with the appropriate arguments.

So, let’s assume that I want to add NetWorker 7.4.4 for Linux 32-bit to a software repository. I’d start with getting the software into a known directory.

[root@nox tmp]# mkdir /tmp/repo
[root@nox tmp]# cd repo
[root@nox repo]# gunzip -c /d/03/share-a/Downloads/NetWorker/v7_4/
NetWorker 7.4 SP4/nw74sp4_linux_x86.tar.gz | tar xvpf -
LGTO_METAFILE.linuxx86
jre-1_5_0_15-linux-i586.bin
lgtoclnt-7.4.4-1.i686.rpm
lgtofr-7.4.4-1.i686.rpm
lgtoja-7.4.4-1.i686.rpm
lgtoko-7.4.4-1.i686.rpm
lgtolicm-7.4.4-1.i686.rpm
lgtoman-7.4.4-1.i686.rpm
lgtonmc-3.4.4-1.i686.rpm
lgtonode-7.4.4-1.i686.rpm
lgtoserv-7.4.4-1.i686.rpm
lgtozh-7.4.4-1.i686.rpm
sd_products.res

Now, the command we want to use in this case is:

# nsrpush -a -p product -v version -P platform -m path -U

(NB: The “-U” is for “Unix”; obviously if you’re running a like-for-like installation on Windows, using “-U” here isn’t going to be a good idea – instead, use “-W”.)

Now, to work out what we need to put into the product, version and platform strings, we want to check the LGTO_METAFILE.linuxx86 file that was included in the distribution. If you look at the start of the file, you’ll find a section detailing this, and for the file specified above on my lab server, this section looks like the following:

# Client Push metafile to be included in linux_x86 NetWorker distributions

OPERATING_SYSTEM=Linux

OS_PLATFORM=linux_x86

PRODUCT_NAME=NetWorker

PRODUCT_VERSION=7.4.4

DESCRIPTION=Networker 7.4.4

So, that means our nsrpush command will look like this:

# nsrpush -a -p NetWorker -v 7.4.4 -P linux_x86 -m /tmp/repo -U
Success adding product from:
/tmp/repo

Add to repository status: succeeded

That’s all there is to it! That software is now in the repository and ready to be rolled out to a client.

Adding software from the other platform

If you need to add Windows software to a Unix server, or Unix software to a Windows server, you’ll need a helper client. This is a client of the same platform as the software you are adding, with NetWorker already running, and a copy of the software extracted on the client. The procedure works along the lines of the following:

1. Unpack the alternate platform software onto a ‘like’ client.
2. Unpack the alternate platform software onto the backup server.
3. Run nsrpush to add the software.

So in this case, I’m going to use a Windows client called ‘medusa’ as a helper/proxy to allow me to add NetWorker 7.5.1 64-bit for Windows to a Unix server’s repository. On my Windows client, I’ve got a directory called “C:temp751”:

Now, I also need the software unpacked on the backup server, so I flush out the contents of my /tmp/repo directory, and unzip the Windows 64-bit NetWorker 7.5.1 archive. This gives me a path to the LGTO_METAFILE.winx64 file of:

/tmp/repo/win_x64/LGTO_METAFILE.xinx64

Now, I need to add – the command will be:

# nsrpush -a -p product -v version -P platform -m path -c client -C path -W

Where:

• Product, version and platform will be extracted from the LGTO_METAFILE as per usual.
• -m path will refer to the local copy, on the backup server, of the software.
• -c client will refer to the helper/proxy client (in this case, ‘medusa’),
• -C path will refer to the path on the helper/proxy client where the software has also been extracted on
• -W tells NetWorker we need to add Windows software.

The relevant product/version/platform details of the LGTO_METAFILE.winx64 looks like the following:

#########################
# Product Definition
#########################

OPERATING_SYSTEM=Windows
OS_PLATFORM=win_x64
PRODUCT_NAME=NetWorker
PRODUCT_PLATFORM_NAME=NetWorker
PRODUCT_VERSION=7.5.1
DESCRIPTION=Networker 7.5.1

So, our command will therefore be:

[root@nox repo]# nsrpush -a -p NetWorker -P win_x64 -v 7.5.1 -m /tmp/repo
-c medusa -C 'C:temp751' -W
Hostname and mount point recorded.
Success adding product from:
/tmp/repo/win_x64

Add to repository status: succeeded

We can subsequently confirm successful addition to the repository with our trusty nsrpush -l command:

[root@nox repo]# nsrpush -l

Products in the repository
================================

   NetWorker 7.5.1
            win_x64                        
                 Client                         
                 Management Console             
                 Chinese Language Pack          
                 Korean Language Pack           
                 Japanese Language Pack         
                 French Language Pack           
                 English Language Pack          
                 Language Packs                 
                 License Manager                
                 Server                         
                 Storage Node

There you have it – software for an alternate platform added to the repository, with very little work.

Updating client software

Finally, there wouldn’t be much use for the repository without being able to push out updates.

The upgrade command is reasonably straight forward:

# nsrpush -u -p product -v version clients

Where ‘clients’ is a space separated list of one or more clients. (Note that while you can in theory use the ‘-all’ option for all clients, I’d really, really strongly recommend against it.)

Now, our client ‘asgard’ from before was still running NetWorker 7.4.2 for Linux, and I’d like to upgrade it to 7.4.4. The command and upgrade process is as follows:

[root@nox tmp]# nsrpush -u -p NetWorker -v 7.4.4 asgard
Starting upgrade operation on selected clients.
Creating client type job for client asgard.
Copying upgrade scripts to client asgard.
Successfully copied upgrade scripts to client asgard.
Checking tmp space on client asgard.
Successfully issued a tmp space check command to client asgard.
Copying upgrade packages to client asgard.
Successfully copied upgrade packages to client asgard.
Starting upgrade of client asgard.
Successfully issued an upgrade command to client asgard.
Waiting for upgrade status for client asgard.
Still waiting for the upgrade to complete on client asgard.
Waiting for upgrade status for client asgard.
Successfully retrieved and processed the package upgrade status data for client asgard
Starting cleanup phase on client asgard.
Cleaning up client asgard.
Successfully issued the cleanup command to client asgard.

Upgrade status: succeeded

Pushing out software to alternate platforms (i.e., Windows from Unix or vice versa) is exactly the same command as above.

Cautions for upgrades

When you run upgrades, here’s what I’d suggest should be your cautions:

1. Always capture the output and confirm you get an “Upgrade status: succeeded” message for each client that you upgrade.
2. Don’t run the command against more than say, 5 clients at once.
3. Don’t run the command for Windows and Unix clients at the same time.
4. If the upgrade fails, confirm from the output whether NetWorker was able to restart on the client before attempting another push. If it doesn’t look like it did, you may have to install manually.

Many errors that can come out of nsrpush are currently undocumented. Thus, if you run an nsrpush operation and get an error that you can’t find in PowerLink from a casual search, you probably need to log a case with EMC. (For instance, in tests today I got an error “client X has products installed that prevent upgrade”. There’s no publically visible documentation on this error, and despite the seeming logic of it, after checking the client, I don’t understand why the error came about.)

[Edit – 2009-08-19 – While reviewing the 7.4.5 release notes I stumbled across the reason for the “client X has products installed that prevent upgrade” error; while I had been aware that you couldn’t use the repository to update hosts with the NetWorker server software installed, I wasn’t aware that you also can’t use it for hosts with NMC or Legato License Manager installed. The client I experienced an issue on was running LLM.]

Why would you use the repository?

It’s worth revisiting this – obviously I’ve couched all of the above in quite a few caveats and thrown in some curly comments (“many errors … are currently undocumented”). With this in mind, you may be wondering now that you’ve read to the bottom of the post why you would want to risk using the repository and running into problems.

To me the most obvious reason is that the software repository is going to continue to improve. For instance, between 7.4.4 and 7.5.1 many known issues were sorted out, making 7.5.1 much more stable to use. It’s already, say, useful now, if you’re careful with it, and thus keeping abreast of it is going to allow you to continue to save time as each successive version comes out. It is already in a position to save you time and to help you keep clients (and modules) up to date – so this in itself, I believe, is reason enough to use it.

[Edit 2009-08-21 Erroneous information left from a draft removed. Thanks Rolf for pointing this out.]

Finally, even as the GUI continues to improve, I have been and always will be a fan of command line functionality because it allows for scripting. So even when there are limitations with what you can do in NMC, using nsrpush on the command line isn’t rocket science once you get used to the syntax.

Much as there are parts of the repository that I find a bit curmudgeonly at the moment, I’m genuinely happy with where it has (finally) got to, and I’m equally genuinely looking forward to increased functionality and stability as newer versions of NetWorker come out. Over time it’s going to evolve from something which is a “partial time saver” to something that’s a real time saver.