So a new version of NetWorker has come out, or is coming out, and it’s been decided that you’re going to upgrade, but you want a few tips for making that upgrade as painless as possible. Here’s my 5 rules for upgrading NetWorker:

1. Read the release notes. If you’re not going to read the release notes, you are better off staying on your current version, no matter what issues you’re having. I can’t stress enough the importance of reading the release notes and having a thorough grasp of:
• What has changed?
• What are the known issues with the current release?
• What were the resolved issues between the current release and the release you’re currently running?
2. Do a bootstrap and index backup if upgrading between major or minor releases. If going between service packs on the same release, you can skip the index backup so long as your backups have been successful lately, but ensure you still do a bootstrap backup.
3. Unload all tapes (physical or virtual) in jukeboxes before the upgrade. You’ll see why shortly.
4. Upgrade in this order:
• Storage node(s) on the day of the upgrade, before the NetWorker server
• Server on the day of the upgrade, after the storage node(s)
• Client(s) later, at suitable times
5. After the upgrade but before the NetWorker services are restarted on the storage node(s) and server, delete the nsr/tmp directory on those hosts.

Obviously standard caveats, such as following any additional instructions in the release notes or upgrade notes should of course be followed, but sticking to the above rules as well can save a lot of hassle over time. I’ve noticed over the years that a odd, random problems following upgrades can be solved by clearing the nsr/tmp directory on the server and storage nodes. If there’s no tapes in the jukeboxes when the services first start after the upgrade, there’s less futzing for NetWorker to take care of before it’s fully up and running, too.

Share this using...

Related posts:

1. 10 NetWorker Configuration Rules Over the last 15 years, I’ve administered, configured and supported...

Related posts brought to you by Yet Another Related Posts Plugin.

[root@tara ~]# nsrjb
1:	VTL1 	[enabled]
2:	VTL2 	[enabled]
No jukebox selected.
Please select a jukebox to use:? [1] _

(Slight aside: never assume the numbered list is the same; NetWorker doesn’t guarantee the order being the same between executions – in fact, I actually only put in an RFE about this a couple of days ago, as I’m hoping it could at least be alphabetically ordered at all times…)

If you want to avoid the jukebox-prompt from nsrjb, one of the easiest ways is to specify the jukebox name as part of the command – e.g.,

[root@tara ~]# nsrjb -j VTL1

That’s fine of course, but if the vast majority of the time you perform operations on a single jukebox, you can specify a default jukebox as an environment variable (NSR_JUKEBOX) and streamline your processes. For example, on Linux, using the bash, this might look as follows:

[root@tara ~]# export NSR_JUKEBOX=VTL1
[root@tara ~]# nsrjb

Jukebox VTL1: (Ready to accept commands)
slot  volume         pool           barcode   volume id        recyclable
1: 800840L4       ClientTesting  800840L4  3814088325       no
2: 800841L4       ClientTesting  800841L4  3797311146       no
3: 800842L4       ClientTesting  800842L4  3847642669       no
4: 800843L4       ClientTesting  800843L4  3780533937       no
5: 800844L4       ClientTesting  800844L4  3763756765       yes
6: 800845L4       ClientTesting  800845L4  3864419885       yes
<snip>

Being an environment variable, this is something you can choose to set locally – say, on a per storage-node basis, when you have multiple storage nodes. It’s relatively common for instance to have a tape library on one or more storage nodes, so for the appropriate logins (or even at a system level) on each storage node it would be possible to set the local jukebox as the default, thereby streamlining usage of the units.

As an example, here’s a lab storage node with the setting in use:

[root@fawn ~]# export NSR_JUKEBOX="rd=fawn:VTL3"
[root@fawn ~]# nsrjb -s tara

Jukebox rd=fawn:VTL3: (Ready to accept commands)
<snip>

For something that can take you less than 30 seconds to set, the environment variable NSR_JUKEBOX can certainly be a big time saver if you have multiple jukeboxes in your environment and (like me) you’re a command line junkie.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

I was pleasantly surprised today to find that a new command had slipped in at some point called “jbverify”. I can immediately see it however entering my stable of must-use commands, particularly in a support environment.

To quote the man page, jbverify:

[V]erifies the devices defined in the NetWorker database, making sure that each one of them is configured properly by checking them  for accessibility  and  usability.

This is the sort of diagnostic tool that support people live for, and sites suddenly experiencing strange jukebox issues should think of as a matter of course.

When run on my lab server this afternoon, I got the following badly formatted but still very useful output:

# jbverify
14866:jbverify:
 Jbverify is running on host nox, Linux 2.6.18-128.1.6.el5

14912:jbverify:
 Processing stand-alone devices...

14913:jbverify:
 Processing /d/nsr/02/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/02/_AF_readonly

14913:jbverify:
 Processing /d/nsr/idata/backup/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/idata/backup/_AF_readonly

14913:jbverify:
 Processing /d/nsr/01/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/01/_AF_readonly

14913:jbverify:
 Processing /d/nsr/idata/backup
14915:jbverify:
 Finished processing /d/nsr/idata/backup

14913:jbverify:
 Processing /d/nsr/idata/clone/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/idata/clone/_AF_readonly

14913:jbverify:
 Processing /d/nsr/01
14915:jbverify:
 Finished processing /d/nsr/01

14913:jbverify:
 Processing /d/nsr/03
14915:jbverify:
 Finished processing /d/nsr/03

14913:jbverify:
 Processing /d/nsr/03/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/03/_AF_readonly

14913:jbverify:
 Processing /d/nsr/02
14915:jbverify:
 Finished processing /d/nsr/02

14913:jbverify:
 Processing /d/nsr/idata/clone
14915:jbverify:
 Finished processing /d/nsr/idata/clone

14917:jbverify:
 Finished processing stand-alone devices.

14918:jbverify:
 Processing jukebox devices...

14920:jbverify:
 Processing jukebox LTO1_LIB:

14733:jbverify:
 Testing drive 1 (/dev/nst0) of JB LTO1_LIB

14927:jbverify:

 Jukebox LTO1_LIB on nox successfully processed.

14929:jbverify:

 Finished processing jukebox devices.

**********************************************************************

         Summary report of jbverify
         ======= ====== == ========

Hostname   Device Handle    Blocksize  Jukebox  Drv No. Status
--------   -------------    ---------  -------  ------- ------
nox        /d/nsr/02/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/idata/backup/_AF_readonly 131072 N/A N/A Pass
nox        /d/nsr/01/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/idata/backup 131072  N/A      N/A     Pass
nox        /d/nsr/idata/clone/_AF_readonly 131072 N/A N/A Pass
nox        /d/nsr/01        131072     N/A      N/A     Pass
nox        /d/nsr/03        131072     N/A      N/A     Pass
nox        /d/nsr/03/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/02        131072     N/A      N/A     Pass
nox        /d/nsr/idata/clone 131072   N/A      N/A     Pass
nox        /dev/nst0        65536      LTO1_LIB 1       Pass

**********************************************************************

If you’ve come from NetBackup, the nature of this program is somewhat reminsicent of the robtest utility. I don’t claim EMC are special for having introduced this tool, but I do applaud that it’s there (and lament that I didn’t notice it sooner).

(One thing to note: after running jbverify, make sure you reset your jukebox.)

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

That being said, in smaller environments, or environments where servers are relatively idle overnight, NetWorker’s datazone encryption can be sufficient to achieve a reasonable modicum of backup protection with minimum effort – and most importantly, cost.

To get started using NetWorker datazone encryption, you first need to assign a pass phrase. This is done in the NetWorker server properties (typically accessed within NMC):

With the pass phrase in place, you can then configure directives within NetWorker to make use of AES 256 bit encryption. However! As soon as you turn encryption on, you lose all potential for hardware based compression for your media. Why? Quite simply, compression is about finding patterns in data and reducing all the matching patterns to a single reference point; however, encryption is all about eliminating patterns, making the data appear completely random.

Thus, if you want to still get some measure of compression, you should, when using this method, employ software based compression in your directive as well.

Thus, a base directive might look like the following:

<< / >>
+compressasm: .
+aes: .

This will apply compression first to all files encountered, then once the file has been compressed, it will be encrypted. A side benefit of this is that by compressing first, you reduce the amount of data to be encrypted*.

So long as the datazone pass phrase is stored in the server, encryption will occur, and no password will be required to recover the data. Remember, this style of encryption, using a single pass-phrase, isn’t about being able to restrict whom within the datazone can recover the data, but instead it’s about keeping the data stored on-tape (which is potentially off-site, or otherwise at higher risk of theft), from being recovered.

[Edit, 2009-08-15]

It’s been pointed out to me that you can’t compress + encrypt at the client side. Indeed, I’ve now found the part in the administration guide that explicitly says this. What is extremely disappointing about this is that NetWorker actually doesn’t warn you that it’s not going to compress + encrypt! To me, that’s a security issue.

So, for the examples above, forget about enabling client side compression as well as encryption – you can have one or the other, but not both.

–
* In the same way that ice-cream that’s 99% fat free, but 87% sugar is a “benefit”.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

While you don’t need it any more to control remote jukeboxes, it can still be useful to know how to do remote control operations in NetWorker – particularly if you’re say, debugging backups on a client but you don’t have console access to that client.

Note that this only applies to commands that obey the following restrictions:

• Name starts with “nsr” or “save”
• Resides in the same directory as the “save” command on the client.

Thus, this isn’t about remote hijacking of a NetWorker client. Such commands can only be executed by an authorised administrator on the backup server as specified in the nsr/servers file. Before you (quite rightly) point out that it would mean that a valid NetWorker administrator could in fact hijack a client by say, doing a directed recovery out to that client of an appropriately named file, they can do that already – that’s part of the trust relationship of the NetWorker administrator anyway.

So, with all those caveats out of the way, it’s remarkably simple. Going from a Unix host, you do the following:

# export RUSER=<user>
# export RCMD=<cmd>
# nsrexec -c <client>

Where:

• user will typically be “root” or “administrator”, depending on whether your backup server is Unix or Windows.
• cmd will be a NetWorker command you want executed.
• clientName is the name of the client the command is to be executed from.

For instance, say I’ve got a NetWorker server called “nox”, and a client called “asgard” that I don’t have administrative login to, but I want to simulate a backup without firing up a savegroup. To do so, I could do the following:

# export RUSER=root
# export RCMD="save -e tomorrow -b Default -LL -q -s nox /tmp"
# nsrexec -c asgard
save: /tmp  36 MB 00:00:05     38 files
completed savetime=1235278395

Admittedly this is not a technique you should need to know often, but it’s useful to know about.

–
* You do populate your nsr/servers file, don’t you? If you don’t, go do it NOW. I mean it, stop what you’re doing, go and fix up the nsr/servers file on every client in your environment!

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

To change the browse or retention time, you’ll need to find out the saveset ID (SSID) of the given saveset. This can be done with mminfo.

For instance, say you had a backup done last night of a machine called ‘archon’ that has now been rebuilt, but you want to keep the old backup for much longer than normal – e.g., ten years instead of the normal 3.

First, to find out what you need to change, get a list of the SSIDs:

# mminfo -q "client=archon,savetime>=24 hours ago" -r name,ssid
 name                          ssid
/                              4036558666
/Volumes/TARDIS/Yojimbo        4019781450
/Volumes/Yu                    4003004234

(If you’re confused about that savetime command, see my other post here.)

Now, for each of those SSIDs that are returned, we’ll run a nsrmm command to adjust the browse and retention time*.

The basic nsrmm command for adjusting the browse and retention time is:

# nsrmm -S ssid -w browse -e retent

or, for a single instance of a saveset:

# nsrmm -S ssid/cloneid -w browse -e retent

Where the ‘browse’ and ‘retent’ values can be either one of the two following:

• A literal date in US date format ** – e.g., “12/31/2019″ for 31 December 2019.
• A ‘fuzzy’ english worded date – e.g., “+10 years” for 10 years from today.

Note that (rather obviously), your browse time cannot exceed your retention time, and generally its recommended that you set browse time to retention time.

So in this case, you’d run for each SSID or SSID/CloneID you want to affect:

# nsrmm -S ssid -w "+10 years" -e "+10 years"

Which will look like the following, based on my mminfo output:

# nsrmm -S 4036558666 -w "+10 years" -e "+10 years"
# nsrmm -S 4019781450 -w "+10 years" -e "+10 years"
# nsrmm -S 4003004234 -w "+10 years" -e "+10 years"

It’s that simple.

–
* You can also do this against an instance of a saveset by using the SSID/Clone ID; to do that variant, request “-r name,ssid,cloneid”, then use the two numbers in the nsrmm command separated by a forward slash – ssid/cloneid.

** The restriction on US date format may have eased in 7.5. I’m going to do some additional playing around with locales sometime soonish.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

As such, when I want to stop and restart NetWorker on Windows, I find it faster on the command line than going to the services panel and scrolling through to the various processes.

Note: All of these options are documented in the NetWorker and License Manager administration manuals, but as always, there’s a difference between something that’s documented and something you can find, so I’m presenting these for those who are pressed for time.

There are lengthy names for each of the NetWorker client service, NetWorker server service, and NetWorker Management Console server service which you may think would make it inconvenient to stop and restart from the command line, but thankfully there’s shortcut names as well. These are:

• NetWorker Remote Exec Service: nsrexecd
• NetWorker Backup and Recover Server: nsrd
• EMC GST Service: gstd
• NetWorker License Manager: lgtolmd

Now, if you’re wanting to quickly stop everything, you can rely on process dependencies and run:

C:\> net stop nsrexecd /y

That will shutdown everything, since everything relies on the nsrexecd (client) process in order to run.

If you want to start NetWorker on a client, all you need to run is:

C:\> net start nsrexecd

If you’re wanting to start NetWorker on a server though, you should run:

C:\> net start nsrd

There’s a simple reason for this – nsrd relies on nsrexecd, so Windows dependency checking will start nsrexecd first for you.

Similarly, if you want to start the management console on a Windows machine, you’d run:

C:\> net start gstd

And if you’re running Legato License Manager, you’d run:

C:\> net start lgtolmd

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

So this post is all about that 1-5% of the time where you may find it necessary to say, search for a saveset that’s reported in the media database that you’re having problems accessing from the disk backup unit.

It’s actually trivially easy, once you know how.

You may be familiar with the following style of query:

# mminfo -q "savetime>=24 hours ago" -r "volume,client,level,sumsize,savetime(22)"

The (22) at the end of the savetime report parameter tells mminfo to allow 22 characters for the reporting of the savetime. The benefit of this is that you get not only the savetime date, but also the time as well.

NetWorker actually allows you to put that (number) postfix onto any field that it can output in mminfo. This can output additional information, such as the above, or give more room to output longer fields, or even limit the size of fieldnames when you don’t need too much information. (E.g., if the first 4 characters of all client names can uniquely identify the client, you might limit the client to 4 characters in an mminfo report.)

Now, where we’re heading with this, is that the sorts of filenames used for the savesets written to disk backup units are not some random collection of strings – they’re actually the long saveset ID.

Consider then a filename of:

/d/nsr/02/63/05/cd3a182f-00000006-7b7801de-497801de-01871a00-3d2a4f4b

This isn’t just a random filename, it’s the saveset ID, but just in a format you may not used to.

To get the long saveset ID in mminfo output, we use the (number) postfix on the ssid field. This would be as follows:

# mminfo -q "ssid=071462366" -r "ssid(53)"
cd3a182f-00000006-7b7801de-497801de-01871a00-3d2a4f4b

With that information in hand, you can then search for a file with the same name as the long saveset ID on disk.

You can also do a reverse lookup. Say for instance, you know there’s an issue with a particular saveset file on a disk backup unit. To find out what the actual saveset ID is for this saveset, you can run the counter-query:

mminfo -q "ssid=cd3a182f-00000006-7b7801de-497801de-01871a00-3d2a4f4b" -r ssid

So, there you go – very easy!

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.