Upgrading NetWorker

So a new version of NetWorker has come out, or is coming out, and it’s been decided that you’re going to upgrade, but you want a few tips for making that upgrade as painless as possible. Here’s my 5 rules for upgrading NetWorker:

  1. Read the release notes. If you’re not going to read the release notes, you are better off staying on your current version, no matter what issues you’re having. I can’t stress enough the importance of reading the release notes and having a thorough grasp of:
    • What has changed?
    • What are the known issues with the current release?
    • What were the resolved issues between the current release and the release you’re currently running?
  2. Do a bootstrap and index backup if upgrading between major or minor releases. If going between service packs on the same release, you can skip the index backup so long as your backups have been successful lately, but ensure you still do a bootstrap backup.
  3. Unload all tapes (physical or virtual) in jukeboxes before the upgrade. You’ll see why shortly.
  4. Upgrade in this order:
    • Storage node(s) on the day of the upgrade, before the NetWorker server
    • Server on the day of the upgrade, after the storage node(s)
    • Client(s) later, at suitable times
  5. After the upgrade but before the NetWorker services are restarted on the storage node(s) and server, delete the nsr/tmp directory on those hosts.

Obviously standard caveats, such as following any additional instructions in the release notes or upgrade notes should of course be followed, but sticking to the above rules as well can save a lot of hassle over time. I’ve noticed over the years that a odd, random problems following upgrades can be solved by clearing the nsr/tmp directory on the server and storage nodes. If there’s no tapes in the jukeboxes when the services first start after the upgrade, there’s less futzing for NetWorker to take care of before it’s fully up and running, too.

 

Just to let you folks know that I’m currently just “mostly missing in action” rather than “missing in action”.

It’s been a hectic few weeks for me, both personally and work-wise, and as my role adjusts to some changes I’ll be doing more travel again which will mean getting used to doing posts on the run.

I’m currently in (not so) sunny Western Australia – specifically, Bunbury, doing some work for a long-term customer, and flying back on Friday. That’ll give me Saturday and Sunday to recharge my batteries and get some Greek coffee into my system. (Maybe, if my friendly tattoo artist will help out, maybe Sunday will also see me getting some new ink. Though maybe that won’t happen if I don’t remember to email her – hmmm…)

Next article: Last week I attended the SNIA AU/NZ first storage blogfest – where myself and a group of other bloggers/tweeters were taken to EMC, IBM, HDS and NetApp for some discussions and technology direction overviews. I had hoped to blog about it by now – but instead I’m aiming to have blogged about it by the time I leave WA – that’ll mean either tonight, or tomorrow night.

Stay tuned – I’m not MIA, just MMIA.

 

On the NetWorker Mailing List, I still frequently see a lot of posts from people who are having various problems with their NetWorker 7.2.x servers.

It’s time to move away from 7.2. I know, it was the last version before nsrjobd; the move to nsrjobd in 7.3, then raw daemon logs in 7.4 can both be a bit shocking, but 7.2 is now critically old and critically out of support. Equally, there’s still a lot of people out there running 7.3 releases of NetWorker. That, too, exited support some time ago, and it’s time to move on from it too.

I’ll agree that within backup, there is a strong logic to the statement “if it ain’t broke, don’t fix it”, but, you have to weigh up that against the simple fact that 7.2.x releases in particular are very old, and 7.3.x releases are fairly aged as well.

Since I’ve been watching more and more of Top Gear, I’ll use a car analogy. Let’s say you’ve got a brand new, top of the line Ferrari. When it needs servicing, do you take it to the official Ferrari shop that provides a 100% warranty on all repairs and whose repairs keep the original vehicle warranty intact, or do you take it to Bill & Joes Motor Fixits ‘R’ Us, who not only might leave you with a car in a worse condition than when you drove it in, but who aren’t certified by Ferrari and thus lose you your new car warranty?

Continuing to backup your environment with a backup product which is long out of support is like outsourcing to Bill & Joes Motor Fixits ‘R’ Us IT Service.

I’ll be the first to admit that even on simple updates you can run into a few hassles. Particularly as you move up the NetWorker version chain you’ll find changes to authentication and name resolution requirements alone that may necessitate some additional work around the time of the update. If your clients are old you’ll also be needing to plan an update for them as soon as possible too, and in some cases, you may find yourself definitely having to update clients if there turns out to be some particularly odd issue.

But I’ll be honest: that little bit of up-front pain is much, much better than hitting a critical backup or recovery problem that can’t be solved without upgrading (or worse, can’t be solved due to incompatibilities between ancient NetWorker versions and modern operating system versions). Planning and implementing a controlled upgrade, even if it does end up having a few hassles, is infinitely better than doing an emergency upgrade without any planning to facilitate a recovery or a backup that has to be done.

 

After 13+ years of using NetWorker, I still tend to interchangeably use the terms ‘upgrade’ and ‘update’ (or to be more precise, mainly use the term ‘upgrade’).

However, there is, and always has been, a difference between the two terms in NetWorker nomenclature, and it’s useful knowing it in case you’re being asked to qualify your environment to a support person.

Here’s what they mean for NetWorker:

  • An upgrade is transitioning from one licensed feature set to a more advanced licensed feature set. For example, you might upgrade from NetWorker, Network Edition to NetWorker, Power Edition. Previously (when tiered licensing was still used for Windows modules), you might upgrade from say, Exchange Module Tier 1 to Exchange Module Tier 2. Alternatively, you can buy an upgrade to slot capacity for an Autochanger license (e.g., upgrading from a 1-64 slot license to a 1-128 slot license).
  • An update is where you change the version of a NetWorker product. E.g., you update from NetWorker 7.4.4 to NetWorker 7.4.5, or from NetWorker 7.3 to NetWorker 7.5.1. You would equally update from Oracle Module 4.5 to Oracle Module 5.

Since in both support and data protection it’s useful to avoid ambiguities, understanding the difference between these two terms can be important.

 

Over the years I’ve been forced to come to one key conclusion about the NetWorker ‘tmp’ directory and NetWorker upgrades.

While it’s not stated in any upgrade documentation, and in theory it shouldn’t matter, anyone upgrading their NetWorker server software should, as a matter of course, delete the server’s NetWorker ‘tmp’ directory prior to starting the new version of the software.

If you’re not familiar with this, you’ll find it:

  • On Unix/Linux platforms as: /nsr/tmp
  • On Windows platforms in the NetWorker install directory, as “Legato\nsr\tmp”.

This directory contains a plethora of files, some relating to savegroups, some relating to operations – in general ‘tmp’ is almost a poor choice of nomenclature for it: I like to think of it as a ‘state’ directory instead.

Nine times out of ten, or perhaps even 49 times out of 50, if I do an upgrade of a NetWorker server and forget to delete the ‘tmp’ directory, what can only be described as weird stuff will happen within 72 hours. Groups may not finish. Media may not unload. Libraries may forget state. The mouse on your desk may spontaneously quantum ooze to the floor. You may hear the Twilight Zone theme music playing in the background when it should be entirely quiet – that sort of stuff.

So, if looking for a new rule to follow when upgrading NetWorker on your server, please make sure you delete the NetWorker ‘tmp’ directory. You’ll save yourself a lot of time and hassles.

(Note: Deleting the NetWorker server ‘tmp’ directory prevents any backup that previously failed or was stopped from being restarted after the failure – it will need to be started as a whole new operation.)

© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha