Upgrading NetWorker

So a new version of NetWorker has come out, or is coming out, and it’s been decided that you’re going to upgrade, but you want a few tips for making that upgrade as painless as possible. Here’s my 5 rules for upgrading NetWorker:

  1. Read the release notes. If you’re not going to read the release notes, you are better off staying on your current version, no matter what issues you’re having. I can’t stress enough the importance of reading the release notes and having a thorough grasp of:
    • What has changed?
    • What are the known issues with the current release?
    • What were the resolved issues between the current release and the release you’re currently running?
  2. Do a bootstrap and index backup if upgrading between major or minor releases. If going between service packs on the same release, you can skip the index backup so long as your backups have been successful lately, but ensure you still do a bootstrap backup.
  3. Unload all tapes (physical or virtual) in jukeboxes before the upgrade. You’ll see why shortly.
  4. Upgrade in this order:
    • Storage node(s) on the day of the upgrade, before the NetWorker server
    • Server on the day of the upgrade, after the storage node(s)
    • Client(s) later, at suitable times
  5. After the upgrade but before the NetWorker services are restarted on the storage node(s) and server, delete the nsr/tmp directory on those hosts.

Obviously standard caveats, such as following any additional instructions in the release notes or upgrade notes should of course be followed, but sticking to the above rules as well can save a lot of hassle over time. I’ve noticed over the years that a odd, random problems following upgrades can be solved by clearing the nsr/tmp directory on the server and storage nodes. If there’s no tapes in the jukeboxes when the services first start after the upgrade, there’s less futzing for NetWorker to take care of before it’s fully up and running, too.

 

Clients

The Question

It’s usually the case that the biggest part of a NetWorker environment – in terms of resources that are configured, and software deployed, are the clients themselves. When sites look at upgrading their NetWorker environments though, the normal procedure is to upgrade the server and any storage nodes as the first step, then plan to upgrade clients on an “as needed” or “when we get around to it” basis.

This prompted a customer to recently ask me to write a blog article about this topic (thanks, Robert!) Specifically, Robert’s question was – why should I upgrade my clients?

Having worked with several of my clients now for close to a decade, I’m familiar with the scenario: the servers and storage nodes will be at appropriately supported versions of the NetWorker software, but clients are trailing behind, and before you know it your versions may stretch out like a long tail behind your backup server and storage nodes:

Client versionsSo it begs the question – when NetWorker is so good at supporting older client versions, what’s the rush in upgrading old clients? This is a question where an answer of “…because…?” isn’t sufficient, so perhaps first it’s worthwhile considering some common arguments for not upgrading the clients:

  • If it’s not broken, don’t fix it.
  • We had some problems with version X, it’s stable on X+n, so keep it that way. (A variant of the above.)
  • It’s working, so it’s a low priority task.
  • Admins are too busy fire fighting to do unnecessary upgrades.
  • Change control is too tedious.
  • This is the last supported version for this <old> operating system.

The Answer

The generic answer

Each of the above reasons, in their own right, can be a perfectly valid reason. Temporarily stepping away from backup software and looking at say, operating systems, here’s some example reasons why we eventually choose to upgrade operating systems:

  • We explicitly need the new features.
  • New applications require the new features.
  • Poor support on old OS for new hardware (and vice versa).
  • More efficient.
  • Faster.
  • More secure.

We can evaluate a whole host of  reasons, but we can actually boil any upgrade rationale down to one of the following three generic reasons:

  1. Risk – The risk in not upgrading overrides the cost of upgrading. Two common risks are security or reliability.
  2. Features – The currently installed version lacks features that are both available and required in a newer version available.
  3. Support – The currently installed version is either out of support, or is scheduled to no longer be supported as of a known, unacceptably close date.

Note – regarding features: To be a valid upgrade reason, it should be both available and required, not one or the other – and yes, sometimes upgrades are done based on features being required without first checking if they’re available!

When we boil down upgrade reasons to just three generic terms, risk, features and support, it becomes easier to justify either:

  • Having an active programme in place to keep clients up to date or
  • Periodically updating clients.

So going back to NetWorker clients, we can evaluate what sort of reasons in each of the generic categories might prompt an upgrade; I’m going to go backwards through the previous list.

The NetWorker answer

Support

To me, unsupported = broken. So, “if it’s not broken, don’t fix it” stops being a valid reason at the point where client software installed is no longer supported. So for sites that have v7.3.x and lower clients laying around – or come October 1 2010, v7.4.x and lower clients around, you should either:

  • Upgrade to a supported version or
  • Upgrade to the last supported version that is compatible with the client (for very old clients/applications).

If a client is on an unsupported version of the software and it can be upgraded to a support version, leaving it on that unsupported version can introduce unnecessary risk in the environment. While a current version of NetWorker will more than likely keep communicating with an older version of NetWorker, that doesn’t mean that issues can’t happen, and if they do, you want to be able to resolve the issue as quickly as possible. By having a supported version of the client installed, you can considerably streamline the resolution process.

Features

We have a tendency to focus on the backup server (and to a lesser degree), storage node, when looking for features support. For instance, we may want disk backups to be able to do X, or NDMP backups to be able to do Y, and so on. However, feature support isn’t enhanced only at the server layer. In actual fact, a lot of feature support comes from the client software. For instance:

  • If you’re working with Solaris 10 clients that are deployed in non-global domains, having up-to-date client software ensures that you maximise your support of that configuration;
  • If you’re looking at upgrading a host from Windows 2003 to Windows 2008 R2, you’re likely going to need to upgrade the NetWorker client – you need a newer client instance that has more up to date support for the newer operating systems;
  • If you’re wanting to eliminate no-longer-needed licenses within your backup environment, and are looking at getting rid of those ClientPak licenses, you’ll need to make sure that the clients themselves support the removal of the licenses;
  • If you want to be able to do VSS filesystem backups but not have to buy VSS licenses, you’ll need to have a version of the NetWorker client that supports this option;
  • If you want to replace your Oracle 9 database with Oracle 11, you may find yourself needing to upgrade the database module. This in turn may necessitate an upgrade of the client software to support the newer module, too.

Suffice it to say, feature support can be just as important at the client level as it is at the backup server level. In this regard, the release notes will always be an excellent reference – if you’re not sure whether you need to upgrade, check to see what new functionality comes into play on the latest versions of the software.

Risk

The final reason to upgrade is risk – risk that there is a bug or a security issue in the currently installed version of the software that may be resolved in a newer version. Like “Features”, above, your best bet for determining the risk of not upgrading is by referring to the release notes for newer versions of the software. Read the “fixed issues” notes very carefully; it could be that intermittent issues you haven’t yet found time to investigate – or that you have been actively trying to resolve – are actually resolved in a newer version of the software. While we often look at fixed issues in NetWorker release notes for the server and storage node, they can be equally applicable at the client level, too.

When should clients be upgraded?

Once we’ve determined that we can decide to upgrade clients on the basis of either support, features or risk, we must next ask ourselves the question – when should the clients be upgraded? There’s a sister question to this too – how frequently should clients be upgraded?

I’m not going to suggest that your backup server and all its clients should be kept in absolute version lock-step the entire time. If you have the processes, personnel and time to do this, then by all means go ahead – but it isn’t something that you should obsessively worry about. Instead, I’ll offer some generic suggestions; to do this though I’ll refer to major and significant version numbers. Consider say, NetWorker 7.5 SP2; I’d consider the major version number to be 7, the significant version number to be 5, and the service pack to be 2.

  • Aim to keep all clients that support it on at least the same major version number as the backup server;
  • Where time permits try to get clients on the same (or higher*) major+significant version number as the backup server – but as a general rule, ensure that the clients are at least on a supported major+significant version number.
  • Consider getting clients onto the same major+significant+service pack version as the backup server where there are support, risk or feature reasons, i.e.:
    • Where there are new features in the service pack you need, or,
    • Where there are risks in remaining at the current version, or,
    • Where there are support reasons for updating. (E.g., patch available for new SP that would need to be back-ported to your existing version).

You may think that all these answers are a bit vague – and by necessity, they are, since the issues, needs and processes at each site will govern exactly how and why upgrades are done.


* Yes, or higher. Such as for instance, sites that have been running a NetWorker 7.4.x server, but need to run a 7.5 SP2 client for Windows 2008 R2 systems, etc.

 

On the NetWorker Mailing List, I still frequently see a lot of posts from people who are having various problems with their NetWorker 7.2.x servers.

It’s time to move away from 7.2. I know, it was the last version before nsrjobd; the move to nsrjobd in 7.3, then raw daemon logs in 7.4 can both be a bit shocking, but 7.2 is now critically old and critically out of support. Equally, there’s still a lot of people out there running 7.3 releases of NetWorker. That, too, exited support some time ago, and it’s time to move on from it too.

I’ll agree that within backup, there is a strong logic to the statement “if it ain’t broke, don’t fix it”, but, you have to weigh up that against the simple fact that 7.2.x releases in particular are very old, and 7.3.x releases are fairly aged as well.

Since I’ve been watching more and more of Top Gear, I’ll use a car analogy. Let’s say you’ve got a brand new, top of the line Ferrari. When it needs servicing, do you take it to the official Ferrari shop that provides a 100% warranty on all repairs and whose repairs keep the original vehicle warranty intact, or do you take it to Bill & Joes Motor Fixits ‘R’ Us, who not only might leave you with a car in a worse condition than when you drove it in, but who aren’t certified by Ferrari and thus lose you your new car warranty?

Continuing to backup your environment with a backup product which is long out of support is like outsourcing to Bill & Joes Motor Fixits ‘R’ Us IT Service.

I’ll be the first to admit that even on simple updates you can run into a few hassles. Particularly as you move up the NetWorker version chain you’ll find changes to authentication and name resolution requirements alone that may necessitate some additional work around the time of the update. If your clients are old you’ll also be needing to plan an update for them as soon as possible too, and in some cases, you may find yourself definitely having to update clients if there turns out to be some particularly odd issue.

But I’ll be honest: that little bit of up-front pain is much, much better than hitting a critical backup or recovery problem that can’t be solved without upgrading (or worse, can’t be solved due to incompatibilities between ancient NetWorker versions and modern operating system versions). Planning and implementing a controlled upgrade, even if it does end up having a few hassles, is infinitely better than doing an emergency upgrade without any planning to facilitate a recovery or a backup that has to be done.

 

Recently we’re seeing a lot of people upgrading to Windows 2008 SP2, without first checking to see that release notes and compatibility guides state that NetWorker doesn’t yet support this release.

I fully agree that this represents monumental slowness on the part of EMC … there’s absolutely no excuse – none whatsoever – for them not to be on the relevant developer programmes and partner programmes for all the supported operating systems so they get access to the new releases before they come out and then make sure there’s either hot-fixes or cumulative updates available to support new operating systems.

They don’t have to be on the same day, but it’s foolish short-sightedness at best that they don’t support a new OS release within say, 2 weeks of it hitting the general public, given that partner and developer programmes will give access to it for months in advance of that point.

Now, back to my original point – if you’re planning on rolling out a new service pack to an operating system, please take a few minutes to read the release notes or software compatibility guides – or ask your support team to fill you in, and if it’s not supported, roll out to a test client first so you can confirm the impact to your backup environment.

It takes two to tango – EMC needs to improve their response to new operating systems and new major updates to operating systems, but it’s equally important for people to remember to check these things before they upgrade, not after they get the first backup (or worse! recovery) error.

(Do I do these checks all the time? No – only in lab environments. It’s my job to identify bugs and issues before my customers find them as much as possible.)

 

After 13+ years of using NetWorker, I still tend to interchangeably use the terms ‘upgrade’ and ‘update’ (or to be more precise, mainly use the term ‘upgrade’).

However, there is, and always has been, a difference between the two terms in NetWorker nomenclature, and it’s useful knowing it in case you’re being asked to qualify your environment to a support person.

Here’s what they mean for NetWorker:

  • An upgrade is transitioning from one licensed feature set to a more advanced licensed feature set. For example, you might upgrade from NetWorker, Network Edition to NetWorker, Power Edition. Previously (when tiered licensing was still used for Windows modules), you might upgrade from say, Exchange Module Tier 1 to Exchange Module Tier 2. Alternatively, you can buy an upgrade to slot capacity for an Autochanger license (e.g., upgrading from a 1-64 slot license to a 1-128 slot license).
  • An update is where you change the version of a NetWorker product. E.g., you update from NetWorker 7.4.4 to NetWorker 7.4.5, or from NetWorker 7.3 to NetWorker 7.5.1. You would equally update from Oracle Module 4.5 to Oracle Module 5.

Since in both support and data protection it’s useful to avoid ambiguities, understanding the difference between these two terms can be important.

© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha