Dec 232016

I know, I know, it’s winter up there in the Northern Hemisphere, but NetWorker 9.1 is landing and given I’m in Australia, that makes NetWorker 9.1 a Summer Fresh release. (In fact, my local pub for the start of summer started doing a pale ale infused with pineapple and jalapeños, and that’s sort of reminding me of NetWorker 9.1: fresh, light and inviting you to put your heels up and rest a while.)

NetWorker 9.1


NetWorker 9 was a big – no, a huge – release. It’s a switch to a more service catalogue driven approach to backups, Linux block based filesystem backups, block based application backups, deep snapshot integration and more recently in NetWorker 9.0 SP1, REST API control as well.

NetWorker 9.1 as you’d expect is a smaller jump from 9.0 than we had from 8.2 to 9.0. That being said, it’s introduced some excellent new features:

  • VMAX SmartSnap integration – the ability to backup and restore a VMAX device based on the device WWN, increasing the depth of snapshot support in NetWorker further.
  • Snapshot Alternate Location Rollback – this lets you do a snapshot rollback, but to a different set of devices.
  • Data Domain High Availability integration – Data Domain now supports high-availability on the earlier 9500 platform, in addition to the 9800, 9300 and 6800 systems. And with v9.1, NetWorker fully understands and integrates with DDHA platforms.
  • Cloud Tier Integration – NetWorker gets deep integration into the Cloud Tier functionality introduced in Data Domain OS 6.0. This lets NetWorker cloning policies control the migration of data out to the Cloud Tier, and more seamlessly integrate with the recall process.

Cloud Tier integration is more than just a tick in the box to though. Consider the module space – NetWorker Module for Microsoft Applications, for instance, doesn’t just get the option to recover data from Cloud Tier, but also perform granular recoveries from Cloud Tier – SQL table level recoveries and Exchange granular recoveries as well.

By the way, the NetWorker Usage Survey is still running – don’t forget to fill in how you’re using NetWorker! (And be in the running for a prize.)

I’ve saved the best – and biggest – feature for last, though. This is a doozy. Say goodbye to needing a EBR/VBA for VMware backups. That EBR/VBA functionality is now embedded in the NetWorker server itself, leaving you to just deploy some very lightweight proxies to handle the data transport processes, all controlled by NetWorker.

The current EBR appliance and proxies will continue to work with NetWorker 9.1, but I can’t think of anyone who’d want to upgrade to 9.1 without rapidly transitioning to the new platform. Here are just some of the advantages of the new process:

  • Less virtual infrastructure required – no EBRs
  • Virtual machines stored in raw VMDK file – no additional processing required for the backup, and this will also mean faster instant access processes, too
  • The FLR web GUI now runs on the NetWorker server itself
  • NMC can be used for FLR instead of the web GUI, making it more accessible to the NetWorker administrators if they don’t have access to the virtual machines being protected
  • Proxies support more concurrent virtual machine backups:
    • Maximum 25 concurrent hotadd operations;
    • Maximum 25 concurrent NBD operations
  • Significantly increased File Level Recovery (FLR) counts from VMware Image Level Backups (recommended 20,000 – more on that in a minute)
  • Significantly faster FLR operations.

In fact, I’m going to spend a little bit of time on FLR for this post, and step through the new NMC-based FLR process to give you an overview of the process. This is using the newly deployed NetWorker VMware Protection (NVP) system, with backup to and recovery from Data Domain virtual edition.

Fig 01: Starting a recovery in NMC

Fig 01: Starting a recovery in NMC

You start by telling NMC you want to do a virtual machine recovery and choose the vCenter server that owns the virtual machine(s) you want to recover data from.

Fig 02: Choosing the virtual machine to recover from

Fig 02: Choosing the virtual machine to recover from

There’s various options for choosing the virtual machine to recover data for – you can enter the name directly, search for it, browse the various backups that have been performed, or browse the vCenter server itself.

Fig 03: Virtual Machine selected

Fig 03: Virtual Machine selected

Once you’ve selected a virtual machine for recovery, you can click Next to choose the backup to recover from.

Fig 04: Choosing the backup to recover from

Fig 04: Choosing the backup to recover from

In this case, I only had a single backup under the new NVP system for that virtual machine, so I was able to just click Next to continue the process. At this point you get to choose the type of recovery you want to perform:

Fig 05: Choosing the type of recovery to perform

Fig 05: Choosing the type of recovery to perform

As you can see, there’s a gamut of recovery options for virtual machines within NMC. I’m focusing on the FLR options here so I chose the bottom option and clicked Next.

Fig 06: Choosing backup instance to recover from

Fig 06: Choosing backup instance to recover from

Next you get to choose the backup instance you want to recover from. If the backup has been cloned it may be that there’s topologically a better backup to recover from than the original, and choosing an alternate is as simple as scrolling through a list of clones.

At that point you get to choose where you want to recover to:

Fig 07: Choosing where to recover data to

Fig 07: Choosing where to recover data to

Next, you’ll supply appropriate credentials for the virtual machine to be able to perform the recovery and initiate a mount of the backup into the proxy server:

Fig 08: Supplying virtual machine credentials to mount the backup

Fig 08: Supplying virtual machine credentials to mount the backup

After you’ve supplied the credentials you’ll click “Start Mount” to make the specific backup available for recovery purposes, and after a few seconds that’ll result in log information such as:

Fig 09: Mounted and ready

Fig 09: Mounted and ready

When the mount is done, you’re ready to click Next and start browsing files for recovery.

Fig 10: Choosing files to recover from an image level backup

Fig 10: Choosing files to recover from an image level backup

In this example, I selected a directory with about 7,800 files in it and the marking of files for recovery took just a few seconds to complete. After which, Next to choose where to recover the data to on the selected virtual machine:

Fig 11: Choosing where to recover data to on the virtual machine

Fig 11: Choosing where to recover data to on the virtual machine

In this case I choose to recover to C:\tmp on the virtual machine. Clicking Next allows finalisation of the recovery preparation:

Fig 12: Finalising the recovery configuration

Fig 12: Finalising the recovery configuration

As you would expect with the tightly integrated controls now, FLR is fully visible within the NetWorker environment – even nsrwatch:

Fig 13: FLR in progress shown in nsrwatch

Fig 13: FLR in progress shown in nsrwatch

And finally we have a completed recovery:

Fig 14: Completed recovery

Fig 14: Completed recovery

That’s 7,918 files recovered from an image level backup in 54 seconds:

Fig 15: Recovered content

Fig 15: Recovered content

I wanted to check out the FLR capabilities a little more and decided to risk pushing the system beyond the recommendations. Instead of just recovering a single folder with 7,900 files or thereabouts, I elected to recover the entire E:\ drive on the virtual machine – comprising over 47,000 files. Here’s the results:

Fig 16: Large scale FLR results

Fig 16: Large scale FLR results

The recovered folder:

Fig 17: Recovered Content

Fig 17: Recovered Content

47,198 files, 1,488 folders, 5.01GB of data recovered as an FLR from an image level backup in just 5 minutes and 42 seconds.

If you’re using NetWorker for VMware backups, here’s the version you want to be on.

You can get it from the EMC Support page for NetWorker today.

What’s new in 8.2?

 EMC, NetWorker  Comments Off on What’s new in 8.2?
Jun 302014

NetWorker 8.2 entered Directed Availability (DA) status a couple of weeks ago. Between finishing up one job and looking for a new one, I’d been a bit too busy to blog about 8.2 until now, so here goes…

what's new in 8.2

First and foremost, NetWorker 8.2 brings some additional functionality to VBA. VBA was introduced as the new backup process in NetWorker 8.1. Closely integrating Avamar backup technologies, VBA leverages a special, embedded virtual Avamar node to achieve high performance backup and recovery. Not only can policies be defined in NMC for VBA can be assigned by a VMware administrator in the vSphere Web Client,  … so too can image level backup and recovery operations be executed there. Of course, regularly scheduled backups are still controlled by NetWorker.

That was the lay of the land in 8.1 – 8.2 reintroduces some of the much-loved VADP functionality, allowing for a graphical visualisation map of the virtual environment from within NMC.

Continuing that Avamar/VMware integration, NetWorker 8.2 also gets something that Avamar 7 administrators have had for a while – instant-on recoveries when backups are performed to Data Domain. There’s also an emergency restore option to pull a VM back to an ESX host even if vCenter is unavailable, and greater granularity of virtual machine backups – individual VMDK files can be backed up and restored if necessary. For those environments where VMware administrators aren’t meant to be starting backups outside of the policy schedules, there’s also the option now to turn off VBA Adhoc Backups in NMC.

Moving on from VMware, there’s some fantastic snapshot functionality in NetWorker 8.2. This is something I’ve not yet had a chance to play around with, but by all accounts, it’s off to a promising start and will continue to get deeper integration with NetWorker over time. Currently, NetWorker supports integrating with snapshot technologies from Isilon, VNX, VNX2, VNX2e and NetApp, though the level of integration depends on what is available from each array. This new functionality is called NSM for NAS (NetWorker Snapshot Management).

The NSM integration allows NAS hosts to be integrated as clients within NetWorker for policy management, whilst still working from the traditional “black box” scenario of NAS systems not getting custom agents installed. There’s a long list of functionality, including:

  • Snapshot discovery:
    • Finding snapshots taken on the NAS outside of NetWorker’s control (either before integration, or by other processes)
    • Facilitate roll-over and recovery from those snapshots (deleting isn’t available)
    • Available as a scheduled task or via manual execution
  • Snapshot operations:
    • Create snapshots
    • Replication snapshots
    • Move snapshots out to other storage (Boost, tape etc) using NDMP protocols
    • Lifecycle management of snapshots and replicas via retention policies
    • Recover from snapshots

Data Domain Boost integration gets a … well, boost, with support for Data Domain’s secure multi-tenancy. This support scaling for large systems designed for service providers, with up to 512 Boost devices supported per secure storage unit on the Data Domain. While previously there was a requirement for a single Data Domain Boost user account across all Data Domain devices, this now allows for better tightening of access.

One of my gripes with BBB (Block Based Backup) in NetWorker 8.1 has been addressed in 8.2 – if you’re stuck using ADV_FILE devices rather than Data Domain, you can now perform BBB even if the storage node being written to is not Windows. Another time-saving option that was introduced in 8.1, Parallel Save Stream (PSS), has been extended to support Windows systems, and has also been updated to support Synthetic and Virtual Synthetic Fulls. in 8.1 it had only supported Unix/Linux, and only in traditional backup models.

Continuing the trend towards storage nodes being seen as a fluid rather than locked resource mapping, there’s now an autoselect storage node option, which if enabled allows NetWorker to select the storage node itself during backup and recovery operations. If this is enabled, it will override any storage node preferences assigned to individual clients, and NetWorker looks for local storage nodes wherever possible.

There’s a few things that have left NetWorker in 8.2, which are understandable: Support for Windows XP, Windows 2003 and the Change Journal Manager. If you still to protect Windows XP or Windows Server 2003, be sure to keep your installers for 8.1. and lower client software around.

There’s some documentation updates in NetWorker 8.2 as well:

  • Server Disaster Recovery and Availability Best Practices – This describes the disaster recovery process for the NetWorker server, including best practices for ensuring you’re prepared for a disaster recovery situation.
  • Snapshot Management for NAS Devices Integration – This documents the aforementioned NSM for NAS new feature of NetWorker.
  • Upgrading to NetWorker 8.2 from a Previous Release – This covers off in fairly comprehensive detail how you can upgrade your NetWorker environment to 8.2.

In years gone by I’ve found that documentation updates have been a lagging component of NetWorker, but that’s long since disappeared. With each new version of NetWorker now we’re seeing either entirely new documents, or substantially enhanced documentation (or both). This speaks volumes of the commitment EMC has to NetWorker.

Jan 092012

Upgrading NetWorker

So a new version of NetWorker has come out, or is coming out, and it’s been decided that you’re going to upgrade, but you want a few tips for making that upgrade as painless as possible. Here’s my 5 rules for upgrading NetWorker:

  1. Read the release notes. If you’re not going to read the release notes, you are better off staying on your current version, no matter what issues you’re having. I can’t stress enough the importance of reading the release notes and having a thorough grasp of:
    • What has changed?
    • What are the known issues with the current release?
    • What were the resolved issues between the current release and the release you’re currently running?
  2. Do a bootstrap and index backup if upgrading between major or minor releases. If going between service packs on the same release, you can skip the index backup so long as your backups have been successful lately, but ensure you still do a bootstrap backup.
  3. Unload all tapes (physical or virtual) in jukeboxes before the upgrade. You’ll see why shortly.
  4. Upgrade in this order:
    • Storage node(s) on the day of the upgrade, before the NetWorker server
    • Server on the day of the upgrade, after the storage node(s)
    • Client(s) later, at suitable times
  5. After the upgrade but before the NetWorker services are restarted on the storage node(s) and server, delete the nsr/tmp directory on those hosts.

Obviously standard caveats, such as following any additional instructions in the release notes or upgrade notes should of course be followed, but sticking to the above rules as well can save a lot of hassle over time. I’ve noticed over the years that a odd, random problems following upgrades can be solved by clearing the nsr/tmp directory on the server and storage nodes. If there’s no tapes in the jukeboxes when the services first start after the upgrade, there’s less futzing for NetWorker to take care of before it’s fully up and running, too.

Nov 102010

Just to let you folks know that I’m currently just “mostly missing in action” rather than “missing in action”.

It’s been a hectic few weeks for me, both personally and work-wise, and as my role adjusts to some changes I’ll be doing more travel again which will mean getting used to doing posts on the run.

I’m currently in (not so) sunny Western Australia – specifically, Bunbury, doing some work for a long-term customer, and flying back on Friday. That’ll give me Saturday and Sunday to recharge my batteries and get some Greek coffee into my system. (Maybe, if my friendly tattoo artist will help out, maybe Sunday will also see me getting some new ink. Though maybe that won’t happen if I don’t remember to email her – hmmm…)

Next article: Last week I attended the SNIA AU/NZ first storage blogfest – where myself and a group of other bloggers/tweeters were taken to EMC, IBM, HDS and NetApp for some discussions and technology direction overviews. I had hoped to blog about it by now – but instead I’m aiming to have blogged about it by the time I leave WA – that’ll mean either tonight, or tomorrow night.

Stay tuned – I’m not MIA, just MMIA.

Jan 122010

On the NetWorker Mailing List, I still frequently see a lot of posts from people who are having various problems with their NetWorker 7.2.x servers.

It’s time to move away from 7.2. I know, it was the last version before nsrjobd; the move to nsrjobd in 7.3, then raw daemon logs in 7.4 can both be a bit shocking, but 7.2 is now critically old and critically out of support. Equally, there’s still a lot of people out there running 7.3 releases of NetWorker. That, too, exited support some time ago, and it’s time to move on from it too.

I’ll agree that within backup, there is a strong logic to the statement “if it ain’t broke, don’t fix it”, but, you have to weigh up that against the simple fact that 7.2.x releases in particular are very old, and 7.3.x releases are fairly aged as well.

Since I’ve been watching more and more of Top Gear, I’ll use a car analogy. Let’s say you’ve got a brand new, top of the line Ferrari. When it needs servicing, do you take it to the official Ferrari shop that provides a 100% warranty on all repairs and whose repairs keep the original vehicle warranty intact, or do you take it to Bill & Joes Motor Fixits ‘R’ Us, who not only might leave you with a car in a worse condition than when you drove it in, but who aren’t certified by Ferrari and thus lose you your new car warranty?

Continuing to backup your environment with a backup product which is long out of support is like outsourcing to Bill & Joes Motor Fixits ‘R’ Us IT Service.

I’ll be the first to admit that even on simple updates you can run into a few hassles. Particularly as you move up the NetWorker version chain you’ll find changes to authentication and name resolution requirements alone that may necessitate some additional work around the time of the update. If your clients are old you’ll also be needing to plan an update for them as soon as possible too, and in some cases, you may find yourself definitely having to update clients if there turns out to be some particularly odd issue.

But I’ll be honest: that little bit of up-front pain is much, much better than hitting a critical backup or recovery problem that can’t be solved without upgrading (or worse, can’t be solved due to incompatibilities between ancient NetWorker versions and modern operating system versions). Planning and implementing a controlled upgrade, even if it does end up having a few hassles, is infinitely better than doing an emergency upgrade without any planning to facilitate a recovery or a backup that has to be done.

Basics – Updates vs Upgrades

 Basics, Features, NetWorker  Comments Off on Basics – Updates vs Upgrades
Aug 182009

After 13+ years of using NetWorker, I still tend to interchangeably use the terms ‘upgrade’ and ‘update’ (or to be more precise, mainly use the term ‘upgrade’).

However, there is, and always has been, a difference between the two terms in NetWorker nomenclature, and it’s useful knowing it in case you’re being asked to qualify your environment to a support person.

Here’s what they mean for NetWorker:

  • An upgrade is transitioning from one licensed feature set to a more advanced licensed feature set. For example, you might upgrade from NetWorker, Network Edition to NetWorker, Power Edition. Previously (when tiered licensing was still used for Windows modules), you might upgrade from say, Exchange Module Tier 1 to Exchange Module Tier 2. Alternatively, you can buy an upgrade to slot capacity for an Autochanger license (e.g., upgrading from a 1-64 slot license to a 1-128 slot license).
  • An update is where you change the version of a NetWorker product. E.g., you update from NetWorker 7.4.4 to NetWorker 7.4.5, or from NetWorker 7.3 to NetWorker 7.5.1. You would equally update from Oracle Module 4.5 to Oracle Module 5.

Since in both support and data protection it’s useful to avoid ambiguities, understanding the difference between these two terms can be important.

Upgrading NetWorker and the ‘tmp’ directory

 NetWorker  Comments Off on Upgrading NetWorker and the ‘tmp’ directory
Jul 102009

Over the years I’ve been forced to come to one key conclusion about the NetWorker ‘tmp’ directory and NetWorker upgrades.

While it’s not stated in any upgrade documentation, and in theory it shouldn’t matter, anyone upgrading their NetWorker server software should, as a matter of course, delete the server’s NetWorker ‘tmp’ directory prior to starting the new version of the software.

If you’re not familiar with this, you’ll find it:

  • On Unix/Linux platforms as: /nsr/tmp
  • On Windows platforms in the NetWorker install directory, as “Legatonsrtmp”.

This directory contains a plethora of files, some relating to savegroups, some relating to operations – in general ‘tmp’ is almost a poor choice of nomenclature for it: I like to think of it as a ‘state’ directory instead.

Nine times out of ten, or perhaps even 49 times out of 50, if I do an upgrade of a NetWorker server and forget to delete the ‘tmp’ directory, what can only be described as weird stuff will happen within 72 hours. Groups may not finish. Media may not unload. Libraries may forget state. The mouse on your desk may spontaneously quantum ooze to the floor. You may hear the Twilight Zone theme music playing in the background when it should be entirely quiet – that sort of stuff.

So, if looking for a new rule to follow when upgrading NetWorker on your server, please make sure you delete the NetWorker ‘tmp’ directory. You’ll save yourself a lot of time and hassles.

(Note: Deleting the NetWorker server ‘tmp’ directory prevents any backup that previously failed or was stopped from being restarted after the failure – it will need to be started as a whole new operation.)