Jan 132017
 

Introduction

There’s something slightly deceptive about the title for my blog post. Did you spot it?

It’s: vs. It’s a common mistake to think that Cloud Boost and Cloud Tier compete with one another. That’s like suggesting a Winnebago and a hatchback compete with each other. Yes, they both can have one or more people riding in them and they can both be used to get you around, but the actual purpose of each is typically quite different.

It’s the same story when you look at Cloud Boost and Cloud Tier. Of course, both can move data from A to B. But the reason behind each, the purpose for each is quite different. (Does that mean there’s no overlap? Not necessarily. If you need to go on a 500km holiday and sleep in the car, you can do that in a hatchback or a Winnebago, too. You can often get X to do Y even if it wasn’t built with that in mind.)

So let’s examine them, and look at their workflows as well as a few usage examples.

Cloud Boost

First off, let’s consider Cloud Boost. Version 1 was released in 2014, and since then development has continued to the point where CloudBoost now looks like the following:

CloudBoost Workflow

Cloud Boost Workflow

Cloud Boost exists to allow NetWorker (or NetBackup or Avamar) to write deduplicated data out to cloud object storage, regardless of whether that’s on-premises* in something like ECS, or writing out to a public cloud’s object storage system, like Virtustream Storage or Amazon S3. When Cloud Boost was first introduced back in 2014, the Cloud Boost appliance was also a storage node and data had to be cloned from another device to the Cloud Boost storage node, which would push data out to object. Fast forward a couple of years, and with Cloud Boost 2.1 introduced in the second half of 2016, we’re now at the point where there’s a Cloud Boost API sitting in NetWorker clients allowing full distributed data processing, with each client talking directly to the object storage – the Cloud Boost appliance now just facilitates the connection.

In the Cloud Boost model, regardless of whether we’re backing up in a local datacentre and pushing to object, or whether all the systems involved in the backup process are sitting in public cloud, the actual backup data never lands on conventional block storage – after it is deduplicated, compressed and encrypted it lands first and only in object storage.

Cloud Tier

Cloud Tier is new functionality released in the Data Domain product range – it became available with Data Domain OS v6, released in the second half of 2016. The workflow for Cloud Tier looks like the following:

CloudTier Workflow

CloudTier Workflow

Data migration with Cloud Tier is handled as a function of the Data Domain operating system (or controlled by a fully integrated application such as NetWorker or Avamar); the general policy process is that once data has reached a certain age on the Active Tier of the Data Domain, it is migrated to the Cloud Tier without any need for administrator or user involvement.

The key for the differences – and the different use cases – between Cloud Boost and Cloud Tier is in the above sentence: “once data has reached a certain age on the Active Tier”. In this we’re reminded of the primary use case for Cloud Tier – supporting Long Term Retention (LTR) in a highly economical format and bypassing any need for tape within an environment. (Of course, the other easy differentiator is that Cloud Tier is a Data Domain feature – depending on your environment that may form part of the decision process.)

Example use cases

To get a feel for the differences in where you might deploy Cloud Boost or Cloud Tier, I’ve drawn up a few use cases below.

Cloning to Cloud

You currently backup to disk (Data Domain or AFTD) within your environment, and have been cloning to tape. You want to ensure you’ve got a second copy of your data, and you want to keep that data off-site. Instead of using tape, you want to use Cloud object storage.

In this scenario, you might look at replacing your tape library with a Cloud Boost system instead. You’d backup to your local protection storage, then when it’s time to generate your secondary copy, you’d clone to your Cloud Boost device which would push the data (compressed, deduplicated and encrypted) up into object storage. At a high level, that might result in a workflow such as the following:

CloudBoost Clone To Cloud

CloudBoost Clone To Cloud

Backing up to the Cloud

You’re currently backing up locally within your datacentre, but you want to remove all local backup targets.  In this scenario, you might replace your local backup storage with a Cloud Boost appliance, connected to an object store, and backup via Cloud Boost (via client direct), landing data immediately off-premises and into object storage at a cloud provider (public or hosted).

At a high level, the workflow for this resembles the following:

CloudBoost Backup to Cloud

CloudBoost Backup to Cloud

Backing up in Cloud

You’ve got some IaaS systems sitting in the Cloud already. File, web and database servers sitting in say, Amazon, and you need to ensure you can protect the data they’re hosting. You want greater control than say, Amazon snapshots, and since you’re using a NetWorker Capacity license or a DPS capacity license, you know you can just spin up another NetWorker server without an issue – sitting in the cloud itself.

In that case, you’d spin up not only the NetWorker server but a Cloud Boost appliance as well – after all, Amazon love NetWorker + Cloud Boost:

“The availability of Dell EMC NetWorker with CloudBoost on AWS is a particularly exciting announcement for all of the customers who have come to depend on Dell EMC solutions for data protection in their on-premises environments,” said Bill Vass, Vice President, Technology, Amazon Web Services, Inc. “Now these customers can get the same data protection experience on AWS, providing seamless operational backup and recovery, and long-term retention across all of their environments.”

That’ll deliver the NetWorker functionality you’ve come to use on a daily basis, but in the Cloud and writing directly to object storage.

The high level view of the backup workflow here is effectively the same as the original diagram used to introduce Cloud Boost.

Replacing Tape for Long Term Retention

You’ve got a Data Domain in each datacentre; the backups at each site go to the local Data Domain then using Clone Controlled Replication are copied to the other Data Domain as soon as each saveset finishes. You’d like to replace tape for your long term retention, but since you’re protecting a lot of data, you want to push data you rarely need to recover from (say, older than 2 months) out to object storage. When you do need to recover that data, you want to absolutely minimise the amount of data that needs to be retrieved from the Cloud.

This is a definite Cloud Tier solution. Cloud Tier can be used to automatically extend the Data Domain storage, providing a storage tier for long term retention data that’s very cheap and highly reliable. Cloud Tier can be configured to automatically migrate data older than 2 months out to object storage, and the great thing is, it can do it automatically for anything written to the Data Domain. So if you’ve got some databases using DDBoost for Enterprise Apps writing directly, you can setup migration policies for them, too. Best of all, when you do need to recall data from Cloud Tier, Boost for Enterprise Apps and NetWorker can handle that recall process automatically for you, and the Data Domain only ever recalls the delta between deduplicated data already sitting on the active tier and what’s out in the Cloud.

The high level view of the workflow for this use case will resemble the following:

Cloud Tier to LTR NSR+DDBEA

Cloud Tier to LTR for NetWorker and DDBEA

…Actually, you hear there’s an Isilon being purchased and the storage team are thinking about using Cloud Pools to tier really old data out to object storage. Your team and the storage team get to talking and decide that by pooling the protection and storage budget, you get Isilon, Cloud Tier and ECS, providing oodles of cheap object storage on-site at a fraction of the cost of a public cloud, and with none of the egress costs or cloud vendor lock-in.

Wrapping Up

Cloud Tier and Cloud Boost are both able to push data into object storage, but they don’t have exactly the same use cases. There’s good, clear reasons why you would work with one in particular, and hopefully the explanation and examples above has helped to set the scene on their use cases.


* Note, ‘on-premise’ would mean ‘on my argument’. The correct term is ‘on-premises’ 🙂

Dec 232016
 

I know, I know, it’s winter up there in the Northern Hemisphere, but NetWorker 9.1 is landing and given I’m in Australia, that makes NetWorker 9.1 a Summer Fresh release. (In fact, my local pub for the start of summer started doing a pale ale infused with pineapple and jalapeños, and that’s sort of reminding me of NetWorker 9.1: fresh, light and inviting you to put your heels up and rest a while.)

NetWorker 9.1

 

NetWorker 9 was a big – no, a huge – release. It’s a switch to a more service catalogue driven approach to backups, Linux block based filesystem backups, block based application backups, deep snapshot integration and more recently in NetWorker 9.0 SP1, REST API control as well.

NetWorker 9.1 as you’d expect is a smaller jump from 9.0 than we had from 8.2 to 9.0. That being said, it’s introduced some excellent new features:

  • VMAX SmartSnap integration – the ability to backup and restore a VMAX device based on the device WWN, increasing the depth of snapshot support in NetWorker further.
  • Snapshot Alternate Location Rollback – this lets you do a snapshot rollback, but to a different set of devices.
  • Data Domain High Availability integration – Data Domain now supports high-availability on the earlier 9500 platform, in addition to the 9800, 9300 and 6800 systems. And with v9.1, NetWorker fully understands and integrates with DDHA platforms.
  • Cloud Tier Integration – NetWorker gets deep integration into the Cloud Tier functionality introduced in Data Domain OS 6.0. This lets NetWorker cloning policies control the migration of data out to the Cloud Tier, and more seamlessly integrate with the recall process.

Cloud Tier integration is more than just a tick in the box to though. Consider the module space – NetWorker Module for Microsoft Applications, for instance, doesn’t just get the option to recover data from Cloud Tier, but also perform granular recoveries from Cloud Tier – SQL table level recoveries and Exchange granular recoveries as well.


By the way, the NetWorker Usage Survey is still running – don’t forget to fill in how you’re using NetWorker! (And be in the running for a prize.)


I’ve saved the best – and biggest – feature for last, though. This is a doozy. Say goodbye to needing a EBR/VBA for VMware backups. That EBR/VBA functionality is now embedded in the NetWorker server itself, leaving you to just deploy some very lightweight proxies to handle the data transport processes, all controlled by NetWorker.

The current EBR appliance and proxies will continue to work with NetWorker 9.1, but I can’t think of anyone who’d want to upgrade to 9.1 without rapidly transitioning to the new platform. Here are just some of the advantages of the new process:

  • Less virtual infrastructure required – no EBRs
  • Virtual machines stored in raw VMDK file – no additional processing required for the backup, and this will also mean faster instant access processes, too
  • The FLR web GUI now runs on the NetWorker server itself
  • NMC can be used for FLR instead of the web GUI, making it more accessible to the NetWorker administrators if they don’t have access to the virtual machines being protected
  • Proxies support more concurrent virtual machine backups:
    • Maximum 25 concurrent hotadd operations;
    • Maximum 25 concurrent NBD operations
  • Significantly increased File Level Recovery (FLR) counts from VMware Image Level Backups (recommended 20,000 – more on that in a minute)
  • Significantly faster FLR operations.

In fact, I’m going to spend a little bit of time on FLR for this post, and step through the new NMC-based FLR process to give you an overview of the process. This is using the newly deployed NetWorker VMware Protection (NVP) system, with backup to and recovery from Data Domain virtual edition.

Fig 01: Starting a recovery in NMC

Fig 01: Starting a recovery in NMC

You start by telling NMC you want to do a virtual machine recovery and choose the vCenter server that owns the virtual machine(s) you want to recover data from.

Fig 02: Choosing the virtual machine to recover from

Fig 02: Choosing the virtual machine to recover from

There’s various options for choosing the virtual machine to recover data for – you can enter the name directly, search for it, browse the various backups that have been performed, or browse the vCenter server itself.

Fig 03: Virtual Machine selected

Fig 03: Virtual Machine selected

Once you’ve selected a virtual machine for recovery, you can click Next to choose the backup to recover from.

Fig 04: Choosing the backup to recover from

Fig 04: Choosing the backup to recover from

In this case, I only had a single backup under the new NVP system for that virtual machine, so I was able to just click Next to continue the process. At this point you get to choose the type of recovery you want to perform:

Fig 05: Choosing the type of recovery to perform

Fig 05: Choosing the type of recovery to perform

As you can see, there’s a gamut of recovery options for virtual machines within NMC. I’m focusing on the FLR options here so I chose the bottom option and clicked Next.

Fig 06: Choosing backup instance to recover from

Fig 06: Choosing backup instance to recover from

Next you get to choose the backup instance you want to recover from. If the backup has been cloned it may be that there’s topologically a better backup to recover from than the original, and choosing an alternate is as simple as scrolling through a list of clones.

At that point you get to choose where you want to recover to:

Fig 07: Choosing where to recover data to

Fig 07: Choosing where to recover data to

Next, you’ll supply appropriate credentials for the virtual machine to be able to perform the recovery and initiate a mount of the backup into the proxy server:

Fig 08: Supplying virtual machine credentials to mount the backup

Fig 08: Supplying virtual machine credentials to mount the backup

After you’ve supplied the credentials you’ll click “Start Mount” to make the specific backup available for recovery purposes, and after a few seconds that’ll result in log information such as:

Fig 09: Mounted and ready

Fig 09: Mounted and ready

When the mount is done, you’re ready to click Next and start browsing files for recovery.

Fig 10: Choosing files to recover from an image level backup

Fig 10: Choosing files to recover from an image level backup

In this example, I selected a directory with about 7,800 files in it and the marking of files for recovery took just a few seconds to complete. After which, Next to choose where to recover the data to on the selected virtual machine:

Fig 11: Choosing where to recover data to on the virtual machine

Fig 11: Choosing where to recover data to on the virtual machine

In this case I choose to recover to C:\tmp on the virtual machine. Clicking Next allows finalisation of the recovery preparation:

Fig 12: Finalising the recovery configuration

Fig 12: Finalising the recovery configuration

As you would expect with the tightly integrated controls now, FLR is fully visible within the NetWorker environment – even nsrwatch:

Fig 13: FLR in progress shown in nsrwatch

Fig 13: FLR in progress shown in nsrwatch

And finally we have a completed recovery:

Fig 14: Completed recovery

Fig 14: Completed recovery

That’s 7,918 files recovered from an image level backup in 54 seconds:

Fig 15: Recovered content

Fig 15: Recovered content

I wanted to check out the FLR capabilities a little more and decided to risk pushing the system beyond the recommendations. Instead of just recovering a single folder with 7,900 files or thereabouts, I elected to recover the entire E:\ drive on the virtual machine – comprising over 47,000 files. Here’s the results:

Fig 16: Large scale FLR results

Fig 16: Large scale FLR results

The recovered folder:

Fig 17: Recovered Content

Fig 17: Recovered Content

47,198 files, 1,488 folders, 5.01GB of data recovered as an FLR from an image level backup in just 5 minutes and 42 seconds.

If you’re using NetWorker for VMware backups, here’s the version you want to be on.

You can get it from the EMC Support page for NetWorker today.

Nov 302016
 

Folks, it’s that time of the year again! Each year I run a survey to gauge NetWorker usage patterns – how many clients you’ve got, what plugins you’re using, whether you’re using deduplication, and a plethora of other questions. The survey runs from December 1 (ish) through to January 31 the next year. (This year I’m kicking it off on November 30, just because I have time.)

Take the survey!

That gets assembled into a report in February of the following year, reporting trends across the various years the NetWorker survey has been conducted. If you’d like to see what the reports look like, you can view last year’s report here.

You can fill out the survey anonymously if you’d like, but if you submit your email address at the end you’ll be in the running for a copy of my upcoming book, Data Protection: Ensuring Data Availability, due out February 2017. (Last year’s winner hasn’t been forgotten – the book just got delayed.) If you submit your email address, it will not be used for any purpose other than to notify you if you’re the winner.

The survey is closed now. Results will be published in February 2017.

Basics – Understanding NetWorker Architecture

 Architecture, Basics, NetWorker  Comments Off on Basics – Understanding NetWorker Architecture
Nov 142016
 

With the NetWorker 9 architecture now almost 12 months old, I thought it was long past time I do a Basics post covering how the overall revised architecture for data protection with NetWorker functions.

There are two distinct layers of architecture I’ll cover off – Enterprise and Operational. In theory an entire NetWorker environment can be collapsed down to a single host – the NetWorker server, backing up to itself – but in practice we will typically see multiple hosts in an overall NetWorker environment, and as has been demonstrated by the regular NetWorker Usage Surveys, it’s not uncommon nowadays to see two or more NetWorker servers deployed in a business.

Enterprise Layer

The Enterprise Layer consists of the components that technically sit ‘above’ any individual NetWorker install within your environment, and can be depicted simply with the following diagram:

Enterprise Layer

The key services that will typically be run within the Enterprise Layer are the NetWorker License Server, and the NetWorker Management Console Server (NMC Server). While NetWorker has previously had the option of running an independent license server, with NetWorker 9 this has been formalised, and the recommendation is now to run a single license server for all NetWorker environments within your business, unless network or security rules prevent this.

The License server can be used by a single NetWorker server, or if you’ve got multiple NetWorker servers, by each NetWorker server in your environment, allowing all licenses to be registered against a single host, reducing ‘relicensing’ requirements if NetWorker server details change, etc. This is a very light-weight server, and it’s quite common to the license services run concurrently on the same host as the NMC Server.

Like many applications, NetWorker has separated the GUI management from the core software functionality. This has multiple architectural advantages, such as:

  • The GUI and the Server functionality can be developed with more agility
  • The GUI can be used to administer multiple servers
  • The functional load of providing GUI services does not impact the core Server functionality (i.e., providing backup and recovery services).

While you could, if you wanted to, deploy a NMC Server for each NetWorker Server, it’s by no means necessary, and so it’s reasonably common to see a single NMC Server deployed across multiple NetWorker servers. This allows centralised reporting, management and control for backup administrators and operators.

Operational Layer

At the operational layer we have what is defined as a NetWorker datazone. In fact, at the operational layer we can have as many datazones as is required by the business, all subordinate to the unified Enterprise Layer. In simple parlance, a NetWorker datazone is the collection of all hosts within your environment for which a single NetWorker server provides backup and recovery services. A high level view of a NetWorker datazone resembles the following:

NetWorker Datazone (Operational Layer)

The three key types of hosts within a NetWorker datazone are as follows:

  • Server – A host that provides backup and recovery services (with all the associated management functions) for systems within your environment. There will either be (usually) a single NetWorker server in the datazone, or (in less common situations), a clustered pair of hosts acting as an active/passive NetWorker server.
  • Client – Any system that has backup and recovery services managed by a NetWorker Server
  • Storage Node – A host with access to one or more backup devices, either providing device mapping access to clients (I’ll get to that in a moment) or transferring backup/recovery to/from devices on behalf of clients. (A NetWorker server, by the way, can also function as a storage node.) A storage node can either be a full storage node, meaning it can perform those actions previously described for any number of clients, or a dedicated storage node, meaning it provides those services just to itself.

With such a long pedigree, NetWorker (as described above) is capable of running in a classic three-tier architecture – the server managing the overall environment with clients backing up to and recovering from storage nodes. However, NetWorker is equally able to ditch that legacy mode of operation and function without storage nodes thanks to the benefits of distributed deduplication in tightly integrated systems such as Data Domain and CloudBoost and ClientDirect. That being said, NetWorker still supports a broad range of device types ranging from simple tape through to purpose built backup appliances (Data Domain), Cloud targets, VTL and plain disk. (In fact, I remember years ago NetWorker actually supporting VHS as a tape format!)

ClientDirect, which I mentioned previously, is where clients can communicate directly with network accessible devices such as Data Domain deduplication storage. In these cases, both the NetWorker server and any storage node in the environment is removed from the data path – making for a highly efficient and scalable environment when distributed deduplciation is taking place. (For a more in-depth understanding of the architectural implications of Client Direct, I suggest you review this earlier post.)

Within this operational layer, I’ve drawn the devices off to the side for the following reasons:

  • Devices can (and will) provide backup/recovery media to all layers in the NetWorker datazone – server, storage nodes (if deployed) and individual clients
  • Devices that support appropriate multi-tenancy or partitioning can actually be shared between multiple NetWorker datazones. In years gone by you might have deployed a large tape library with two or more NetWorker servers accessing virtualised autochangers from it, and more recently it’s quite easy to have the same Data Domain system for instance being accessed by multiple NetWorker servers if you want to.

Wrapping Up

The NetWorker architecture has definitely grown since I started using it in 1996. Back then each datazone required completely independent licensing and management, using per-OS-type GUI interfaces or CLI, and it was a very flattened architecture – clients and the server only. Since then the architecture has grown to accommodate the changing business landscape. My largest NetWorker datazone in 1996 had approximately 50 clients in it – these days I have customers with well over 2,000 clients in a single datazone, and have colleagues with customers running even larger environments. As the NetWorker Usage Survey has shown, the number of datazones has also been growing as businesses merge, consolidate functions, and take advantage of simplified capacity based licensing schemes.

By necessity then, the architecture available to NetWorker has grown as well. Perhaps the most important architectural lesson for newcomers to NetWorker is understanding the difference between the enterprise layer and the operational layer (the individual datazones).

If you’ve got any questions on any of the above, drop me a line or add a comment and I’ll clarify in a subsequent post.

Jul 112016
 

Overview

As I mentioned in the previous article, NetWorker 9 SP1 has introduced a REST API. I’ve never previously got around to playing with REST API interfaces, but as is always the case with programming, you either do it because you’re getting paid to or because it’s something that strikes you as interesting.

Accessing NetWorker via a REST API does indeed strike me as interesting. Even more so if I can do it using my favourite language, Perl.

This is by no means meant to be a programming tutorial, nor am I claiming to be the first to experiment with it. If you want to check out an in-development use of the REST API, check out Karsten Bott’s PowerShell work to date over at the NetWorker Community Page. This post covers just the process of bootstrapping myself to the point I have working code – the real fun and work comes next!

REST API

What you’ll need

For this to work, you’ll need a suitably recent Perl 5.x implementation. I’m practicing on my Mac laptop, running Perl 5.18.2.

You’ll also need the following modules:

  • MIME::Base64
  • REST::Client
  • Data::Dumper
  • JSON

And of course, you’ll need a NetWorker server running NetWorker 9, SP1.

Getting started

I’m getting old an crotchety when it comes to resolving dependencies. When I was younger I used to manually download each CPAN module I needed, try to compile, strike dependency requirements, recurse down those modules and keep going until I’d either solved all the dependencies or threw the computer out the window and became a monk.

So to get the above modules I invoked the cpan install function on my Mac as follows:

pmdg@ganymede$ cpan install MIME::Base64
pmdg@ganymede$ cpan install REST::Client
pmdg@ganymede$ cpan install Data::Dumper
pmdg@ganymede$ cpan install JSON

There was a little bit of an exception thrown in the REST::Client installation about packages that could be used for testing, but overall the CPAN based installer worked well and saved me a lot of headaches.

The code

The code itself is extremely simple – as I mentioned this is a proof of concept, not intended to be an interface as such. It’s from here I’ll start as I play around in greater detail. My goal for the code was as follows:

  • Prompt for username and password
  • Connect via REST API
  • Retrieve a complete list of clients
  • Dump out the data in a basic format to confirm it was successful

The actual code therefore is:

pmdg@ganymede$ cat tester.pl

#!/usr/bin/perl -w

use strict;
use MIME::Base64();
use REST::Client;
use Data::Dumper;
use JSON;

my $username = "";
my $password = "";

print "Username: ";
$username = <>;
chomp $username;

print "Password: ";
$password = <>;
chomp $password;

my $encoded = MIME::Base64::encode($username . ":" . $password);
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
my $client = REST::Client->new();
my $headers = { Accept => 'application/json', Authorization => 'Basic ' . $encoded};
$client->setHost('https://orilla.turbamentis.int:9090');
$client->GET('/nwrestapi/v1/global/clients',$headers);
my $response = from_json($client->responseContent);
print Dumper($response);

Notes on the Code

If you’re copying and pasting the code, about the only thing you should need to change is the hostname in the line starting $client->setHost.

It’s not particularly secure in the password prompt as Perl will automatically echo the password as you’re entering it. There are ways of disabling this echo, but they require the Term::Readkey library and that may not be readily available on all systems. So just keep this in mind…

The Results

Here’s the starting output for the code:

pmdg@ganymede$ ./tester.pl
Username: administrator
Password: MySuperSecretPassword
$VAR1 = {
          'clients' => [
                         {
                           'ndmpMultiStreamsEnabled' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
                           'ndmpVendorInformation' => [],
                           'protectionGroups' => [],
                           'resourceId' => {
                                                'sequence' => 79,
                                                'id' => '198.0.72.12.0.0.0.0.132.105.45.87.192.168.100.4'
                                           },
                           'links' => [
                                        {
                                           'rel' => 'item',
                                           'href' => 'https://orilla.turbamentis.int:9090/nwrestapi/v1/global/clients/198.0.72.12.0.0.0.0.132.105.45.87.192.168.100.4'
                                        }
                                      ],
                           'parallelSaveStreamsPerSaveSet' => $VAR1->{'clients'}[0]{'ndmpMultiStreamsEnabled'},
                           'hostname' => 'archon.turbamentis.int',

And so on…

In Summary

The script isn’t pretty at the moment, but I wanted to get it out there as an example. As I hack around with it and get more functionality, I’ll provide updates.

Hopefully however you can see that it’s pretty straight-forward overall to access the REST API!

References

Mar 302016
 

At the start of the week we saw NetWorker 8.2 SP3 released. Now, you might think given NetWorker 9 is out there’s no new features in NetWorker 8.2 SP3, but you’d be wrong.

NetWorker 9 is a jump – it’s a change of processes and it’s a new way of going about configuring your backups. I’m seeing more details every day of people having great experiences with NetWorker 9, but backup is one of those areas where change can often come slowly, so 8.2 still gets a lot of attention. So if you’re the sort of business that needs the features in NetWorker 9 you can dive in, but if you want to hang back for a little while yet, 8.2 will have you covered for a while yet.

nsrwatch in 8.2.3

The all new nsrwatch

OK, I admit I’m a bit of an nsrwatch junkie. Unless I have to setup a Windows NetWorker server I’ll setup NetWorker on Linux every time. (But at heart I’m still a Unix system administrator. It was the Unix integration that drove me to Mac OS X, after all.)

There’s a lot of good new features in NetWorker 8.2 SP3 but I have to admit given my CLI-junkie status, I just love the update to nsrwatch. For me this handy little utility has saved me thousands or more times from having to launch a full GUI, and if you’ve ever seen how many windows I end up having active on my screens at the same time you’ll understand why that’s a good thing.

The good old nsrwatch utility now gives you a lot more control over what you see on-screen. You can resize panels or even turn them off and setup an environment variable to make that your default view. You can switch between different views – e.g., all devices (seen above), mounted devices and active devices:

Mounted Devices

nsrwatch showing mounted devices only

nsrwatch showing active devices

nsrwatch showing active devices

You also get control options directly embedded into nsrwatch now:

nsrwatch with control options

nsrwatch with control options

All up, a great set of changes. I was lucky enough to try out some of the options while they were under development, so I’ve been looking forward to talking about it for some time now!

That’s not the only features in NetWorker 8.2 SP3 though – but it did really appeal to my I’ve-been-using-NetWorker-for-20-years inner-geek – so it’s time to move on to the rest of the enhancements!

Server Capability

There’s big changes under the hood in SP3 – the media catalogue has been migrated to SQLite to take advantage of the huge performance increases this gave in NetWorker 9 – and it’ll make the migration path to NetWorker 9 a little more streamlined as well. This may sound like a minor change, but the switch to SQLite is really important; the old format media database was great and stable, but it had limits on the amount of concurrent operations you could do. SQLite is great and stable and a lot more capable of supporting a number of concurrent operations.

The server daemons have had some tweaks as well – a bunch of issues that could lead to a server hang situation have been quashed, and the number of DNS reverse lookups performed has been pared down. The DNS caches used in a bunch of NetWorker daemons are now populated from nsrd to improve lookup performance as well. Also if you’ve got a lot of storage nodes in your environment, there are options to do a staggered start of the storage node manager daemons to improve startup performance.

Data Domain

8.2 SP3 includes support for DDOS 5.7 with an update of Data Domain libraries to 3.1. This will align it to some new options coming out soon, not to mention the Data Domain High Availability option introduced in the last month for the DD9500. (One of the other things it’ll align to I can blog about in a few days, hopefully.)

There’s performance enhancements for Clone Controlled Replication (CCR) as well, allowing for boosts (no pun intended) in the performance of cloning operations between two Data Domain systems under NetWorker control.

SP3 also introduces support for Distributed Segment Processing and all other Boost goodness into the Mac OS X client. That means if you’ve got some Mac clients within your NetWorker environment they’ll now get all of the Boost advantages you see everywhere else.

Updated Support

There’s a whole bunch of platforms and options that have had support added in this release. Check out the new VBA appliances if you’re backing VMware, too – you’ll definitely want to take advantage of updates there. But it’s not just VMware backups. This version of NetWorker also adds support for:

  • LTO7 tape drives
  • Mac OS X 10.11 El Capitan
  • SAP HANA SPS 11
  • Snapshot Management for NetApp SnapVault and SnapMirror ‘C-mode’ operations – creation, replication, restore and rollover
  • Hitachi NAS token based backups
  • Isilon Fast Incremental – Making backup of really large filesystems a whole lot easier
  • SQL Server AlwaysOn availability groups in a Failover Cluster (great way of offloading backups in SQL Enterprise Server environments)
  • MySQL 5.7.9/MySQL Enterprise Backup 4

In Summary

You won’t see the same sorts of massive features lists in a service pack release as you do in a full new release, but that being said 8.2 SP3 packs some wallop for your environment if you’re still in the 8.2 tree – or using an earlier version still. In addition to all the standard fixes that go into any service pack, rolled up from previous service packs and cumulative releases, 8.2 SP3 has been fine tuned for performance and scaleability and will ensure those customers not yet ready to upgrade to NetWorker 9 have an excellent platform to settle onto.

You can find the 8.2 SP3 binaries in the downloads section of the NetWorker product support page, and you can access the release notes directly from this link.

Dec 222015
 

As we approach the end of 2015 I wanted to spend a bit of time reflecting on some of the data protection enhancements we’ve seen over the year. There’s certainly been a lot!

Protection

NetWorker 9

NetWorker 9 of course was a big part to the changes in the data protection landscape in 2015, but that’s not by any means the only advancement we saw. I covered some of the advances in NetWorker 9 in my initial post about it (NetWorker 9: The Future of Backup), but to summarise just a few of the key new features, we saw:

  • A policy based engine that unites backup, cloning, snapshot management and protection of virtualisation into a single, easy to understand configuration. Data protection activities in NetWorker can be fully aligned to service catalogue requirements, and the easier configuration engine actually extends the power of NetWorker by offering more complex configuration options.
  • Block based backups for Linux filesystems – speeding up backups for highly dense filesystems considerably.
  • Block based backups for Exchange, SQL Server, Hyper-V, and so on – NMM for NetWorker 9 is a block based backup engine. There’s a whole swathe of enhancements in NMM version 9, but the 3-4x backup performance improvement has to be a big win for organisations struggling against existing backup windows.
  • Enhanced snapshot management – I was speaking to a customer only a few days ago about NSM (NetWorker Snapshot Management), and his reaction to NSM was palpable. Wrapping NAS snapshots into an effective and coordinated data protection policy with the backup software orchestrating the whole process from snapshot creation, rollover to backup media and expiration just makes sense as the conventional data storage protection and backup/recovery activities continue to converge.
  • ProtectPoint Integration – I’ll get to ProtectPoint a little further below, but being able to manage ProtectPoint processes in the same way NSM manages file-based snapshots will be a big win as well for those customers who need ProtectPoint.
  • And more! – VBA enhancements (notably the native HTML5 interface and a CLI for Linux), NetWorker Virtual Edition (NVE), dynamic parallel savestreams, NMDA enhancements, restricted datazones and scaleability all got a boost in NetWorker 9.

It’s difficult to summarise everything that came in NetWorker 9 in so few words, so if you’ve not read it yet, be sure to check out my essay-length ‘summary’ of it referenced above.

ProtectPoint

In the world of mission critical databases where impact minimisation on the application host is a must yet backup performance is equally a must, ProtectPoint is an absolute game changer. To quote Alyanna Ilyadis, when it comes to those really important databases within a business,

“Ideally, you’d want the performance of a snapshot, with the functionality of a backup.”

Think about the real bottleneck in a mission critical database backup: the data gets transferred (even best case) via fibre-channel from the storage layer to the application/database layer before being passed across to the data protection storage. Even if you direct-attach data protection storage to the application server, or even if you mount a snapshot of the database at another location, you still have the fundamental requirement to:

  • Read from production storage into a server
  • Write from that server out to protection storage

ProtectPoint cuts the middle-man out of the equation. By integrating storage level snapshots with application layer control, the process effectively becomes:

  • Place database into hot backup mode
  • Trigger snapshot
  • Pull database out of hot backup mode
  • Storage system sends backup data directly to Data Domain – no server involved

That in itself is a good starting point for performance improvement – your database is only in hot backup mode for a few seconds at most. But then the real power of ProtectPoint kicks in. You see, when you first configure ProtectPoint, a block based copy from primary storage to Data Domain storage starts in the background straight away. With Change Block Tracking incorporated into ProtectPoint, the data transfer from primary to protection storage kicks into high gear – only the changes between the last copy and the current state at the time of the snapshot need to be transferred. And the Data Domain handles creation of a virtual synthetic full from each backup – full backups daily at the cost of an incremental. We’re literally seeing backup performance improvements in the order of 20x or more with ProtectPoint.

There’s some great videos explaining what ProtectPoint does and the sorts of problems it solves, and even it integrating into NetWorker 9.

Database and Application Agents

I’ve been in the data protection business for nigh on 20 years, and if there’s one thing that’s remained remarkably consistent throughout that time it’s that many DBAs are unwilling to give up control over the data protection configuration and scheduling for their babies.

It’s actually understandable for many organisations. In some places its entrenched habit, and in those situations you can integrate data protection for databases directly into the backup and recovery software. For other organisations though there’s complex scheduling requirements based on batch jobs, data warehousing activities and so on which can’t possibly be controlled by a regular backup scheduler. Those organisations need to initiate the backup job for a database not at a particular time, but when it’s the right time, and based on the amount of data or the amount of processing, that could be a highly variable time.

The traditional problem with backups for databases and applications being handled outside of the backup product is the chances of the backup data being written to primary storage, which is expensive. It’s normally more than one copy, too. I’d hazard a guess that 3-5 copies is the norm for most database backups when they’re being written to primary storage.

The Database and Application agents for Data Domain allow a business to sidestep all these problems by centralising the backups for mission critical systems onto highly protected, cost effective, deduplicated storage. The plugins work directly with each supported application (Oracle, DB2, Microsoft SQL Server, etc.) and give the DBA full control over managing the scheduling of the backups while ensuring those backups are stored under management of the data protection team. What’s more, primary storage is freed up.

Formerly known as “Data Domain Boost for Enterprise Applications” and “Data Domain Boost for Microsoft Applications”, the Database and Application Agents respectively reached version 2 this year, enabling new options and flexibility for businesses. Don’t just take my word for it though: check out some of the videos about it here and here.

CloudBoost 2.0

CloudBoost version 1 was released last year and I’ve had many conversations with customers interested in leveraging it over time to reduce their reliance on tape for long term retention. You can read my initial overview of CloudBoost here.

2015 saw the release of CloudBoost 2.0. This significantly extends the storage capabilities for CloudBoost, introduces the option for a local cache, and adds the option for a physical appliance for businesses that would prefer to keep their data protection infrastructure physical. (You can see the tech specs for CloudBoost appliances here.)

With version 2, CloudBoost can now scale to 6PB of cloud managed long term retention, and every bit of that data pushed out to a cloud is deduplicated, compressed and encrypted for maximum protection.

Spanning

Cloud is a big topic, and a big topic within that big topic is SaaS – Software as a Service. Businesses of all types are placing core services in the Cloud to be managed by providers such as Microsoft, Google and Salesforce. Office 365 Mail is proving very popular for businesses who need enterprise class email but don’t want to run the services themselves, and Salesforce is probably the most likely mission critical SaaS application you’ll find in use in a business.

So it’s absolutely terrifying to think that SaaS providers don’t really backup your data. They protect their infrastructure from physical faults, and their faults, but their SLAs around data deletion are pretty straight forward: if you deleted it, they can’t tell whether it was intentional or an accident. (And if it was an intentional delete they certainly can’t tell if it was authorised or not.)

Data corruption and data deletion in SaaS applications is far too common an occurrence, and for many businesses sadly it’s only after that happens for the first time that people become aware of what those SLAs do and don’t cover them for.

Enter Spanning. Spanning integrates with the native hooks provided in Salesforce, Google Apps and Office 365 Mail/Calendar to protect the data your business relies on so heavily for day to day operations. The interface is dead simple, the pricing is straight forward, but the peace of mind is priceless. 2015 saw the introduction of Spanning for Office 365, which has already proven hugely popular, and you can see a demo of just how simple it is to use Spanning here.

Avamar 7.2

Avamar got an upgrade this year, too, jumping to version 7.2. Virtualisation got a big boost in Avamar 7.2, with new features including:

  • Support for vSphere 6
  • Scaleable up to 5,000 virtual machines and 15+ vCenters
  • Dynamic policies for automatic discovery and protection of virtual machines within subfolders
  • Automatic proxy deployment: This sees Avamar analyse the vCenter environment and recommend where to place virtual machine backup proxies for optimum efficiency. Particularly given the updated scaleability in Avamar for VMware environments taking the hassle out of proxy placement is going to save administrators a lot of time and guess-work. You can see a demo of it here.
  • Orphan snapshot discovery and remediation
  • HTML5 FLR interface

That wasn’t all though – Avamar 7.2 also introduced:

  • Enhancements to the REST API to cover tenant level reporting
  • Scheduler enhancements – you can now define the start dates for your annual, monthly and weekly backups
  • You can browse replicated data from the source Avamar server in the replica pair
  • Support for DDOS 5.6 and higher
  • Updated platform support including SLES 12, Mac OS X 10.10, Ubuntu 12.04 and 14.04, CentOS 6.5 and 7, Windows 10, VNX2e, Isilon OneFS 7.2, plus a 10Gbe NDMP accelerator

Data Domain 9500

Already the market leader in data protection storage, EMC continued to stride forward with the Data Domain 9500, a veritable beast. Some of the quick specs of the Data Domain 9500 include:

  • Up to 58.7 TB per hour (when backing up using Boost)
  • 864TB usable capacity for active tier, up to 1.7PB usable when an extended retention tier is added. That’s the actual amount of storage; so when deduplication is added that can yield actual protection data storage well into the multiple-PB range. The spec sheet gives some details based on a mixed environment where the data storage might be anywhere from 8.6PB to 86.4PB
  • Support for traditional ES30 shelves and the new DS60 shelves.

Actually it wasn’t just the Data Domain 9500 that was released this year from a DD perspective. We also saw the release of the Data Domain 2200 – the replacement for the SMB/ROBO DD160 appliance. The DD2200 supports more streams and more capacity than the previous entry-level DD160, being able to scale from a 4TB entry point to 24TB raw when expanded to 12 x 2TB drives. In short: it doesn’t matter whether you’re a small business or a huge enterprise: there’s a Data Domain model to suit your requirements.

Data Domain Dense Shelves

The traditional ES30 Data Domain shelves have 15 drives. 2015 also saw the introduction of the DS60 – dense shelves capable of holding sixty disks. With support for 4 TB drives, that means a single 5RU data Domain DS60 shelf can hold as much as 240TB in drives.

The benefits of high density shelves include:

  • Better utilisation of rack space (60 drives in one 5RU shelf vs 60 drives in 4 x 3RU shelves – 12 RU total)
  • More efficient for cooling and power
  • Scale as required – each DS60 takes 4 x 15 drive packs, allowing you to start with just one or two packs and build your way up as your storage requirements expand

DDOS 5.7

Data Domain OS 5.7 was also released this year, and includes features such as:

  • Support for DS60 shelves
  • Support for 4TB drives
  • Support for ES30 shelves with 4TB drives (DD4500+)
  • Storage migration support – migrate those older ES20 style shelves to newer storage while the Data Domain stays online and in use
  • DDBoost over fibre-channel for Solaris
  • NPIV for FC, allowing up to 8 virtual FC ports per physical FC port
  • Active/Active or Active/Passive port failover modes for fibre-channel
  • Dynamic interface groups are now supported for managed file replication and NAT
  • More Secure Multi-Tenancy (SMT) support, including:
    • Tenant-units can be grouped together for a tenant
    • Replication integration:
      • Strict enforcing of replication to ensure source and destination tenant are the same
      • Capacity quota options for destination tenant in a replica context
      • Stream usage controls for replication on a per-tenant basis
    • Configuration wizards support SMT for
    • Hard limits for stream counts per Mtree
    • Physical Capacity Measurement (PCM) providing space utilisation reports for:
      • Files
      • Directories
      • Mtrees
      • Tenants
      • Tenant-units
  • Increased concurrent Mtree counts:
    • 256 Mtrees for Data Domain 9500
    • 128 Mtrees for each of the DD990, DD4200, DD4500 and DD7200
  • Stream count increases – DD9500 can now scale to 1,885 simultaneous incoming streams
  • Enhanced CIFS support
  • Open file replication – great for backups of large databases, etc. This allows the backup to start replicating before it’s even finished.
  • ProtectPoint for XtremIO

Data Protection Suite (DPS) for VMware

DPS for VMware is a new socket-based licensing model for mid-market businesses that are highly virtualized and want an effective enterprise-grade data protection solution. Providing Avamar, Data Protection Advisor and RecoverPoint for Virtual Machines, DPS for VMware is priced based on the number of CPU sockets (not cores) in the environment.

DPS for VMware is ideally suited for organisations that are either 100% virtualised or just have a few remaining machines that are physical. You get the full range of Avamar backup and recovery options, Data Protection Advisor to monitor and report on data protection status, capacity and trends within the environment, and RecoverPoint for a highly efficient journaled replication of critical virtual machines.

…And one minor thing

There was at least one other bit of data protection news this year, and that was me finally joining EMC. I know in the grand scheme of things it’s a pretty minor point, but after years of wanting to work for EMC it felt like I was coming home. I had worked in the system integrator space for almost 15 years and have a great appreciation for the contribution integrators bring to the market. That being said, getting to work from within a company that is so focused on bringing excellent data protection products to the market is an amazing feeling. It’s easy from the outside to think everything is done for profit or shareholder value, but EMC and its employees have a real passion for their products and the change they bring to IT, business and the community as a whole. So you might say that personally, me joining EMC was the biggest data protection news for the year.

In Summary

I’m willing to bet I forgot something in the list above. It’s been a big year for Data Protection at EMC. Every time I’ve turned around there’s been new releases or updates, new features or functions, and new options to ensure that no matter where the data is or how critical the data is to the organisation, EMC has an effective data protection strategy for it. I’m almost feeling a little bit exhausted having come up with the list above!

So I’ll end on a slightly different note (literally). If after a long year working with or thinking about Data Protection you want to chill for five minutes, listen to Kate Miller-Heidke’s cover of “Love is a Stranger”. She’s one of the best artists to emerge from Australia in the last decade. It’s hard to believe she did this cover over two years ago now, but it’s still great listening.

I’ll see you all in 2016! Oh, and don’t forget the survey.

NetWorker Scales El Capitan

 Basics, NetWorker  Comments Off on NetWorker Scales El Capitan
Dec 202015
 

When Mac OS X 10.11 (El Capitan) came out, I upgraded my personal desktop and laptop to El Capitan. The upgrade went pretty smoothly on both machines, but then I noticed overnight my home lab server reported backup errors for both systems.

When I checked the next day I noticed that NetWorker actually wasn’t installed any more on either system. It seemed odd for NetWorker to be removed as part of the install, but hardly an issue. I found an installer, fired it up and to my surprise found the operating system warning me the installer might cause system problems if I continued.

Doing what I should have done from the start, I set up an OS X virtual machine to test the installation on, and it seemed to go through smoothly until the very end when it reported a failure and backed out of the process. That was when I started digging into some of the changes in El Capitan. Apple, it turns out, is increasing system security by locking down third party access to /bin, /sbin, /usr/bin and /usr/sbin. As NetWorker’s binaries on Unix systems install into /usr/bin and /usr/sbin, this meant NetWorker was no longer allowed to be installed on the system.

El Capitan

Fast forward a bit, and as of NetWorker 8.2 SP2 Cumulative Release 2 (aka 8.2.2.2), was released a week or so ago including a relocated NetWorker installer for Mac OS X – now the binaries are located in /usr/local/bin and /usr/local/sbin instead. (Same goes for NetWorker 9.) Having run 8.2.2.2 on my home Macs for a couple of weeks, with backup and recovery testing, the new location works.

If you’ve got Mac OS X systems being upgraded to El Capitan, be sure to download NetWorker 8.2.2.2.

Oh, and don’t forget to fill in the 2015 NetWorker Usage Survey!

Nov 052015
 

While NetWorker 9 went DA at the end of September and is seeing healthy uptake around the world, NetWorker 8.2 is still getting updates.

Released a couple of weeks ago, NetWorker 8.2 SP2 includes a slew of changes. While undoubtedly NetWorker 9 will be seeing the majority of new-feature development henceforth, that’s not to say that NetWorker 8.2 won’t get refinements as required and planned.

Road to Recovery

Some of the changes and updates to NetWorker with the 8.2 service pack 2 release include:

  • Support for JRE 1.8/Java 8 – You can now run the NMC interface using Java 8
  • Reduced name resolution checking – NetWorker still requires name resolution (of some form), as you’d hope to see in an enterprise product, but there’s been a refinement to the times NetWorker will perform name resolution. Thus, if your DNS service is particularly slow or experiencing faults, NetWorker should not be as impacted as before*.
  • Automated checks:
    • Peer Information
    • Storage nodes
    • Usergroup hosts
  • Maximum device limit can be increased to 1024. If you’ve got a big datazone that’s running at or near 750 devices, you can now increase the maximum device count to 1024.
  • DDBoost Library upgrade – NetWorker 8.2 SP2 uses libDDBoost 3.0.3.0.

It’s the automated checks I want to spend a little time on. As a NetWorker admin or implementation consultant I would have killed for these sorts of tests; and indeed, I even wrote some scripts which did some similar things. If you’ve not used them before, you should have a look at some of the tests previously added (here and here).

The new options though continue to enhance that automated checking and will be a boon for any NetWorker administrator.

Storage Node Checking

The first new option I’d like to show you is the option to automatically check the status of all the storage nodes in your environment. This can be performed by executing the command:

# nsradmin -C "NSR Storage Node"

The output from this will obviously be dependent on your environment. In my lab I get output such as the following:

nsradmin -C "NSR Storage Node"

So for each storage node, you get name resolution checking for the storage node (forward and reverse), as well as a dump of the devices attached to the storage node. If you have an environment that has a mesh of Data Domain device mounts, or dynamic drive sharing/library sharing, this will make getting a very quick overview of devices a piece of cake.

Usergroup Host Checking

As a means of more tightly checking security options, you can now query the hosts referenced in NetWorker Usergroups and determine whether they can be correctly resolved and reached. This command is run as follows:

# nsradmin -C "NSR usergroup"

And the output (for my system) looks a bit like the following:

nsradmin -C "NSR usergroup"

(Note that it will check the referenced hosts for each user specified for that host.)

NSR Peer Information

NetWorker uses peer information – generated certificates – to validate that a host which connects to the NetWorker server saying it’s client-X really can prove it’s client-X. (It’s like the difference between asking someone their name and asking them for their driver’s license.) This prevents hosts from attaching to your network, impersonating one of your servers, and then recovering data for that server.

Sometimes if things radically change on a client that peer information may get outdated and require refreshing. (Fixing NSR Peer Information Errors remains the most accessed post on my blog.) Now you can use the automated checking routine to have a NetWorker host check all the peer certificates in its local client resource database.

The command in this case has to reference the client daemon, and so becomes:

# nsradmin -p nsrexec -s clientName -C "NSR peer information"

If you run it from the NetWorker server, for the NetWorker server, the clientName in the above becomes the server name, since you’re checking the client resource for the server. Note if you’re feeling particularly old-school (like I do all the time with NetWorker), you can replace nsrexec with 390113 in the above as well. This is actually also a good way of checking client connectivity, since we verify any local client certificates by comparing the locally cached certificate against the certificated stored on each client. (Given I’m running this on a lab server, it’s reasonable to see some timeouts and errors.)

For my lab, the results look like the following:

http://nsrd.info/blog/2009/02/23/basics-fixing-nsr-peer-information-errors/

NSR Peer Information Check 2 of 2

If you intend to wait a while (or until it hits GA) before you upgrade to NetWorker 9, I’d heartily recommend upgrading to NetWorker 8.2 SP2 if for no other reason than the incredibly useful automated checks that have been introduced.


* If your DNS/name resolution is improperly configured or faulty, I’d suggest it should be dealt with quickly.

Oct 272015
 

NetWorker 9 introduces a new action that can be incorporated into workflows, Check Connectivity. You can use this prior to a backup action to confirm that you have connectivity to a host before starting the backup. Now, you may think this is a little odd, since NetWorker effectively checks connectivity as part of the backup process, but that’s if you’re looking at Check Connectivity on a per-host basis. Used optimally, Check Connectivity allows you to easily streamline the process of confirming that all hosts are available before starting the backup.

This option is important when we consider multi-host applications and services within environments where it’s actually deemed critical that the backup either run for everything or nothing. That way you can’t (in theory) capture logically inconsistent backups of the environment – for example, getting a backup of an application server but not the database that runs in conjunction with it.

In the example policy below I’ve created a simple workflow that does the following:

  • Checks client connectivity
  • If that’s successful:
    • Executes a backup of the hosts in question to the AFTD_Backup pool
    • Clones those backups to the AFTD_Clone pool

Multihost Workflow and Policy

I’ll step through the check connectivity activity so you can see what it looks like:

Check Connectivity Action Screen 1

Check Connectivity Action Screen 1

Check Connectivity Action Screen 2

Check Connectivity Action Screen 2

This is probably the most important option in the check connectivity action: “Succeed only after all clients succeed” – in other words, the action will fail if any of the clients we want to backup can’t be contacted.

Check Connectivity Action Screen 3

Check Connectivity Action Screen 3

Check Connectivity Action Screen 4

Check Connectivity Action Screen 4

It’s a pretty simple action, as you can see.

Zooming in on a little on the workflow visualisation, you’ll see it in more detail here:

Multihost Workflow Visualisation

Multihost Workflow Visualisation

By the way, I’m loving the option to edit components of the workflow and actions from the visualisation, i.e.:

Multihost Workflow Visualisation Pool Edit

Multihost Workflow Visualisation Pool Edit

In order to test and demonstrate the check connectivity action, I configured 6 backup clients:

  • test01
  • test02
  • test03
  • test04
  • test05
  • test06

On the first test, I made sure NetWorker was running on all 6 clients, and the backup/clone actions were permitted to execute after a successful connectivity test:

Multihost Workflow Executing Successfully

Multihost Workflow Executing Successfully

Now, after that finished, I shutdown the NetWorker services on one of the clients, test06, to see how this would impact the check connectivity action:

Stopping NetWorker on a Client

Stopping NetWorker on a Client

With NetWorker stopped, the workflow failed as a result of the connectivity check failing for one of the hosts. The high level failure looked like this:

Multihost Workflow Failure

Multihost Workflow Failure

Double-clicking on the check connectivity action results in the Monitoring view of NMC showed me the following:

Check Connectivity Error Dialog

Check Connectivity Error Dialog

To see the messages in more detail I just copied and pasted it into Notepad, which revealed the full details of the connectivity testing:

Multihost Workflow Check Connectivity Results

Multihost Workflow Check Connectivity Results

And there you have it. For sure, I’ve done this sort of multi-host connectivity testing in the past using NetWorker 8 and 7 (actually, even using NetWorker 6), but it’s always required nesting savegroups where the parent savegroup executes a pre-command to check via rpcinfo the availability of each host in the child savegroup before using nsradmin to invoke the child savegroup. It’s a somewhat messy approach and requires executing at least some form of backup in the parent savegroup (otherwise NetWorker declares the parent group a failure). The new functionality is simple, straight forward and is easily incorporated into a workflow.

If you have the requirement in your environment to ensure all or no clients in a group backup, this is an excellent reason to upgrade to NetWorker 9. If you’re already on NetWorker 9, keep an eye out for where you can incorporate this into your policies and workflows.