NetWorker 9.1 FLR Web Interface

 NVP, Recovery, vProxy  Comments Off on NetWorker 9.1 FLR Web Interface
Apr 042017

Hey, don’t forget, my new book is available. Jam packed with information about protecting across all types of RPOs and RTOs, as well as helping out on the procedural and governance side of things. Check it out today on Amazon! (Kindle version available, too.)

In my introductory NetWorker 9.1 post, I covered file level recovery (FLR) from VMware image level backup via NMC. I felt at the time that it was worthwhile covering FLR from within NMC as the VMware recovery integration in NMC was new with 9.1. But at the same time, the FLR Web interface for NetWorker has also had a revamp, and I want to quickly run through that now.

First, the most important aspect of FLR from the new NetWorker Virtual Proxy (NVP, aka “vProxy”) is not something you do by browsing to the Proxy itself. In this updated NetWorker architecture, the proxies are very much dumb appliances, completely disposable, with all the management intelligence coming from the NetWorker server itself.

Thus, to start a web based FLR session, you actually point your browser to:


The FLR web service now runs on the NetWorker server itself. (In this sense quite similarly to the FLR service for Hyper-V.)

The next major change is you no longer have to use the FLR interface from a system currently getting image based backups. In fact, in the example I’m providing today, I’m doing it from a laptop that isn’t even a member of the NetWorker datazone.

When you get to the service, you’ll be prompted to login:

01 Initial Login

For my test, I wanted to access via the Administration interface, so I switched to ‘Admin’ and logged on as the NetWorker owner:

02 Logging In as Administrator

After you login, you’re prompted to choose the vCenter environment you want to restore from:

03 Select vCenter

Selecting the vCenter server of course lets you then choose the protected virtual machine in that environment to be recovered:

04 Select VM and Backup

(Science fiction fans will perhaps be able to intuit my host naming convention for production systems in my home lab based on the first three virtual machine names.)

Once you’ve selected the virtual machine you want to recover from, you then get to choose the backup you want to recover – you’ll get a list of backups and clones if you’re cloning. In the above example I’ve got no clones of the specific virtual machine that’s been protected. Clicking ‘Next’ after you’ve selected the virtual machine and the specific backup will result in you being prompted to provide access credentials for the virtual machine. This is so that the FLR agent can mount the backup:

05 Provide Credentials for VM

Once you provide the login credentials (and they don’t have to be local – they can be an AD specified login by using the domain\account syntax), the backup will be mounted, then you’ll be prompted to select where you want to recover to:

06 Select Recovery Location

In this case I selected the same host, recovering back to C:\tmp.

Next you obviously need to select the file(s) and folder(s) you want to recover. In this case I just selected a single file:

07 Select Content to Recover

Once you’ve selected the file(s) and folder(s) you want to recover, click the Restore button to start the recovery. You’ll be prompted to confirm:

08 Confirm Recovery

The restore monitor is accessible via the bottom of the FLR interface, basically an upward-pointing arrow-head to expand. This gives you a view of a running, or in this case, a complete restore, since it was only a single file and took very little time to complete:

09 Recovery Success

My advice generally is that if you want to recover thousands or tens of thousands of files, you’re better off using the NMC interface (particularly if the NetWorker server doesn’t have a lot of RAM allocated to it), but for smaller collections of files the FLR web interface is more than acceptable.

And Flash-free, of course.

There you have it, the NetWorker 9.1 VMware FLR interface.

Hey, don’t forget, my new book is available. Jam packed with information about protecting across all types of RPOs and RTOs, as well as helping out on the procedural and governance side of things. Check it out today on Amazon! (Kindle version available, too.)


Jun 272016

NetWorker 9 introduced a new, pure HTML5 web interface for the File Level Recovery interface for VBA, which works much the same way as the v8.x FLR, just without Flash.


However, it also introduced nsrvbaflr, a command line utility that comes with the base NetWorker client install, which can be used on Linux or Windows virtual machines to execute file level recovery from VMware image level backups.

Hang on, I hear you say – VMware image level backups are meant to be clientless, so does that mean I have to start installing the client software just for FLR? Well, actually – no.

A NetWorker Linux client install will include the nsrvbaflr utility in /usr/sbin, and this is a standalone binary. It doesn’t rely on any other binaries or libraries, so in order to use it on a Linux VMware instance, all you have to do is copy the binary across from a compatible client install. Since my NetWorker server (orilla) is a Linux host itself, that’s as simple as:

[Mon Jun 27 14:23:16]
[• ~ •]
$ ssh root@orilla
root@orilla's password: <<password>>
Last login: Mon Jun 27 12:25:45 2016 from
[root@orilla ~]# scp /usr/sbin/nsrvbaflr root@krell:/root
root@krell's password: 
nsrvbaflr                         100%         5655KB      5.5MB/s    00:00

With the binary copied across FLR is only a step away.

The nsrvbaflr utility can be run in interactive or non-interactive mode. I wanted to try it out in interactive mode, so the session started off like this:

[root@krell tmp]# nsrvbaflr
-bash: nsrvbaflr: command not found
[root@krell tmp]# /root/nsrvbaflr
VBA hostname|IP:
 Successfully connected to VBA: (
vmware-flr> locallogin
 Username: root
 Password: <<password>>

I then had a bit of an exercise in debugging. You see, I’d finally rebuilt my home lab recently and part of that involved spinning up a whole bunch of individual virtual machines running CentOS 6.x to takeover functions previously collapsed to a single machine. So I’ve got independent Mail, Wiki and DNS/DHCP servers, and of course I accepted the defaults on most of those systems leaving me with ext4 filesystems, which the base VBA appliance can’t handle. This, of course, I’d forgotten. So of course, when I then tried out any command that would access the filesystem of a backup, I had this happen:

vmware-flr> cd root
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
 Backup browse request failed. Reason: (Unknown)
vmware-flr> pwd
 Backup working folder: Backup root
vmware-flr> ls
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
 Backup browse request failed. Reason: (Unknown)

After a little while wearing a thinking cap again, I remembered the ext4 limitation, so I quickly provisioned a VBA Proxy within my home lab. (If you review the documentation for NetWorker VMware Integration, this is fairly clearly spelt out. Dolt that I was, I forgot.) Once that proxy was deployed, things went a whole lot more smoothly:

[root@krell tmp]# /root/nsrvbaflr
VBA hostname|IP:
 Successfully connected to VBA: (
vmware-flr> locallogin
 Username: root
 Password: <<password>>
 Successfully logged into client: (/
vmware-flr> backups
 Backups for client: /
 Backup number: 54 Date: 2016/06/27 01:56 PM
 Backup number: 53 Date: 2016/06/27 02:00 AM
 Backup number: 52 Date: 2016/06/26 02:00 AM
 Backup number: 51 Date: 2016/06/25 02:01 AM
 Backup number: 50 Date: 2016/06/24 02:00 AM
 Backup number: 49 Date: 2016/06/23 02:01 AM
 Backup number: 48 Date: 2016/06/22 02:00 AM
 Backup number: 47 Date: 2016/06/21 02:01 AM
 Backup number: 46 Date: 2016/06/20 02:01 AM
 Backup number: 45 Date: 2016/06/19 02:01 AM
 Backup number: 44 Date: 2016/06/18 02:01 AM
 Backup number: 43 Date: 2016/06/17 02:01 AM
 Backup number: 42 Date: 2016/06/16 02:01 AM
 Backup number: 41 Date: 2016/06/15 02:01 AM
 Backup number: 40 Date: 2016/06/14 02:00 AM
 Backup number: 39 Date: 2016/06/13 02:01 AM
 Backup number: 38 Date: 2016/06/12 02:01 AM
 Backup number: 37 Date: 2016/06/11 02:01 AM
 Backup number: 36 Date: 2016/06/10 02:00 AM
 Backup number: 35 Date: 2016/06/09 02:01 AM
 Backup number: 34 Date: 2016/06/08 02:01 AM
 Backup number: 33 Date: 2016/06/07 02:01 AM
 Backup number: 32 Date: 2016/06/06 02:01 AM
 Backup number: 31 Date: 2016/06/05 02:01 AM
 Backup number: 30 Date: 2016/06/04 02:01 AM
 Backup number: 29 Date: 2016/06/03 02:01 AM
 Backup number: 28 Date: 2016/06/02 09:05 AM
 Backup number: 27 Date: 2016/06/02 02:01 AM
 Backup number: 26 Date: 2016/06/01 02:01 AM
 Backup number: 25 Date: 2016/05/31 02:01 AM
 Backup number: 24 Date: 2016/05/30 02:01 AM
 Backup number: 23 Date: 2016/05/29 02:01 AM
 Backup number: 22 Date: 2016/05/28 03:08 PM
 Backup number: 21 Date: 2016/05/28 02:00 AM
vmware-flr> backup 53
 Backup: (53) selected.
vmware-flr> cd root
. . . . . . . . . . . . . . . . . . 
vmware-flr> ls
 Folder: root
 Folder: .ssh 4 KB 2016/06/02 09:08 PM
 Folder: bin 4 KB 2016/06/07 11:09 PM
 File: .bash_history 4.9 KB 2016/07/20 07:58 AM
 File: .bash_logout 18 B 2009/06/20 10:45 AM
 File: .bash_profile 176 B 2009/06/20 10:45 AM
 File: .bashrc 176 B 2004/10/23 03:59 AM
 File: .cshrc 100 B 2004/10/23 03:59 AM
 File: .tcshrc 129 B 2005/01/03 09:42 PM
 File: anaconda-ks.cfg 1.5 KB 2016/06/02 08:25 PM
 File: install.log 26.7 KB 2016/06/02 08:25 PM
 File: install.log.syslog 7.4 KB 2016/06/02 08:24 PM

2 Folder(s)
 9 File(s)
vmware-flr> add install.log
 Path: (root/install.log) successfully added to the recover queue.
vmware-flr> targetpath
 Enter "." to set working folder: () as the target path or enter an absoulte path.
 path: tmp
 Target path successfully set to: (/tmp)
vmware-flr> queue
 Recover queue: root/install.log
vmware-flr> status
 VBA host:     
 VBA version:  
 Local user:             root
 Source client FQN:      /
 Selected backup:        Backup #: 53 Date: 2016/06/27 02:00 AM
 Backup working folder:  /root
 Recover queue:          root/install.log
 Target client FQN:      /
 Target working folder:  Client root
 Target path:            /tmp
vmware-flr> recover
 The restore request has been successfully issued to the VBA.
vmware-flr> quit
[root@krell tmp]# ls /tmp/install.log

That’s how simple FLR is from VMware image level backups under NetWorker 9. The same limitations for FLR in terms of the number of files and folders, etc., apply to command line as much as they do the web interface, so keep that in mind when you’re using it. Beyond that, this makes it straight-forward to perform FLR for Linux hosts without needing to launch X11.

Testing (and debugging) an emergency restore

 Data Domain, NetWorker, Recovery, VBA  Comments Off on Testing (and debugging) an emergency restore
Feb 252015

A few days ago I had some spare time up my sleeve, and I decided to test out the Emergency Restore function in NetWorker VBA/EBR. After all, you never want to test out emergency recovery procedures for the first time in an emergency, so I wanted to be prepared.

If you’ve not seen it, the Emergency Restore panel is accessed from your EBR appliance (https://applianceName:8580/ebr-configure) and looks like the following:

EBR Emergency Restore Panel

The goal of the Emergency Restore function is simple: you have a virtual machine you urgently need to restore, but the vCenter server is also down. Of course, in an ideal scenario, you should never need to use the Emergency Restore function, but ideal and reality don’t always converge with 100% overlap.

In this scenario, to simulate my vCenter server being down, I went into vCenter, selected the ESX server I wanted to recover a virtual machine for (c64), and disconnected from it. To all intents and purposes to the ESX server, vCenter was down – at least, enough to satisfy VBA that I really needed to use the Emergency Restore function.

Once you’ve selected the VM, and the backup of the VM you want to restore, you click the Restore button to get things underway. The first prompt looks like the following:

EBR ESX Connection Prompt(Yes, my ESX server is named after the Commodore 64. For what it’s worth, my vCenter server is c128 and a smaller ESX server I’ve got configured is plus4.)

Entering the ESX server details and login credentials, you click OK to jump through to the recovery options (including the name of the new virtual machine):

EBR - Recovery OptionsAfter you fill in the new virtual machine name and choose the datastore you want to recover from, it’s as simple as clicking Restore and the ball is rolling. Except…

EBR Emergency Restore Error

After about 5 minutes, it failed, and the error I got was:

Restore failed.

Server could not create a restore task at this time. Please ensure your ESX host is resolvable by your DNS server. In addition, as configuration changes may take a few minutes to become effective, please try again at a later time.

From a cursory inspection, I couldn’t find any reference to the error on the support website, so I initially thought I must have done something wrong. Having re-read the Emergency Restore section of the VMware Integration Guide a few times, I was confident I hadn’t missed anything, so I figured the ESX server might have been taking a few minutes to be sufficiently standalone after the disconnection, and gave it a good ten or fifteen minutes before reattempting, but got the same error.

So I went through and did a bit of digging on the actual EBR server itself, diving into the logs there. I eventually re-ran the recovery while tailing the EBR logs, and noticed it attempting to connect to a Data Domain system I knew was down at the time … and had my ahah! moment.

You see I’d previously backed up the virtual machine to one Data Domain, but when I needed to run some other tests, changed my configuration and started backing up the virtual infrastructure to another Data Domain. EBR needed both online to complete the recovery, of course!

Once I had the original Data Domain powered up and running, the Emergency Restore went without a single hitch, and I was pleased to see this little message:

Successful submission of restore job

Before too long I was seeing good progress on the restore:

Emergency Restore Progress

And not long after that, I saw the sort of message you always want to see in an emergency recovery:

EBR Emergency Recovery Complete

There you have it – the Emergency Restore function tested well away from any emergency situation, and a bit of debugging while I was at it.

I’m sure you’ll hope you never need to use the Emergency Restore feature within your virtual environment, but knowing it’s there – and knowing how simple the process is – might help you avoid serious problems in an emergency.



Spy vs Agent

 Architecture, NetWorker  Comments Off on Spy vs Agent
Nov 162013

Agent vs Spy

Since VMware entered the server space, a simple problem has plagued backup administrators:

  • Backup via a client installed in the VM? or
  • Backup the virtual machine files.

I like to all this agent vs spy. The agent is the conventional client software, and the spy is the mechanism that allows a backup of the virtual machine files. It’s a “spy” of course because it’s ‘serverless’ as far as the virtual machine is concerned.

NetWorker has supported three distinct backup mechanisms – VCB, VADP and now VBA. Each have had their own unique qualities, but the mechanism has been becoming progressively more sophisticated over time.

The question often remains … when should you backup via agent, and when should you backup via a spy?

For the most part, if you’re running database style applications within virtual machines, the answer is still a simple one – the closer the backup software is to your data, the more guarantee you have of getting a fully application-consistent backup. So if you’re running Oracle or Exchange in a virtual machine, you’ll still want to do an agent-based backup with the appropriate module software also installed. Additionally, when you want cross-system consistency (e.g., for Sharepoint, Blackboard, Documentum systems, etc.), you’ll likely want to resort to in-VM agents to keep a highly granular control.

There’s no doubt that’s going to change over the coming years. I don’t see a point where in-guest agents will completely disappear, but there will very likely be a time where they’re no longer the majority method for backup.

So what are some good use-case scenarios for out-of-guest backup scenarios in NetWorker for virtual machines?

Here’s a quick list:

  1. LAN minimisation: I wouldn’t call it LAN-free, but if you can use fibre-channel connectivity between VMware LUNs and proxies in NetWorker, you may have the potential to drastically reduce the amount of LAN traffic involved in a backup. This could come in one of two ways:
    • Fibre-channel accessible media (e.g., tape or virtual tape) directly connected to a proxy – this results in only backup metadata traversing the IP network;
    • Dedicated backup IP network – OK, this could be done regardless of whether you’re using guest or host based backup software, but if the data is coming off fibre from the disks and going over a private backup network divorced from the network of the virtual machine, you’ve effectively gone LAN-free as far as the virtual machine is concerned;
  2. License minimisation: If you’re using conventional NetWorker licensing, and you have a reasonably dense allocation of virtual machines per ESX server, then you could get considerably more bang for buck out of your backup environment using virtual client licensing rather than per-VM client licensing;
  3. Disaster recovery: If you don’t have SRM or other replication technology in your environment, then NetWorker’s ability to do image level recovery from Virtual Machine backups is the next closest thing you’ll get;
  4. Side-stepping firewalls: Even permissive firewalls can be a pain when it comes to backup – it’s easier to swamp the link if there’s a limited number of ports open; if your firewalled machines are accessible from a vCenter server sans-firewall, you’ll likely get better backup throughput targeting the virtual machine files than the guest files. Also, the backup process is going to be completely hidden from the guest, improving security. You might be able to completely sidestep the firewall via fibre-channel connection to LUNs, or you might be able to at least minimise it by keeping communication between proxies and vCenter/ESX servers;
  5. Easier enabling of backups: OK, by rights you should have decent change control within your organisation, and no new host should be able to commissioned without a checkbox being ticked or crossed on “[] Backups required”, but that’s not always guaranteed. Close integration between NetWorker and vCenter can allow easier identification of what is and isn’t being backed up … and that’s just getting better as time goes by;
  6. Instant-on recovery: Introduced in Avamar 7, and sure to eventually make an appearance in NetWorker 8.x is instant-on recovery. That’s where Avamar, when writing to Data Domain, can generate virtual machine backups that can be powered on from the Data Domain and VMotion’d back into production storage while they’re being used.

They’re not the only reasons of course – I wasn’t trying to create a definitive list. As always, each site is different. But if you’ve got NetWorker and VMware, you really owe it to yourself to check out the features and see if they can work together to make your life as a backup administrator easier.

Jul 122013

iStock Racing

As is typically the case, EMC and my timing has been a little out of whack. They announced their “backup to the future” event around the time that I suddenly had to move, and a few days after the event, I still haven’t been able to watch any of the coverage due to the dubious honour of having to subsist on mobile internet for a couple of weeks while I wait for ADSL to be installed.

Sigh. Clearly this is a serious problem … maybe EMC will have to employ me before NetWorker 8.2 comes out so we have a better chance of keeping our calendars in sync on big events. That way they won’t accidentally schedule a major backup release when I have to move again … 🙂

While I haven’t been able to see the “Backup to the Future” material, I had spent a chunk of time working with NetWorker 8.1 through the beta testing phase, so I can have a bit of a chat about that. So, grab whatever your favourite beverage is, pull up a chair, and let me spin you a yarn or two. (In a couple of weeks I’ll likely have a few things to say about Backup to the Future … a lot of the material out of EMC lately about accidental architecture aligns very closely to my attitudes of where companies go wrong with data protection.)

It’s not surprising that EMC’s main staff backup blog is called thebackupwindow. Windows are terms that pretty much everyone who works in backup eats, lives and breathes. (Not just backup windows of course, but recovery windows too.) You might say that Moore’s law has been a governing factor in computing. But there’s another law that, to be perfectly honest, is a pain in the proverbial for every person who is involved in backup and recovery, and for want of a better term, I’m going to call it Newton’s Third Law of Data Protection – i.e., to every action there is always an equal and opposite reaction.

The net result? Data keeps on getting bigger, and in turn backup windows for that data keeps on shrinking.

So, EMC’s primary blog being called the backup window makes perfect sense.

As does the feature set of NetWorker 8.1.

(See, I was getting to the point, even if I was walking around it a few times.)

While some of the features of NetWorker 8.1 are geared around interface changes, and others around security, the vast bulk of them are focused on meeting the demands of a shrinking backup window. Let’s take a quick look at some of those new features…

Window Work

Parallel Saveset Streams (Unix)

The bane of every backup administrator is dense filesystems, and the PSS feature is designed to help get around this. Got a Unix filesystem with tens of millions of files? Likely it’s got a good disk structure underneath it, but filesystems suck for full sequential walks. Turning on the Parallel Saveset Streams features for key Unix/Linux clients with dense filesystems will start to make a difference here – NetWorker will spawn multiple save processes to separately walk, and save data from the filesystem.

Block Level Backups (Windows)

That dense filesystem problem isn’t just limited to Unix servers, of course. Backup administrators with large Windows servers in their environments equally feel the pain, and enabling BLB functionality on Windows servers for key, large filesystems, will allow the bypass of the filesystems entirely, achieving high speed backup with file level recovery capabilities.

Storage Node Load Balancing

Sure to be a boon for big datazones, storage node load balancing will allow businesses to deploy multiple storage nodes in relatively small but dense network segments and have clients spread their backups automatically between the storage nodes, rather than having to juggle which clients should backup to where.

Optimised Deduplication Filesystem Backups for Windows

Windows 2012 Server introduced deduplication for the filesystem. NetWorker 8.1 introduces the ability to backup the deduplicated blocks. Net result? If you’ve got a 2TB filesystem which represents 800GB of deduplicated data, NetWorker gives you the option of just backing up 800GB of data rather than 2TB of data. I’m hoping, of course, that this isn’t just going to be limited to Windows deduplication filesystems … there’s a lot of ZFS users out there for instance who’ll be thinking “Um? We got there first…”

Virtual Synthetic Fulls on Data Domains

Synthetic fulls, introduced in NetWorker 8, can work wonders at reducing the required backup windows within an environment, but, creating a new synthetic full when the target was a Data Domain would result in a full rehydration of the data. Under NetWorker 8.1 though, that fabulous Boost integration continues apace, and the generation of a synthetic full is handed over to the Data Domain when it’s the operation source and target. Net result? Synthetic fulls with a Data Domain involved don’t need to rehydrate the data to generate the new full.

Boost over Fibre Channel

A long time ago in a source tree a long time ago, advanced file type devices showed a lot of promise but had some disappointments. Those disappointments were removed in NetWorker 8 with the complete re-engineering of AFTDs, but in the meantime, a lot of businesses that had deployed Data Domain systems had gone down the VTL route to try to ameliorate those backup-to-disk headaches. Unfortunately, when true backup to disk was fixed with NetWorker 8, that left those businesses in an undesirable situation: the advantages of Boost were clear, but it could only be implemented over IP, and since fibre-channel infrastructure isn’t cheap, not everyone was keen to just switch their investments across to IP. NetWorker 8.1 helps that transition. Of course, it’s not the same as making a Data Domain system fully addressable on an IP network, but it does allow the creation of Boost backup to disk devices over Fibre Channel, which means that technology transition can be phased and handled more smoothly. I suspect this will see a noticeable reduction in the number of NetWorker installs using VTLs.

Efficiency Improvements to nsrclone

Smaller than the other changes mentioned above, the nsrclone process has been improved in terms of media database fetch processes, which means it starts cloning sooner. That’s a good thing, of course.

Faster Space Reclamation on AFTD/Data Domain Systems

Unfortunately you don’t always get to control the filesystem you write to for backups. When I’m backing up to traditional disk on Linux, I pretty much always deploy AFTDs on XFS. That way, when I decide to delete 4TB of backups they delete quickly. If I was using say, ext3, I’d issue the delete command, go off, have a coffee, come back, curse at the server, go away again, have lunch, come back… well, you get the picture.

While some of the delete process is bound up in how long it takes for the OS/Filesystem to respond to a file delete command (particularly for a large file), some of that space reclamation process is bound in NetWorker’s media database operations. That part has been improved in NetWorker 8.1.

The Other Bits

I mentioned NetWorker 8.1 wasn’t all about shrinking the backup window, and there are some other features. Quickly running through them…

VMware Backup Appliance (VBA)

Virtual Machines … they really are the bane of everyone’s lives. Of course, operationally they’re great, but sometimes backing them up leaves you wishing they were all physical, still. Well, maybe not wishing, but you get the drift.

NetWorker 8.0 introduced full VADP support. NetWorker 8.1 goes one step further in working with the Virtual Backup Appliance option introduced in newer versions of ESX. This isn’t something I’ve had a chance to play with – my lab is all Parallels due to Fusion not liking my Mac Pro’s CPUs, but I imagine it’s something I’ll see deployed soon enough.

NetWorker Snapshot Management

NSM replaces the old and somewhat crotchety PowerSnap functionality. For long-term PowerSnap users who have been looking for a solid update, this will undoubtedly be a big bonus.

Recovery Comes Home

8.1 introduces a Recovery interface within NMC, where it’s belonged since NMC was first created. This seems the immediate termination of the old, legacy nwrecover interface from the Unix install of NetWorker, and it’s undoubtedly going to see the Windows recovery GUI killed off over time as well. In fact, if you want to recover from Windows block level backups, you better get used to the new recovery interface.

What I really like about this interface is that you can create a recovery session and then save it to re-run it later. A lot of administrators and operators are going to love this new interface.


…I’m annoyed with Block Level Backups. It’s completely understandable that it has to be done to disk backup (i.e., AFTD or Data Domain), and that it requires client direct. Again, that’s understandable. However, if want to do block level backups to AFTDs presented from Unix/Linux servers, you’re out of luck. AFTDs must be presented from Windows servers.

know this is a relatively small limitation, but I have to be honest – I just don’t like it. I want to see it fixed in NetWorker 8.2. I’ll settle for some sort of proxy mechanism if necessary, but I really do think it should be fixed.

Then again, I do come from a long-term Unix background. So take my complaint with whatever bias you want to attribute to it.


So there you have it – NetWorker 8.1 is out on the starting line, revving, and ready to make your backups run faster. It’s going to be a welcome upgrade for a lot of environments, and gives us a tantalising taste of improvements that are coming to our backup windows.


May 222012


It’s a fantastic blade to wield through a datacentre. Sweeping and scything, whole racks of equipment are reduced to single servers presenting dozens of hosts. All those driver disks? All those complex and fiddly options for hardware components during OS installation? Brushed aside – all the virtual components are simple and have rock solid drivers. Virtual machine host failing? That’s OK, just push the virtual machines across to another server without the users even noticing.

The improvements virtualisation has made to system efficiency, reliability, etc., in the x86/x86_64 field have been unquestionable.

Yet, like any other sword, it’s double edged.

Virtualisation is about cramming as many systems as is practical within a single bucket.

Backup is something that virtualisation has always handled poorly. And there’s a reason for this – virtualisation is designed for environments where the hosts cooperatively share access to resources. Thin provisioning isn’t just about storage – it’s also about CPU, networking and memory.

Backup isn’t about cooperative sharing of CPU, networking or memory. It’s about needing to get as much data from A to B as possible as quickly as can be done:

The problem with virtualisation backup

Backup at the guest level wants to suck as much data from the virtual network pipes provided by all those machines on the same host at the same time. You want to see the biggest, most powerful virtualisation server your company has ever bought grind to a halt and saturate the network as well? There’s a good chance backing up every guest it runs simultaneously will do the trick just nicely.

When VMware first came up with VCB, it was meant to be the solution. Pull the backup away from the guest, make it part of the hypervisor, and voilà, the problem is solved!

Except it was written by people who believed virtualisation applied only to Windows systems. And thus, it was laughably sad. No, I’m not having a dig at Windows here. But I am having a dig at the notion of homogeneous virtual environments. Sure, they exist, but designing products around them when you’re the virtualisation vendor is … well, I have to say, short sighted.

Perhaps for this reason, or perhaps for less desirable reasons, VCB never really gained the traction VMware likely hoped for, and so something else had to be developed. Something more expansive.

So, VADP was meant to be the big, grand solution to this. And indeed, the VADP API allows more than just Windows systems backups to be performed in such a way that file level recovery from those backups is possible.

What’s the vendor support like though? Haphazard, irregular and inconsistent would probably be the best description. Product X: “Oh, you want to backup a database as well? You need to revert to a guest agent.” Product Y: “Huh? Linux? Guest agent.” Product Z: “Linux? Sure! For any system – well, any that uses ext2 or ext3 filesystems” … you get the picture.

So the problem with VADP is that it’s only a partial solution. In fact, it’s less than half the solution for backing up virtual machines on VMware. It’s maybe 40%. The other 40% is provided by whatever backup product you’re using, and there’s 20% glue.

Between that 40%, 20% and 40%, there’s a lot of scope for things to fall through the cracks.

Where “things” are:

  • Guests using operating systems the backup product doesn’t support VADP with;
  • Guest using filesystems the backup product doesn’t support VADP with;
  • Guests using databases or applications the backup product doesn’t support VADP with.

VADP is the emperor’s new clothes. Everyone is sold on it until the discussions start around what they can’t do with it.

I’m tired of VADP being seen as a silver bullet. That’s the real problem – it doesn’t matter how many hoozits a widget has – if it doesn’t have the hoozit you need, the widget is not fit for your purposes.

I’m not pointing the finger at EMC here. I don’t see a single backup vendor, enterprise or otherwise, providing complete backup solutions under VADP. There’s always something missing.

Until that isn’t the case, you’ll excuse me if I don’t drink the VADP koolaid.

After all, my job is to make sure we can backup everything, not just the easy bits.

Jul 172011

A wise man once said in a meeting:

If you want to see how indispensable you are, stick your finger in a glass of water and measure the size of the hole left when you pull it back out.

This week I’ve been reflecting a lot on that statement given the radical licensing changes that have originated out of VMware for vSphere 5.

I want to reflect on the background to the “are they right or are they wrong” argument here – I think every business is entitled to make a fair and reasonable profit. I also say this as an outsider – my area of interest remains backup and recovery, not virtualisation. In short, for me, virtualisation is a tool, a means to an end – it’s a butler, not the work.

So I think I can look at this as someone who is exposed to the business of virtualisation, but isn’t directly bound by it.

For any company that sells software rather than hardware, there are going to be times when licensing is re-evaluated and new cost models are developed. NetWorker for years had a licensing model that was growing in complexity. Over the last few years EMC has been working at simplifying that, with the most recent change being the capacity licensing. This hasn’t been a big hit because it’s more aimed at people who can’t quite step up to the enterprise license, rather than the average business, but it’s still a step in the right direction, and a portent of things to come.

VMware has clearly hit the point where they’re having to say to the market, “the way we’ve previously been pricing this is no longer sustainable”.

As has been so often the case within the IT industry over the past 20 years, pricing has raced to the bottom, and once it’s hit the bottom, there’s a need for an adjustment. I do partly blame Microsoft on this front – they’re renown for dropping their pricing pants in order to smack around the competition. That’s not a healthy business model.

Much is premised around a false sense of entitlement. “Someone produces X so I should get X for as cheap a price as possible”. It’s the logic of the IT industry, it seems. Yet let’s look at say, the car industry as a comparison. That business model – “get customers by giving it to them as cheap as possible” almost wiped out the US car industry. It was reported, for instance, that between the rebates and the discounts on offer by 2008, some US car companies were losing up to $500 per vehicle sold.

Selling volume at discount is fine.

Selling volume at loss isn’t.

VMware are by no means indispensable in the IT industry. The pricing model change will undoubtedly drive some companies to consider the alternatives out there – Hypervisor, Xen and Parallels, for instance.

But I think we, as an industry, have to take some responsibility here – we have to accept our part that this is a mea culpa of sorts: we’ve allowed the “race to the bottom” pricing model to become too pervasive, and are now getting to reap the rewards of that.

Jan 122010

Over at Backup Central, Curtis Preston has written a couple of excellent blog posts to do with VSS.

The first, What is Windows VSS and why should you care? is an excellent overview of how the VSS process works within Windows. Even if you’ve been using VSS within your environment, if you’re not quite sure how it works, this is a great piece to read.

The second delves into issues relating to VMware VCB’s (in)ability to perform consistent application backups – i.e., via VSS for say, an Exchange or Microsoft SQL guest. Titled Hyper-V ahead of VMware in the Backup Race, it’s a justifiable kick in the pants to VMware, and a pointed warning regarding VMware/VCB backups of applications.

(These two articles, Curtis mentions, came about from some posts by Scott Waterhouse on The Backup Blog, which talked about vSphere backups.)

Dec 092009

So in an earlier post, I mentioned that I’d been looking at first comparisons between VMware Fusion 3.0 and Parallels Desktop 5 for Mac, and I thought it was time to follow-up with longer term impressions.

To be blunt, VMware Fusion 3 is unpolished and unpleasant to use on an almost continual basis. I’ll keep it around for only two reasons: (a) so I can run ESX/vSphere within a VM for testing purposes, and (b) I can periodically play with the demo/test images provided by EMC for particular products that won’t convert into Parallels images.

So what’s there to dislike about Fusion?

  • Unity. It’s like someone at VMware declared “Make it slow. Make it inefficient. Make it periodically take 10+ seconds to redraw windows. Make it work but glitchy enough that it makes the user grind their teeth in frustration.” Well, if someone did decree that as a product feature, they did a remarkably good job of achieving it. Here’s a tip, folks at VMware: Buy a copy of Parallels and see how professionals do an integrated windowing feature. Unity in Fusion v3 is worse than Coherence when it was first introduced (which was fine) – i.e., you have a long, long way to go.
  • Import another VM. What VM would you like to import? Parallels? Forget it. Why offer to import VMs from Parallels if every VM comes in unusable? (I’m sure other people must have better experiences than this, but I’m certainly not impressed.)
  • Performance. OK, so VMware Fusion performance isn’t atrocious – it’s actually OK. However, I’d been led to believe that VMware Fusion kicked Parallels Desktop out of the ballpark when it came to performance. I’ve not seen anything to indicate that it exceeds the performance of Parallels, and so I see that as a negative.
  • Quit. Don’t pester me, just suspend my VM.

As I said, I’ll be keeping Fusion around, but only for those situations where I can’t use Parallels.

Take a peek at Going Virtual

 Aside  Comments Off on Take a peek at Going Virtual
Jul 112009

My colleague Brian Norris is continuing his excellent coverage of virtualisation topics over at Going Virtual, this time with a new article that delves further into ESX security – this time focusing on securing VMware Tools.

If you’re interested in ESX security, I’d invite you to check out his latest article.