Preston de Guise

Dec 012017
Survey Image

It seems like only a few weeks ago, 2017 was starting. But here we are again, and it’s time for another NetWorker usage survey. If you’re a recent blog subscriber, you may not have seen previous surveys, so here’s how it works:

Every year a survey is run on the NetWorker blog to capture data on how businesses are using NetWorker within their environment. As per previous years, the survey runs from December 1 to January 31. At the end of survey, I analyse the data, crunch the numbers, sacrifice a tape to the data protection deities and generate a report about how NetWorker is being used in the community.

My goal isn’t just for the report to be a simple regurgitation of the data input by respondents. It’s good to understand the patterns that emerge, too. Is deduplication more heavily used in the Americas, or APJ? Who keeps data for the longest? Is there any correlation between the longevity of NetWorker use and the number of systems being protected? You can see last year’s survey results here.

Survey Image

To that end, it’s time to run the 2017 NetWorker survey. Once again, I’m also going to give away a copy of my latest book, Data Protection: Ensuring Data Availability. All you have to do in order to be in the running is to be sure to include your email address in the survey. Your email address will only be used to contact you if you win.

The survey should hopefully only take you 5-10 minutes.


What’s new in NetWorker 9.2.1?

 Features, NetWorker  Comments Off on What’s new in NetWorker 9.2.1?
Nov 232017

NetWorker 9.2.1 has just been released, and the engineering and product management teams have delivered a great collection of features, in addition to the usual roll-up of prior cumulative patch releases, etc.Cloud TransferThere’s some great Cloud features in 9.2.1, regardless of whether you’re working with a private cloud in your datacentre, consuming VMware in AWS, or using CloudBoost to get to object storage.

VMware’s move into the AWS cloud environment has opened up new avenues for customers seeking a seamless hybrid cloud model, and that wouldn’t be complete without being able to extend an on-premises data protection policy into the VMware environment running in AWS. So, NetWorker 9.2.1 includes the option to perform image based backups of VMware virtual machines in AWS. This is round one of support for VMware backups in AWS – as is always the case (and why for traditional AWS virtual machine backups you usually have to deploy an agent), what you can or can’t do is dependent on what level of access you have to the hypervisor. In this case, VMware have focused in their first round of enablement for data protection on image based backup and recovery. If you’ve got VMware systems in AWS that require highly granular file level recovery regularly, you may want to install an agent for those, but for ‘regular’ VMware virtual machines in AWS where the focus is being able to do high speed image level backups and recoveries, NetWorker 9.2.1 has you covered.

NetWorker 9.2.1 also supports vRealize Data Protection Extension 4.0.2, so if you’re building a private cloud experience within your environment, NetWorker will be up to date on that front for you, as well.

Finishing up with on cloud support, CloudBoost has also been updated with this release, and has some enhancements for efficiency and reporting that’ll make this update a must if you’re using it to get data onto public object storage.

Regular VMware users aren’t left out either – there’s now a new option to recover virtual machine configuration metadata as well as virtual machine data, which can be a particularly useful option when you’re doing an in-place recovery of a virtual machine. Also, if you’ve got an ESXi server or servers within your environment that aren’t under vCenter control, you’re now covered by NetWorker as well – virtual machines on these systems can also be backed up.

9.2.1 also lets you used unserved NetWorker licenses. In prior 9.x releases, it was necessary to use a license server – either one for your entire company, or potentially more if you had different security zones. Now you can associate issued licenses directly with NetWorker servers if you need to, rather than serving them out of the license server – a very handy option for dark or secured sites.

Directives. With 9.2.1 (and rolling back to the 9.2 clients too), you can now use wildcards in the directory paths. This is going to be a really useful scenario for database servers. For instance, it was quite common to see directives such as:

<< /d/u01/database >>
+skip: *.dbf

<< /d/u02/database >>
+skip: *.dbf

<< /d/u03/database >>
+skip: *.dbf

Under earlier versions of NetWorker, if a new mount point, /d/04 were added, you had to remember to update the directives to exclude (say in this example), database files from that filesystem from your backup. Now instead, you can just have a catch-all directive of:

<< /d/*/database >>
+skip: *.dbf

Or if you wanted to be more specific and only do it against mount-points that started with a ‘u’ and had 2 letters following:

<< /d/u??/database >>
+skip: *.dbf

Check out the wildcard support in the administrator’s guide for 9.2.1 (around p339).

From an availability perspective, you can now do a NetWorker disaster recovery straight from a Data Domain Cloud Tier device if you so need to. You also now get the option of using RecoverPoint for Virtual Machines (RP4VM) to replicate your NetWorker server between sites for seamless failover capabilities. If you’re using RP4VM at the moment, this really boosts your capability when it comes to protecting your backup environment.

There’s some command line enhancements as well, such as being able to force an ad-hoc backup for a workflow, even if the workflow’s policy is set to do a skip on the particular day you’re doing it.

Security isn’t missed, too. In situations where you have to backup via a storage node (it happens, even to the best of us – particularly in secure zones), you’ve now got the option to enable in-flight encryption of data between the client and the storage node. This therefore allows a totally encrypted datapath of say:

Client <Encrypted> Storage Node <–DDBoost Encrypted–> Data Domain


Client <Encrypted> Storage Node <–Tape FC Encryption–> Tape Drive

Obviously, the preference is always to go direct from the client to protection storage, but if you do have to go via a storage node, you’re covered now.

Logging gets enhanced, too. There’s now automatic numbering of lines in the log files to make it easier to trace messages for support and self-support.

Finally, NAS isn’t left out, either. There’s NetWorker Snapshot Management support now for XtremIO X2, and NetApp OnTAP 9.2. (On the NDMP front though – and I know snapshots are different, check out the insane levels of performance that we can get from dense filesystems on Isilon now, though.)

So, I’d planned to have a pretty quiet weekend of sitting under a freezing air-conditioner, out of the heat, and doing as little as possible. It seems instead I’ll be downloading NetWorker 9.2.1 and getting it up and running in my lab. (Lucky there’s room under the air-conditioner for my laptop, too.)

Basics – NMC VMware Viewer

 Basics, vProxy  Comments Off on Basics – NMC VMware Viewer
Nov 222017

As you might have noticed in other posts, I’m a big fan of using NVP (NetWorker Virtual Proxy – also referred to as vProxy) to backup VMware virtual machines. Introduced with NetWorker 9.2, the new VMware image backup system is lightweight and fast – both for backup and recovery operations.

One of other things you’ve probably noticed, using NetWorker, is that it’s all about giving you options on how to do things. At one point that was simply a choice between using the GUI, doing interactive command line operations, or scripted command line operations. More recently, the REST API was introduced, giving an additional level of interaction, ideal for private cloud or devops style environments.

In VMware environments, NetWorker also gives some flexibility between whether you want to use the vSphere Web UI (ideal for VMware administrators), or the above NetWorker options – GUI/NMC, CLI, CLI-scripted or REST API. But one of the real hidden gems, I think, is the VMware View section in NMC. This lets you start tackling a VMware environment from a “big picture” point of view, and that’s what I want to run through in this blog post.

First, let’s set the scene – you access the VMware View panel under the Protection tab in NMC:

VMware View 01 NMC

Finding VMware View in NMC

VMware View is in its own area, as you can see there. Now, you can still do VMware policy configuration, etc., as part of the standard Policies and Groups configuration areas, and indeed you’ll need to do at least some preliminary setup via standard policy/workflow management. However, once you’ve got the framework in place, VMware View gives you a fantastic way of quickly and simply interacting with your VMware environment. If you expand out the view, you’ll get details of vCenter servers/clusters and the defined datacentres. For my home lab, it’s pretty straight forward:

VMware View 02 Datacentres

vCenters/Datacentres in VMware View

Once you’ve selected a vCenter or Datacentre, you can start to visually see your virtual machine layout and the protection policies virtual machines are protected to. Here’s my home lab view, for instance:

vCenter System Tree

vCenter System Tree

The layout of that is straight forward – home is the virtual Datacentre, and there’s two ESX servers in the environment – kobol and tauron (astute observers will note I have a penchant for (mostly) naming systems after fictional planets, or at least things associated with science fiction. I am, after all, an adherent to RFC 1178).

You’ll see the resource groups for virtual machines as well, and over on the right from the virtual machines, you’ll see the individual policies, with dotted line connections running from protected virtual machines to the policies. You’ll also note there’s a [+] mark next to virtual machines and policy names, and [–] options in places as well. The [+] mark lets you expand out details – for a virtual machine, that’ll expand out to show the individual disks contained within the virtual machine (very useful if you only want to backup specific disks in the VM):

Expanded Virtual Machine View

Expanded Virtual Machine View

The [–] lets you effectively select an area of the configuration you want to focus on – it’ll highlight the entire tree for just that section, regardless of whether it’s a VMware resource group or an individual ESX server. In this case, for a resource group, you see:

VMware View Component Focus

VMware View Component Focus

The graphical view (I’ll call it a system tree) is handy in itself, but there’s some options to the right that can help you really focus on things you might need to do:

VMware View Quick Details

VMware View Quick Details

Here you get to see a zoomed out map of the system tree (and can control the zoom level on the system tree proper), but you can also choose to quickly jump between viewing specific things of high interest, viz.:

  • All virtual machines
  • All protected virtual machines
  • All unprotected virtual machines
  • All overprotected virtual machines
  • Any virtual machines that can’t be protected.

The initial system tree I showed earlier was the ‘All’ option. The most important view you can get in my opinion is the “VMs Unprotected” – this lets you focus only on those virtual machines that haven’t been added to protection policies:

Unprotected virtual machines

Unprotected virtual machines

Of course, you don’t have to jump back to the regular protection policies if you spot a virtual machine that you need adding to a protection policy. Any virtual machine in any view can be right-clicked on to expose the option to add or remove it to/from a protection policy:

Adjusting VM protection

Adjusting VM protection

From there you just click ‘Add to Group’ to add a virtual machine into a group, and by extension most likely, into an actual protection policy.

The over protected virtual machine view will show you virtual machines that belong to more than one policy:

Overprotected virtual machines

Overprotected virtual machines

The “VMs cannot be protected” view will show you any virtual machines which cannot be added to protection policies. In my environment, that’s just the virtual proxy machine itself:

VMs unable to be protected

VMs unable to be protected

And finally, you can view virtual machines that are members of protection policies:

Protected virtual machines

Protected virtual machines

The VMware View option in NMC really is quite straight forward to use, but knowing it’s there, and knowing what you can quickly see and do is a real boon for busy NetWorker administrators and operators. Don’t forget to ensure it’s in your collection of tools if you’re protecting VMware!

Nov 212017

If you’re a long term NetWorker user, you might be forgiven for focusing on just a few specific aspects of documentation whenever there’s a new release of the product. Usually, most people focus on the release notes, and then, branching out from the release notes, key details they think will be useful for their environment – particularly where it relates to significantly altered, or newly updated functions.

But it’s not just NetWorker-the-software that continues to be developed at DellEMC – there’s a continuing focus on enhancing the utility of the documentation as well.bigStock Book in Library

I was reminded of this important fact recently on an internal education session about NetWorker’s support for the Network Data Management Protocol, or NDMP. Chances are if you’ve backed up a NAS with NetWorker, you’ve used NDMP – the other approaches (such as backup via a redirected mount-point) are painful and the sort of thing you only resort to if for some reason you have to backup a Itsy-Bitsy home-NAS that doesn’t support NDMP.

If you’ve not revisited the NDMP documentation for NetWorker for a while, you’re in for a surprise.

In NetWorker 8.2 and earlier, NDMP was covered as a chapter in the main NetWorker administration guide – running from page 531 to 577 in the admin guide I have, or just a little over 45 pages. In NetWorker 9.0, NDMP coverage got broken out into its own document, the NDMP User Guide, running to 338 pages. (And then, additionally, there was a 95 page NAS Snapshot Management guide as well.)

In NetWorker 9.1, the NDMP user guide grew to 372 pages, and the NAS Snapshot Management Guide was 100 pages. A couple of extra pages appeared in the NDMP guide in 9.2, and there was a significant jump, up to 172 pages, in the NAS Snapshot Management Guide.

Now, that’s not just filler content – that’s targeted material, often broken down by array type, to provide you much more comprehensive information about managing your NDMP and NAS snapshot backups. If you’re still doing NDMP backups today the same way you were 5 or more years ago, you may very well be missing out on useful and more modern tips for protecting your large-scale unstructured data sources by not staying up to date on the documentation changes.

While we’re talking about NDMP, I want to mention some numbers I saw being discussed from a real customer environment. On an Isilon cluster, they had a 23TB region with over 200,000,000 files. That is your absolute “worst case scenario” archetypal dense filesystem sitting right there. Doing a single-threaded NetWorker backup in older versions of NetWorker, such a dense filesystem took a few days to complete a backup. However, NetWorker 9.0.1 and OneFS 8.0 introduced a very cool new feature – automatic multi-streaming for up to 32 save-streams from a single saveset definition. (This is effectively an evolution of Parallel Save Streams, or PSS, in NetWorker for traditional filesystems.) By upgrading to a more recent version of NetWorker and making use of multi-streaming on a couple of their Isilon nodes, they were able to bring that full backup down to 17 hours, and since full backups now completed in well under a day, they were also able to get incrementals done in around 2 hours. Think about that: processing 11.7 million files per hour out of an ultra dense filesystem. That really is smoking performance.

NetWorker Success Stories

 NetWorker  Comments Off on NetWorker Success Stories
Nov 062017

Last month I ran a birthday giveaway competition – tell me a NetWorker success story, and go in the running for a signed copy of my book, Data Protection: Ensuring Data Availability. Since then, it’s been a bit quiet on the NetWorker Hub, and I apologise for that: my time has been considerably occupied with either work or much needed downtime of late. Sometimes it really does seem that every month gets busier for me than the last. (And by “sometimes”, I sort of mean “every month”).

Knight in shining armour

One of the original symbols for NetWorker was a knight in shining armour – very much reflective of its purpose to protect the most valuable asset in your castle: your data. So it seems fitting that as I share some of the success stories I received, I use a knight as the image for the post. So let’s have at it.

Success Story #1:

With the book and blog it make me clear where lots of thing confusing on the Data Protection and helps me to present buying Data protection suite over TSM.

Hey, it may not specifically be a NetWorker success story, but I’m chuffed, regardless!

Success Story #2:

NetWorker gave me a career to be honest. I have come across multiple situations where a critical recovery or ad-hoc backup has saved someone’s job.

This is a story I can really identify with – NetWorker definitely gave me a career, too!

Success Story #3:

Had the experience recently where senior management was amazed with the fact that we managed to recover data up to last 24 hours with no loss otherwise for 7 file servers that were part of a BCP triggered to recover from bad weather in Houston. Came down to the team sharing with management on how the environment is backed up and how validation is done as a check and balance. Awesome experience when you realise that the investment on a good backup strategy and the governed implementation of the same does pay off during business continuity efforts.

Backup is good, but recovery is great. Being able to pull backup systems in to help provide business continuity is a great example of planning.

Success Story #4:

Saved my customers a lot of times when files has been deleted.

Again, I can agree with this one. That’s why NetWorker has been so important to me over the years – it’s helped so many customers in so many challenging situations.

Success Story #5:

Working with NetWorker since 7.6, I would say NetWorker and I are growing up together. I’m getting a better engineer year by year and NetWorker did the same. Today I’m doing things (like cluster backups and VM backups) I couldn’t imagine years ago.

My first NetWorker server really was called mars (you’ll get what I mean if you read enough NetWorker man pages), and we’ve both grown a lot since my earlier career as a Unix system administrator. My first server was v4.1, and I had v3 clients back then on a variety of systems. (I think the last time I used a v3 client was in 2000 to backup Banyan Vines systems.) File type devices, advanced file type devices, storage nodes, cluster support, Windows support, Linux support … the list goes on for things I’ve seen added to NetWorker over the years!

Success Story #6:

It does what it says on the tin.

Backs up and recovers servers.

What more can you ask for?

Succinct and true.

Success Story #7:

BMR recovery during a virus attack in environment really helped to tackle and restore multiple servers quickly.

(I hear great stories regularly about backups saving businesses during virus and ransomware attacks. Snapshots can help in those situations, of course, too, but the problem with snapshots is that a potent virus or ransomware attack can overwhelm your snapshot storage space, making a bad situation worse.)

Success Story #8:

When looking for a suitable replacement for IBM TSM 5.5/DataDomain VTL. We started to look Networker 8/DataDomain. We were blown away how it’s was so flexible and a powerfull integration with ESX.  We have better backup performance/restore and VM backup was so easy that management couldn’t believe I could backup 800 VM without deploying an agent on each server.

Here’s the thing: Data Domain will boost (no pun intended) a lot of average backup products, but you get the true power of that platform when you’re using a fully integrated product like NetWorker or Avamar.

Success Story #9:

We do BAU backup and restore with Networker and not much surprises there, but one capability/feature that saved us a lot of time/money was migrating from legacy DataDomain VTLs to NEW Datadomain Boost Target by just Cloning legacy VTLs.That gave us the opportunity to de-comm old system and still have access to legacy backups without requiring keeping the old devices and servers.

This is a great architectural story. Data Domain is by far the best VTL you can get on the market, but if you want to switch from VTL into true disk based backups, you can handle that easily with NetWorker. NetWorker makes moving backup data around supremely easy – and it’s great at ‘set and forget’ cloning or migration operations, too.

Success Story #10:

Restoring an entire environment of servers with Windows BMR special ISO.

I don’t see much call for BMR these days given the rise of virtualisation in the midrange market, but it’s still an option if you really need it.

Success Story #11:

I was able to take our backup tapes to a remote site in a different city and was able to recover the production servers, including the database servers, in less time than was planned for, thus proving that DR is possible using NetWorker.

NetWorker isn’t all about deduplication. It started at a time when deduplication didn’t exist, and it can still solve problems when you don’t have deduplication in your environment.

Success Story #12:

There are many however let me speak about latest. Guest level backups would put hell lot of load on underlying hypervisor on VM infrastructure. So we deployed NVP and moved all our file systems to it . The blazing speed and FLR helped us to achieve our SLA. Integration with NVP was seamless with 98% deduplication.

NVP really is an awesome success story. The centres of excellence have run high scale backups showing thousands of virtual machines backed up per hour. It really is game changing for most businesses. (Check at the end of the blog article for a fantastic real customer success story that one of the global data protection presales team shared recently.)

Success Story #13:

Have worked on multiple NMDA, NMSAP and DDBEA cases and have resolved them and the customer appreciates the DELL EMC support team.

Success stories come from customers and the people sitting on the other side of the fence, too. There’s some amazingly dedicated people in the DellEMC NetWorker (and more broadly, data protection) support teams … some of them I’ve known for over 15 years, in fact. These are people who take the call when you’re having a bad day, and they’re determined to make sure your day improves.

Success Story #14:

I believe to understand the difference between Networking and Networker was the biggest challenge as I was completely from the networking background.

There are a lot of success stories but I think to state or iterarte success in terms of networker is something which has been set by you and the bench mark for which is very high, so no success stories.

Hopefully I can replicate 5% of your success then probably I would be successful in terms of me.

I remember after I’d been using NetWorker for about 3 years, I managed to get into my first NetWorker training course. There was someone in the course who thought he was going into a generic networking course. And any enterprise backup product like NetWorker really well help you understand your business network a lot more, so this is a pretty accurate story, I think.

Success Story #15:

My success story is simple … every time I restore data for the company/users. Either it may be whole NetWorker server restore or Database (SAP,SQL,ORACLE etc) or file/folder or maybe a BMR.

Every “Thank You” Message I receive from end user gives me immense happiness when I restore data and I am privileged to help others by doing Data Protection. Highly satisfied with my work as its like a game for me. every time I  restore Something i treat it as win (Winning the Game).

Big or small, every recovery is important!

Success Story #16:

This story comes from Daniel Itzhak in the DPS Presales team. Dan recently shared a fantastic overview of a customer who’d made the switch to NVP backups with NetWorker. Dan didn’t share it for the competition, but it’s such a great view that I wanted to share it as part of this anyway. Here’s the numbers:

  • 1,124 Virtual Machines across multiple sites and vCenter clusters
  • 30 days of backups – Average 350 TB per day front end data being protected, 10.2PB logical data protected after 30 days.
  • Largest client in the environment – 302 TB. (That is one seriously big virtual machine!)
  • Overall deduplication ratio: 35x (to put that in perspective, 350TB per day at 35x deduplication ratio would mean on average 10TB stored per day)
  • More than 34,700 jobs processed in that time (VM environments tend to have lower job counts) … 99% of backups finish in under 2 hours every day.

That sounds impressive, right? Well, that’s not the only thing that’s impressive about it. Let’s think back to the NetWorker and Data Domain architecture … optimised data path, source based deduplication, minimum data hops, and storage nodes relegated to device access negotiation only. Competitive products would require big, expensive physical storage nodes/media servers to process that sort of data – I know, I’ve seen those environments. Instead, what did Dan’s customer need to run their environment? Let’s review:

  • 1 x RHEL v7.3 NetWorker Server, – 4 vCPUs with 16GB of RAM
  • 3 x Storage Nodes (1 remote, 2 local), each with: 4 vCPU and 32GB of RAM
  • 2 x NVP – Which you might recall, requires 8 GB of RAM and 4 vCPU

You want to backup 1000+ VMs in under 2 hours every night at 35x deduplication? Look no further than NetWorker and Data Domain.

I’ve contacted the winner – thanks to everyone who entered!

Hypervisor Direct – Convergent Data Protection

 Convergent Data Protection, Data Domain  Comments Off on Hypervisor Direct – Convergent Data Protection
Oct 102017

At VMworld, DellEMC announced a new backup technology for virtual machines called Hypervisor Direct, which represents a paradigm that I’d refer to as “convergent data protection”, since it mixes layers of data protection to deliver optimal results.

First, I want to get this out of the way: hypervisor direct is not a NetWorker plugin, nor an Avamar plugin. Instead, it’s part of the broader Data Protection Suite package (a good reminder that there are great benefits in the DPS licensing model).

As its name suggests, hypervisor direct is about moving hypervisor backups directly onto protection storage without a primary backup package being involved. This fits under the same model available for Boost Plugins for Databases – centralised protection storage with decentralised access allowing subject matter experts (e.g., database and application administrators) to be in control of their backup processes.

Now, VMware backups are great, but there’s a catch. If you integrate with VMware’s snapshot layer, there’s always a risk of virtual machine stun. The ‘stun’, we refer to there, happens when logged data to the snapshot delta logs are applied to the virtual machine once the snapshot is released. (Hint: if someone tries to tell you otherwise, make like Dorothy in Wizard of Oz and look behind the curtain, because there’s no wizard there.) Within NetWorker and Avamar, we reduce the risk of virtual machine stun significantly by doing optimised backups:

  • Leveraging changed block tracking to only need to access the parts of the virtual machine that have changed since the last backup
  • Using source based deduplication to minimise the amount of data that needs to be sent to protection storage

Those two techniques combined will allow you seamless virtual machine backups in almost all situations – in fact, 90% or more. But, as the old saying goes (I may be making this saying up, bear with me) – it’s that last 10% that’ll really hurt you. In fact, there’s two scenarios that’ll cause virtual machine stun:

  • Inadequate storage performance
  • High virtual machine change rates

In the case of the first scenario, it’s possible to run virtual machines on storage that doesn’t meet their performance requirements. This is particularly so when people are pointing older or under-spec NAS appliances at their virtual machine farm. Now, that may not have a significant impact on day to day operations (other than a bit of user grumbling), but it will be noticed during the snapshot processes around virtual machine backup. Ideally, we want to avoid the first scenario by always having appropriately performing storage for a virtual infrastructure.

Now, the second scenario, that’s more interesting. That’s the “10% that’ll really hurt you”. That’s where a virtualised Oracle or SQL database is 5-10TB with a 40-50% daily change rate. That size, and that change rate will smash you into virtual machine stun territory every time.

Traditionally, the way around that has been one or two (or both) data protection strategies:

  • LUN or array based replication, ignoring the virtual machine layer entirely. That’s good for a secondary copy but it’s going to be at best crash consistent. (It’s also going to be entirely storage dependent – locking you into a vendor and making refreshes more expensive/complex – and will lock you out of technology like vVOL and vSAN.)
  • In-guest agents. That’ll give you your backup, but it’ll be at agent-based performance levels creating additional workload stresses on the virtual machine and the ESX environment. And if we’re talking a multi-TB database with a high change rate – well, that’s not necessarily a good thing to do.

So what’s the way around it? How can you protect those sorts of environments without locking yourself into a storage platform, or preventing yourself from making architectural changes to your overall environment?

You get around it by being a vendor that has a complete continuum of data protection products and creating a convergent data protection solution. That’s what hypervisor direct does.

Hypervisor Direct

Hypervisor direct merges the Boost-direct technology you get in DDBEA and ProtectPoint with RecoverPoint for Virtual Machines (RP4VM). By integrating the backup process in via the Continuous Data Protection (CDP) functionality of RP4VM, we don’t need to take snapshots using VMware at all. That’s right, you can’t get virtual machine stun even in large virtual machines with high IO because we don’t work at that layer. Instead, leveraging the ESXi write splitter technology in RP4VM’s CDP, the RecoverPoint journal system is used to allow a virtual machine backup to be taken, direct to Data Domain, without impact to the source virtual machine.

Do you want to know the really cool feature of this? It’s application consistent, too. That 5-10TB Oracle or SQL database with a high change rate I was talking about earlier? Well, your DBA or Application Administrator gets to run their normal Oracle RMAN backup script for a standard backup, and everything is done at the back-end. That’s right, the Oracle backup or SQL backup (or a host of other databases) triggers the appropriate virtual machine copy functions automatically. (And if a particular database isn’t integrated, there’s still filesystem integration hooks to allow a two-step process.)

This isn’t an incremental improvement to backup options, this is an absolute leapfrog – it’s about enabling efficient, high performance backups in situations where previously there was no actual option available. And it still lets your subject matter experts be involved in the backup process as well.

If you do have virtual machines that fall into this category, reach out to your local DellEMC DPS team for more details. You can also check out some of the official details here.

Basics – device `X’ is marked as suspect

 Basics, Data Domain, NetWorker  Comments Off on Basics – device `X’ is marked as suspect
Sep 282017

So I got myself into a bit of a kerfuffle today when I was doing some reboots in my home lab. When one of my DDVE systems came back up and I attempted to re-mount the volume hosted on that Data Domain in NetWorker, I got an odd error:

device `X’ is marked as suspect

Now, that’s odd, because NetWorker marks savesets as suspect, not volumes.

Trying it out on the command line still got me the same results:

[root@orilla ~]# nsrmm -mv -f adamantium.turbamentis.int_BoostClone
155485:nsrd: device `adamantium.turbamentis.int_BoostClone' is marked as suspect

Curiouser curiouser, I thought. I did briefly try to mark the volume as not suspect, but this didn’t make a difference, of course – since suspect applies to savesets, not volumes:

[root@orilla ~]# nsrmm -o notsuspect BoostClone.002
6291:nsrmm: Volume is invalid with -o [not]suspect

I could see the volume was not marked as scan needed, and even explicitly re-marking the volume as not requiring a scan didn’t change anything.

Within NMC I’d been trying to mount the Boost volume under Devices > Devices. I viewed the properties of the relevant device and couldn’t see anything about the device being suspect, so I thought I’d pop into Devices > Data Domain Devices and view the device details there. Nothing different there, but when I attempted to mount the device from there, it instead told me the that the ‘ddboost’ user associated with the Data Domain didn’t have the rights required to access the device.

Insufficient Rights

That was my Ahah! moment. To test my theory I tried to login as the ddboost user onto the Data Domain:

[Thu Sep 28 10:15:15]
[• ~ •]
$ ssh ddboost@adamantium
EMC Data Domain Virtual Edition
You are required to change your password immediately (password aged)
Changing password for ddboost.
(current) UNIX password:



I knew I’d set up that particular Data Domain device in a hurry to do some testing, and I’d forgotten to disable password ageing. Sure enough, when I logged into the Data Domain Management Console, under Administration > Access > Local Users, the ‘ddboost’ account was showing as locked.

Solution: edit the account properties for the ‘ddboost’ user and give it a 9999 day ageing policy.

Huzzah! Now the volume would mount on the device.

There’s a lesson here – in fact, a couple:

  1. Being in a rush to do something and not doing it properly usually catches you later on.
  2. Don’t stop at your first error message – try operations in other ways: command line, different parts of the GUI, etc., just in case you get that extra clue you need.

Hope that helps!

Oh, don’t forget – it was my birthday recently and I’m giving away a copy of my book. To enter the competition, click here.

Birthday give-away competition

 NetWorker  Comments Off on Birthday give-away competition
Sep 272017
iStock Balloons

Towards the end of September each year, I get to celebrate another solar peregrination, and this year I’m celebrating it with my blog readers, too.

iStock Balloons

Here’s how it works: I’ve now been blogging about NetWorker on since late 2009. I’ve chalked up almost 700 articles, and significantly more than a million visitors during that time. I’ve got feedback from people over the years saying how useful the blog has been to them – so, running from today until October 15, I’m asking readers to tell me one of their success stories using NetWorker.

I’ll be giving away a prize to a randomly selected entrant – a signed copy of my book, Data Protection: Ensuring Data Availability.

The competition is open to everyone, but here’s the catch: I do intend to share the submitted stories. I take privacy seriously: no contact details will be shared with anyone, and success stories will be anonymised, too. If you want to be in the running for the book, you’ll need to supply your email address so I can get in contact with the winner!

The competition has closed.

Oh, don’t forget I’ve got a new project running over at Fools Rush In, about Ethics in Technology.

Basics – Understanding NetWorker Dependency Tracking

 Backup theory, NetWorker  Comments Off on Basics – Understanding NetWorker Dependency Tracking
Sep 162017

Dependency tracking is an absolutely essential feature within a backup product. It’s there to ensure you can recover data through the entire specified retention period for your backups, regardless of what mix of full, differential and/or incremental backups you do. It’s staggering to think there are some backup products out there (*cough* net *cough* ‘backup’), that treat backup retention with such contempt that they don’t bother to enforce dependency preservation.

Without dependency tracking, you’ve always got the risk that a recovery you want to do on the edge of your specified retention period might fail.

NetWorker does dependency tracking by default. In fact, it only does dependency tracking. To understand how dependency tracking works, and what that means for protecting your backups, check out my video below. (Make sure to switch it into High Definition – it’s not about being able to see more of my beard, but it is to make sure you can see all the screen content!)

Dependency tracking is such an important feature in data protection that you’ll find it’s also covered in my book, Data Protection: Ensuring Data Availability.

On another note, I’m starting a new project. I may work in IT, but I’ve always been a fan of philosophy, too. The new project is called Fools Rush In, and it’s going to be an ongoing weekly exploration of topics relating to ethics in IT and modern technology. It’s going to be long-form in its approach – the perfect thing to sit down and read over a cup of coffee or tea. This’ll be an exciting journey, and I’d love it if you joined me on it. The introductory article is …where angels fear to tread, and the latest post, What is Ethics? gives a bit of a primer on schools of ethical thought and how we can start approaching ethics in IT/technology.

Talking about Ransomware

 Architecture, Backup theory, General thoughts, Recovery, Security  Comments Off on Talking about Ransomware
Sep 062017

The “Wannacry” Ransomware strike saw a particularly large number of systems infected and garnered a great deal of media attention.

Ransomware Image

As you’d expect, many companies discussed ransomware and their solutions for it. There was also backlash from many quarters suggesting people were using a ransomware attack to unethically spruik their solutions. It almost seems to be the IT equivalent of calling lawyers “ambulance chasers”.

We are (albeit briefly, I am sure), between major ransomware outbreaks. So, logically that’ll mean it’s OK to talk about ransomware.

Now, there’s a few things to note about ransomware and defending against it. It’s not as simplistic as “I only have to do X and I’ll solve the problem”. It’s a multi-layered issue requiring user education, appropriate systems patching, appropriate security, appropriate data protection, and so on.

Focusing even on data protection, that’s a multi-layered approach as well. In order to have a data protection environment that can assuredly protect you from ransomware, you need to do the basics, such as operating system level protection for backup servers, storage nodes, etc. That’s just the beginning. The next step is making sure your backup environment itself follows appropriate security protocols. That’s something I’ve been banging on about for several years now. That’s not the full picture though. Once you’ve got operating systems and backup systems secured via best practices, you need to then look at hardening your backup environment. There’s a difference between standard security processes and hardened security processes, and if you’re worried about ransomware this is something you should be thinking about doing. Then, of course, if you really want to ensure you can recover your most critical data from a serious hactivism and ransomware (or outright data destruction) breach, you need to look at IRS as well.

But let’s step back, because I think it’s important to make a point here about when we can talk about ransomware.

I’ve worked in data protection my entire professional career. (Even when I was a system administrator for the first four years of it, I was the primary backup administrator as well. It’s always been a focus.)

If there’s one thing I’ve observed in my career in data protection is that having a “head in the sand” approach to data loss risk is a lamentably common thing. Even in 2017 I’m still hearing things like “We can’t back this environment up because the project which spun it up didn’t budget for backup”, and “We’ll worry about backup later”. Not to mention the old chestnut, “it’s out of warranty so we’ll do an Icarus support contract“.

Now the flipside of the above paragraph is this: if things go wrong in any of those situations, suddenly there’s a very real interest in talking about options to prevent a future issue.

It may be a career limiting move to say this, but I’m not in sales to make sales. I’m in sales to positively change things for my customers. I want to help customers resolve problems, and deliver better outcomes to their users. I’ve been doing data protection for over 20 years. The only reason someone stays in data protection that long is because they’re passionate about it, and the reason we’re passionate about it is because we are fundamentally averse to data loss.

So why do we want to talk about defending against or recovering from ransomware during a ransomware outbreak? It’s simple. At the point of a ransomware outbreak, there’s a few things we can be sure of:

  • Business attention is focused on ransomware
  • People are talking about ransomware
  • People are being directly impacted by ransomware

This isn’t ambulance chasing. This is about making the best of a bad situation – I don’t want businesses to lose data, or have it encrypted and see them have to pay a ransom to get it back – but if they are in that situation, I want them to know there are techniques and options to prevent it from striking them again. And at that point in time – during a ransomware attack – people are interested in understanding how to stop it from happening again.

Now, we have to still be considerate in how we discuss such situations. That’s a given. But it doesn’t mean the discussion can’t be had.

To me this is also an ethical consideration. Too often the focus on ethics in professional IT is around the basics: don’t break the law (note: law ≠ ethics), don’t be sexist, don’t be discriminatory, etc. That’s not really a focus on ethics, but a focus on professional conduct. Focusing on professional conduct is good, but there must also be a focus on the ethical obligations of protecting data. It’s my belief that if we fail to make the best of a bad situation to get an important message of data protection across, we’re failing our ethical obligations as data protection professionals.

Of course, in an ideal world, we’d never need to discuss how to mitigate or recover from a ransomware outbreak during said outbreak, because everyone would already be protected. But harking back to an earlier point, I’m still being told production systems were installed without consideration for data protection, so I think we’re a long way from that point.

So I’ll keep talking about protecting data from all sorts of loss situations, including ransomware, and I’ll keep having those discussions before, during and after ransomware outbreaks. That’s my job, and that’s my passion: data protection. It’s not gloating, it’s not ambulance chasing, it’s let’s make sure this doesn’t happen again.

On another note, sales are really great for my book, Data Protection: Ensuring Data Availability, released earlier this year. I have to admit, I may have squealed a little when I got my first royalty statement. So, if you’ve already purchased my book: you have my sincere thanks. If you’ve not, that means you’re missing out on an epic story of protecting data in the face of amazing odds. So check it out, it’s in eBook or Paperback format on Amazon (prior link), or if you’d prefer to, you can buy direct from the publisher. And thanks again for being such an awesome reader.

%d bloggers like this: