NetWorker 9.2 – A Focused Release

 NetWorker  Comments Off on NetWorker 9.2 – A Focused Release
Jul 292017
 

NetWorker 9.2 has just been released. Now, normally I pride myself for having kicked the tyres on a new release for weeks before it’s come out via the beta programmes, but unfortunately my June, June and July taught me new definitions of busy (I was busy enough that I did June twice), so instead I’ll be rolling the new release into my lab this weekend, after I’ve done this initial post about it.

bigStock Focus

I’ve been working my way through NetWorker 9.2’s new feature set, though, and it’s impressive.

As you’ll recall, NetWorker 9.1 introduced NVP, or vProxy – the replacement to the Virtual Backup Appliance introduced in NetWorker 8. NVP is incredibly efficient for backup and recovery operations, and delivers hyper-fast file level recovery from image level recovery. (Don’t just take my written word for it though – check out this demo where I recovered almost 8,000 files in just over 30 seconds.)

NetWorker 9.2 expands on the virtual machine backup integration by adding the capability to perform Microsoft SQL Server application consistent backup as part of a VMware image level backup. That’s right, application consistent, image level backup. That’s something Avamar has been able to do for a little while now, and it’s now being adopted in NetWorker, too. We’re starting with Microsoft SQL Server – arguably the simplest one to cover, and the most sought after by customers, too – before tackling other databases and applications. In my mind, application consistent image level backup is a pivot point for simplifying data protection – in fact, it’s a topic I covered as an emerging focus for the next several years of data protection in my book, Data Protection: Ensuring Data Availability. I think in particular app-consistent image level backups will be extremely popular in smaller/mid-market customer environments where there’s not guaranteed to be a dedicated DBA team within the IT department.

It’s not just DBAs that get a boost with NetWorker 9.2 – security officers do, too. In prior versions of NetWorker, it was possible to integrate Data Domain Retention Lock via scripting – now in NetWorker 9.2, it’s rolled into the interface itself. This means you’ll be able to establish retention lock controls as part of the backup process. (For organisations not quite able to go down the path of having a full isolated recovery site, this will be a good mid-tier option.)

Beyond DBAs and security officers, those who are interested in backing up to the cloud, or in the cloud, will be getting a boost as well – CloudBoost 2.2 has been introduced with NetWorker 9.2, and this gives Windows 64-bit clients the CloudBoost API as well, allowing a direct to object storage model from both Windows and Linux (which got CloudBoost client direct in a earlier release). What does this mean? Simple: It’s a super-efficient architecture leveraging an absolute minimum footprint, particularly when you’re running IaaS protection in the Cloud itself. Cloud protection gets another option as well – support for DDVE in the Cloud: AWS or Azure.

NMC isn’t left out – as NetWorker continues to scale, there’s more information and data within NMC for an administrator or operator to sort through. If you’ve got a few thousand clients, or hundred of client groups created for policies and workflows, you might not want to scroll through a long list. Hence, there’s now filtering available in a lot of forms. I’m always a fan of speeding up what I have to do within a GUI, and this will be very useful for those in bigger environments, or who prefer to find things by searching rather than visually eye-balling while scrolling.

If you’re using capacity licensing, otherwise known as Front End TB (FETB) licensing, NetWorker now reports license utilisation estimation. You might think this is a synch, but it’s only a synch if you count whitespace everywhere. That’s not something we want done. Still, if you’ve got capacity licensing, NetWorker will now keep track of it for you.

There’s a big commitment within DellEMC for continued development of automation options within the Data Protection products. NetWorker has always enjoyed a robust command line interface, but a CLI can only take you so far. The REST API that was introduced previously continues to be updated. There’s support for the Data Domain Retention Lock integration and the new application consistent image level backup options, just to name a couple of new features.

NetWorker isn’t just about the core functionality as well – there’s also the various modules for databases and applications, and they’ve not been left unattended, either.

SharePoint and Exchange get tighter integration with ItemPoint for granular recovery. Previously it was a two step process to mount the backup and launch ItemPoint – now the NMM recovery interface can automatically start ItemPoint, directing it to the mounted backup copies for processing.

Microsoft SQL Server is still of course supported for traditional backup/recovery operations via the NetWorker Module for Microsoft, and it’s been updated with some handy new features. Backup an recovery operations no longer need Windows administrative privileges in all instances, and you can do database exclusions now via wild-cards – very handy if you’ve got a lot of databases on a server following a particular naming convention and you don’t need to protect them all, or protect them all in a single backup stream. You also get the option during database recovery now to terminate other user access to the database; previously this had to be managed manually by the SQL administrator for the target database – now it can be controlled as part of the recovery process. There’s also a bunch of new options for SQL Always On Availability Groups, and backup promotion.

In addition to the tighter ItemPoint integration mentioned previously for Exchange, you also get the option to do ItemPoint/Granular Exchange recovery from a client that doesn’t have Exchange installed. This is particularly handy when Exchange administrators want to limit what can happen on an Exchange server. Continuing the tight Data Domain Cloud Tier integration, NMM now handles automatic and seamless recall of data from Cloud Tier should it be required as part of a recovery option.

Hyper-V gets some love, too: there’s processes to remove stale checkpoints, or merge checkpoints that exceed a particular size. Hyper-V allows a checkpoint disk (a differencing disk – AVHDX file) to grow to the same size as its original parent disk. However, that can cause performance issues and when it hits 100% it creates other issues. So you can tell NetWorker during NMM Hyper-V backups to inspect the size of Hyper-V differencing disks and automatically merge if they exceed a certain watermark. (E.g., you might force a merge when the differencing disk is 25% of the size of the original.) You also get the option to exclude virtual hard disks (either VHD or VHDX format) from the backup process should you desire – very handy for virtual machines that have large disks containing transient or other forms of data that have no requirement for backup.

Active Directory recovery browsing gets a performance boost too, particularly for large AD trees.

SAP IQ (formerly known as Sybase IQ) gets support in NetWorker 9.2 NMDA. You’ll need to be running v16 SP11 and a simplex architecture, but you’ll get a variety of backup and recovery options. A growing trend within database vendors is to allow designation of some data files within the database as read-only, and you can choose to either backup or skip read-only data files as part of a SAP IQ backup, amongst a variety of other options. If you’ve got a traditional Sybase ASE server, you’ll find that there’s now support for backing up database servers with >200 databases on them – either in sequence, or with a configured level of parallelism.

DB2 gets some loving, too – NMDA 9.1 gave support for PowerLink little-endian DB2 environments, but with 9.2 we also get a Boost plugin to allow client-direct/Boost backups for DB2 little-endian environments.

(As always, there’s also various fixes included in any new release, incorporating fixes that were under development concurrently in earlier releases.)

As always, when you’re planning to upgrade NetWorker, there’s a few things you should do as a matter of course. There’s a new approach to making sure you’re aware of these steps – when you go to support.emc.com and click to download the NetWorker server installer or either Windows or Linux, you’ll initially find yourself redirected to a PDF: the NetWorker 9.2 Recommendations, Training and Downloads for Customers and Partners. Now, I admit – in my lab I have a tendency sometimes to just leap in and start installing new packages, but in reality when you’re using NetWorker in a real environment, you really do want to make sure you read the documentation and recommendations for upgrades before going ahead with updating your environment. The recommendations guide is only three pages, but it’s three very useful pages – links to technical training, references to the documentation portfolio, where to find NetWorker focused videos on the Community NetWorker and YouTube, and details about licensing and compatibility. There’s also very quick differences details between NetWorker versions, and finally the download location links are provided.

Additional key documentation you should – in my mind, you must – review before upgrading include the release notes, the compatibility guide, and of course, the ever handy updating from a prior version guide. That’s in addition to checking standard installation guides.

Now if you’ll excuse me, I have a geeky data protection weekend ahead of me as I upgrade my lab to NetWorker 9.2.

Jan 132017
 

Introduction

There’s something slightly deceptive about the title for my blog post. Did you spot it?

It’s: vs. It’s a common mistake to think that Cloud Boost and Cloud Tier compete with one another. That’s like suggesting a Winnebago and a hatchback compete with each other. Yes, they both can have one or more people riding in them and they can both be used to get you around, but the actual purpose of each is typically quite different.

It’s the same story when you look at Cloud Boost and Cloud Tier. Of course, both can move data from A to B. But the reason behind each, the purpose for each is quite different. (Does that mean there’s no overlap? Not necessarily. If you need to go on a 500km holiday and sleep in the car, you can do that in a hatchback or a Winnebago, too. You can often get X to do Y even if it wasn’t built with that in mind.)

So let’s examine them, and look at their workflows as well as a few usage examples.

Cloud Boost

First off, let’s consider Cloud Boost. Version 1 was released in 2014, and since then development has continued to the point where CloudBoost now looks like the following:

CloudBoost Workflow

Cloud Boost Workflow

Cloud Boost exists to allow NetWorker (or NetBackup or Avamar) to write deduplicated data out to cloud object storage, regardless of whether that’s on-premises* in something like ECS, or writing out to a public cloud’s object storage system, like Virtustream Storage or Amazon S3. When Cloud Boost was first introduced back in 2014, the Cloud Boost appliance was also a storage node and data had to be cloned from another device to the Cloud Boost storage node, which would push data out to object. Fast forward a couple of years, and with Cloud Boost 2.1 introduced in the second half of 2016, we’re now at the point where there’s a Cloud Boost API sitting in NetWorker clients allowing full distributed data processing, with each client talking directly to the object storage – the Cloud Boost appliance now just facilitates the connection.

In the Cloud Boost model, regardless of whether we’re backing up in a local datacentre and pushing to object, or whether all the systems involved in the backup process are sitting in public cloud, the actual backup data never lands on conventional block storage – after it is deduplicated, compressed and encrypted it lands first and only in object storage.

Cloud Tier

Cloud Tier is new functionality released in the Data Domain product range – it became available with Data Domain OS v6, released in the second half of 2016. The workflow for Cloud Tier looks like the following:

CloudTier Workflow

CloudTier Workflow

Data migration with Cloud Tier is handled as a function of the Data Domain operating system (or controlled by a fully integrated application such as NetWorker or Avamar); the general policy process is that once data has reached a certain age on the Active Tier of the Data Domain, it is migrated to the Cloud Tier without any need for administrator or user involvement.

The key for the differences – and the different use cases – between Cloud Boost and Cloud Tier is in the above sentence: “once data has reached a certain age on the Active Tier”. In this we’re reminded of the primary use case for Cloud Tier – supporting Long Term Retention (LTR) in a highly economical format and bypassing any need for tape within an environment. (Of course, the other easy differentiator is that Cloud Tier is a Data Domain feature – depending on your environment that may form part of the decision process.)

Example use cases

To get a feel for the differences in where you might deploy Cloud Boost or Cloud Tier, I’ve drawn up a few use cases below.

Cloning to Cloud

You currently backup to disk (Data Domain or AFTD) within your environment, and have been cloning to tape. You want to ensure you’ve got a second copy of your data, and you want to keep that data off-site. Instead of using tape, you want to use Cloud object storage.

In this scenario, you might look at replacing your tape library with a Cloud Boost system instead. You’d backup to your local protection storage, then when it’s time to generate your secondary copy, you’d clone to your Cloud Boost device which would push the data (compressed, deduplicated and encrypted) up into object storage. At a high level, that might result in a workflow such as the following:

CloudBoost Clone To Cloud

CloudBoost Clone To Cloud

Backing up to the Cloud

You’re currently backing up locally within your datacentre, but you want to remove all local backup targets.  In this scenario, you might replace your local backup storage with a Cloud Boost appliance, connected to an object store, and backup via Cloud Boost (via client direct), landing data immediately off-premises and into object storage at a cloud provider (public or hosted).

At a high level, the workflow for this resembles the following:

CloudBoost Backup to Cloud

CloudBoost Backup to Cloud

Backing up in Cloud

You’ve got some IaaS systems sitting in the Cloud already. File, web and database servers sitting in say, Amazon, and you need to ensure you can protect the data they’re hosting. You want greater control than say, Amazon snapshots, and since you’re using a NetWorker Capacity license or a DPS capacity license, you know you can just spin up another NetWorker server without an issue – sitting in the cloud itself.

In that case, you’d spin up not only the NetWorker server but a Cloud Boost appliance as well – after all, Amazon love NetWorker + Cloud Boost:

“The availability of Dell EMC NetWorker with CloudBoost on AWS is a particularly exciting announcement for all of the customers who have come to depend on Dell EMC solutions for data protection in their on-premises environments,” said Bill Vass, Vice President, Technology, Amazon Web Services, Inc. “Now these customers can get the same data protection experience on AWS, providing seamless operational backup and recovery, and long-term retention across all of their environments.”

That’ll deliver the NetWorker functionality you’ve come to use on a daily basis, but in the Cloud and writing directly to object storage.

The high level view of the backup workflow here is effectively the same as the original diagram used to introduce Cloud Boost.

Replacing Tape for Long Term Retention

You’ve got a Data Domain in each datacentre; the backups at each site go to the local Data Domain then using Clone Controlled Replication are copied to the other Data Domain as soon as each saveset finishes. You’d like to replace tape for your long term retention, but since you’re protecting a lot of data, you want to push data you rarely need to recover from (say, older than 2 months) out to object storage. When you do need to recover that data, you want to absolutely minimise the amount of data that needs to be retrieved from the Cloud.

This is a definite Cloud Tier solution. Cloud Tier can be used to automatically extend the Data Domain storage, providing a storage tier for long term retention data that’s very cheap and highly reliable. Cloud Tier can be configured to automatically migrate data older than 2 months out to object storage, and the great thing is, it can do it automatically for anything written to the Data Domain. So if you’ve got some databases using DDBoost for Enterprise Apps writing directly, you can setup migration policies for them, too. Best of all, when you do need to recall data from Cloud Tier, Boost for Enterprise Apps and NetWorker can handle that recall process automatically for you, and the Data Domain only ever recalls the delta between deduplicated data already sitting on the active tier and what’s out in the Cloud.

The high level view of the workflow for this use case will resemble the following:

Cloud Tier to LTR NSR+DDBEA

Cloud Tier to LTR for NetWorker and DDBEA

…Actually, you hear there’s an Isilon being purchased and the storage team are thinking about using Cloud Pools to tier really old data out to object storage. Your team and the storage team get to talking and decide that by pooling the protection and storage budget, you get Isilon, Cloud Tier and ECS, providing oodles of cheap object storage on-site at a fraction of the cost of a public cloud, and with none of the egress costs or cloud vendor lock-in.

Wrapping Up

Cloud Tier and Cloud Boost are both able to push data into object storage, but they don’t have exactly the same use cases. There’s good, clear reasons why you would work with one in particular, and hopefully the explanation and examples above has helped to set the scene on their use cases.


* Note, ‘on-premise’ would mean ‘on my argument’. The correct term is ‘on-premises’ 🙂

Data Domain Updates

 Data Domain  Comments Off on Data Domain Updates
Oct 182016
 

I was on annual leave last week (and this week I find myself in Santa Clara).

Needless to say, the big announcements are often seemed to neatly align with when I go on annual leave, and last week was no different – it saw the release of a new set of Data Domain systems, DDOS 6.0, and the new Cloud Tiering functionality.

Cloud Transfer

Now, I know this is a NetWorker blog, but if repeated surveys have shown one consistent thing, it’s that a vast majority of NetWorker environments now have Data Domain in them, and for many good reasons.

You can find the official press release over at Dell EMC, but I’ll dig into a few of the pertinent details.

New Models

The new models that have been released are the 6300, 6800, 9300 and 9800. The key features of the new models are as follows:

  • Data Domain 6300
    • Max throughput per hour using Boost – 24 TB/hr
    • Max usable capacity – 178 TB
  • Data Domain 6800
    • Max throughput per hour using Boost – 32 TB/hr
    • Max usable capacity (active tier) – 288 TB
    • Max addressable Cloud Tier – 576 TB
    • Max total addressable (active + Cloud) – 864 TB
  • Data Domain 9300
    • Max throughput per hour using Boost – 41 TB/hr
    • Max usable capacity (active tier) – 720 TB
    • Max addressable Cloud Tier – 1,440 TB
    • Max total addressable (active + Cloud) – 2,160 TB
  • Data Domain 9800
    • Max throughput per hour using Boost – 68 TB/hr
    • Max usable capacity (active tier) – 1 PB
    • Max addressable Cloud Tier – 2 PB
    • Max total addressable (active + Cloud) – 3 PB

Those are all the sizes of course of actual storage – once your deduplication comes in your logical stored capacity can be considerably higher than the above.

All the models above introduce flash as part of the storage platform for metadata. (If you’re wondering where this will be handy, have a think about Instant Access, the feature where we can power up a Virtual Machine directly from its backup on the Data Domain.)

High Availability was previously only available on the DD9500 – it’s now available on the 6800, 9300, 9500 and 9800, making that extra level of data protection availability accessible to more businesses than ever.

DDOS 6

DDOS 6 is a big release, including the following new features:

  • Cloud Tier (more of that covered further on)
  • Boost FS Plugin – Allows a Linux host with an NFS mount from a DDOS 6 system to participate in Boost, reducing the amount of data that has to be sent over the filesystem mount to Data Domain storage
  • Enhancements to Secure Multi-Tenancy
  • Improvements to garbage collection/filesystem cleaning (and remember, it’s still something that can be run while other operations are taking place!)
  • Improvements to replication performance, speeding up virtual synthetic replication further
  • Support for ProtectPoint on systems with extended retention
  • Support for ProtectPoint on high availability systems
  • New minimally disruptive upgrades – starting in this release, individual software components will be able to be upgraded without full system reboots (unless otherwise required). This will reduce downtime requirements for upgrades and allow for more incremental approaches to upgrades.
  • Client groups – manage/monitor client activity and workflows for collections of clients, either at the Boost or NFS level. This includes being able to set hard/soft limits on stream counts, reporting on client group activities, and logging by client group. You can have up to 64 client groups per platform. (I can see every Data Domain administrator where DBAs are using DDBoost wanting to upgrade for this feature.)

Cloud Tier

Cloud Tier allows the Data Domain system to directly interface with compatible object storage systems and is primarily targeted for handling long term retention workloads. Data lands on the active tier still, but policies can be established to push that data you’re retaining for your long term retention out to cloud/object storage. While it supports storage such as Amazon and Azure, the real cost savings actually come in when you consider using it with Elastic Cloud Storage (ECS). (I’ve already been involved in deals where we’ve easily shown a 3-year TCO being substantially cheaper for a customer on ECS than Amazon S3+Glacier.)

But hang on, you might be asking – what about CloudBoost? Well, CloudBoost is still around and it still has a variety of use cases, but Cloud Tier is about having the Data Domain do the movement of data automatically – and without any need to rehydrate outgoing data. It’s also ideal for mixed access workloads.

Cloud Tier

Cloud Tier enables the Data Domain to actually address twice the maximum active tier capacity for any Data Domain model in object storage, drastically increasing the overall logical amount of data that can be stored on a model by model basis, and by pushing deduplicated data out to object storage, the processing time for data movement is unparalleled.

Summary/Wrapping Up

It was a big week for Data Domain, and DDOS 6 is setting the bar for deduplication systems – as well as laying the groundwork for even more enhancements over time.

(On a side node – apologies for the delay in posts. Leading up to taking that week off I was swamped.)

Apr 182016
 

I’ve been working my way through a pretty intense cold the last few days. To avoid spending the entire weekend playing Minecraft while I convalesce, I downloaded the newly released Data Domain Virtual Edition to start refreshing my lab. With DDVE including a performance tester, I was curious to see what my current lab setup would yield. (Until I finish updating my server, my lab is VMware ESX running within VMware Fusion on my late-2015 iMac. With 32GB of RAM and Thunderbolt-2 RAID, it’s serviceable but hardly ideal.)

I should point out – the title of this blog article is slightly inaccurate. It took me less than 30 minutes to install DDVE including filing a change request* – but it did take me about two hours to download the OVA file, thanks to ADSL speeds. The OVA is just 1.2GB in a zip file though, so if you’re not using internet based on RFC 1149 you should find it coming down very quickly.

Installing DDVE is such a cinch there’s no excuse not to have one running in your lab already! (Here’s the download link. Don’t forget to mosey along to the support.emc.com site as well and download the Installation guide for DDVE.)

Once the DDVE OVA was installed and my DNS was prepped, it was an incredibly straight forward install.

DDVE OVA Deployment

DDVE OVA Deployment #1

 

DDVE Deployment #2

DDVE Deployment #2

 

DDVE Deployment #3

DDVE Deployment #3

 

DDVE Deployment #4

DDVE Deployment #4

 

DDVE Deployment #5

DDVE Deployment #5

(Being the “free and frictionless” version, I chose the option for the 4TB configuration – there are a few tiers of options, and the 4TB option covers everything from the free 0.5TB through to the 4TB option.)

DDVE Deployment #6

DDVE Deployment #6

 

DDVE Deployment #7

DDVE Deployment #7

 

DDVE Deployment #8

DDVE Deployment #8

 

If you’re deploying DDVE for production use, or for earnest testing, you really should deploy it with thick provisioning (recommended in the install guide). Because I’m doing this just in my home lab, I switched over to thin provisioning, which I’ve done in the past and had adequate home testing performance.

After the OVA was deployed I edited the virtual machine before powering it up, adding a 500GB virtual disk (again for my purposes, thinly provisioned – you should use thick). The “free and frictionless” version of DDVE does not expire, but is limited to 0.5TB. (Even at this size, it’s actually quite generous once deduplication sizes come into play.)

DDVE Deployment #9

DDVE Deployment #9

Once the deployment was completed, I did something I’ve never done with a Data Domain before – elected to use the GUI configuration. This consisted of providing enough networking configuration to allow a web-browser connection to the DDVE, and then once logged in I could start configuring it graphically.

DDVE Deployment #10

DDVE Deployment #10

 

DDVE Deployment #11

DDVE Deployment #11

 

DDVE Deployment #12

DDVE Deployment #12

 

DDVE Deployment #13

DDVE Deployment #13

I was pretty stoked by this! Not only did my DDVE deployment assessment pass, but it passed by flying colours. That’s on a late 2015 iMac running DDVE within VMware ESX within VMware Fusion sitting on a 4 x 2TB 7200 RPM drives in a Thunderbolt-2 RAID-5 enclosure. (When I’ve done DDVE tests in the past on my iMac I’ve actually got great performance out of it so I’m not surprised, but it’s great to see the test results.)

It was just a few short steps after that and I had a Data Domain fully up and running, fully virtualised within my network.

In coming posts I’ll walk through connecting NetWorker to Data Domain and show some performance results of this setup, but I felt it worthwhile stepping through just how simple and easy it is to get a Data Domain setup in your environment now thanks to DDVE. If you’ve not worked with Data Domain before, there’s never been a better time to give it a go!


* The change request, roughly put, was to shout up the stairway, “Hey, I’m going to restart DNS for a few seconds for some hostname updates. Is that OK?”

Mar 302016
 

At the start of the week we saw NetWorker 8.2 SP3 released. Now, you might think given NetWorker 9 is out there’s no new features in NetWorker 8.2 SP3, but you’d be wrong.

NetWorker 9 is a jump – it’s a change of processes and it’s a new way of going about configuring your backups. I’m seeing more details every day of people having great experiences with NetWorker 9, but backup is one of those areas where change can often come slowly, so 8.2 still gets a lot of attention. So if you’re the sort of business that needs the features in NetWorker 9 you can dive in, but if you want to hang back for a little while yet, 8.2 will have you covered for a while yet.

nsrwatch in 8.2.3

The all new nsrwatch

OK, I admit I’m a bit of an nsrwatch junkie. Unless I have to setup a Windows NetWorker server I’ll setup NetWorker on Linux every time. (But at heart I’m still a Unix system administrator. It was the Unix integration that drove me to Mac OS X, after all.)

There’s a lot of good new features in NetWorker 8.2 SP3 but I have to admit given my CLI-junkie status, I just love the update to nsrwatch. For me this handy little utility has saved me thousands or more times from having to launch a full GUI, and if you’ve ever seen how many windows I end up having active on my screens at the same time you’ll understand why that’s a good thing.

The good old nsrwatch utility now gives you a lot more control over what you see on-screen. You can resize panels or even turn them off and setup an environment variable to make that your default view. You can switch between different views – e.g., all devices (seen above), mounted devices and active devices:

Mounted Devices

nsrwatch showing mounted devices only

nsrwatch showing active devices

nsrwatch showing active devices

You also get control options directly embedded into nsrwatch now:

nsrwatch with control options

nsrwatch with control options

All up, a great set of changes. I was lucky enough to try out some of the options while they were under development, so I’ve been looking forward to talking about it for some time now!

That’s not the only features in NetWorker 8.2 SP3 though – but it did really appeal to my I’ve-been-using-NetWorker-for-20-years inner-geek – so it’s time to move on to the rest of the enhancements!

Server Capability

There’s big changes under the hood in SP3 – the media catalogue has been migrated to SQLite to take advantage of the huge performance increases this gave in NetWorker 9 – and it’ll make the migration path to NetWorker 9 a little more streamlined as well. This may sound like a minor change, but the switch to SQLite is really important; the old format media database was great and stable, but it had limits on the amount of concurrent operations you could do. SQLite is great and stable and a lot more capable of supporting a number of concurrent operations.

The server daemons have had some tweaks as well – a bunch of issues that could lead to a server hang situation have been quashed, and the number of DNS reverse lookups performed has been pared down. The DNS caches used in a bunch of NetWorker daemons are now populated from nsrd to improve lookup performance as well. Also if you’ve got a lot of storage nodes in your environment, there are options to do a staggered start of the storage node manager daemons to improve startup performance.

Data Domain

8.2 SP3 includes support for DDOS 5.7 with an update of Data Domain libraries to 3.1. This will align it to some new options coming out soon, not to mention the Data Domain High Availability option introduced in the last month for the DD9500. (One of the other things it’ll align to I can blog about in a few days, hopefully.)

There’s performance enhancements for Clone Controlled Replication (CCR) as well, allowing for boosts (no pun intended) in the performance of cloning operations between two Data Domain systems under NetWorker control.

SP3 also introduces support for Distributed Segment Processing and all other Boost goodness into the Mac OS X client. That means if you’ve got some Mac clients within your NetWorker environment they’ll now get all of the Boost advantages you see everywhere else.

Updated Support

There’s a whole bunch of platforms and options that have had support added in this release. Check out the new VBA appliances if you’re backing VMware, too – you’ll definitely want to take advantage of updates there. But it’s not just VMware backups. This version of NetWorker also adds support for:

  • LTO7 tape drives
  • Mac OS X 10.11 El Capitan
  • SAP HANA SPS 11
  • Snapshot Management for NetApp SnapVault and SnapMirror ‘C-mode’ operations – creation, replication, restore and rollover
  • Hitachi NAS token based backups
  • Isilon Fast Incremental – Making backup of really large filesystems a whole lot easier
  • SQL Server AlwaysOn availability groups in a Failover Cluster (great way of offloading backups in SQL Enterprise Server environments)
  • MySQL 5.7.9/MySQL Enterprise Backup 4

In Summary

You won’t see the same sorts of massive features lists in a service pack release as you do in a full new release, but that being said 8.2 SP3 packs some wallop for your environment if you’re still in the 8.2 tree – or using an earlier version still. In addition to all the standard fixes that go into any service pack, rolled up from previous service packs and cumulative releases, 8.2 SP3 has been fine tuned for performance and scaleability and will ensure those customers not yet ready to upgrade to NetWorker 9 have an excellent platform to settle onto.

You can find the 8.2 SP3 binaries in the downloads section of the NetWorker product support page, and you can access the release notes directly from this link.

Mar 092016
 

I’ve been working with backups for 20 years, and if there’s been one constant in 20 years I’d say that application owners (i.e., DBAs) have traditionally been reluctant to have other people (i.e., backup administrators) in control of the backup process for their databases. This leads to some environments where the DBAs maintain control of their backups, and others where the backup administrators maintain control of the database backups.

Junction

So the question that many people end up asking is: which way is the right way? The answer, in reality is a little fuzzy, or, it depends.

When we were primarily backing up to tape, there was a strong argument for backup administrators to be in control of the process. Tape drives were a rare commodity needing to be used by a plethora of systems in a backup environment, and with big demands placed on them. The sensible approach was to fold all database backups into a common backup scheduling system so resources could be apportioned efficiently and fairly.

DB Backups with Tape

Traditional backups to tape via a backup server

With limited tape resources and a variety of systems to protect, backup administrators needed to exert reasonably strong controls over what backed up when, and so in a number of organisations it was common to have database backups controlled within the backup product (e.g., NetWorker), with scheduling negotiated between the backup and database administrators. Where such processes have been established, they often continue – backups are, of course, a reasonably habitual process (and for good cause).

For some businesses though, DBAs might feel there was not enough control over the backup process – which might be agreed with based on the mission criticality of the applications running on top of the database, or because of the perceived licensing costs associated with using a plugin or module from the backup product to backup the database. So in these situations if a tape library or drives weren’t allocated directly to the database, the “dump and sweep” approach became quite common, viz.:

Dump and Sweep

Dump and Sweep

One of the most pervasive results of the “dump and sweep” methodology however is the amount of primary storage it uses. Due to it being much faster than tape, database administrators would often get significantly larger areas of storage – particularly as storage became cheaper – to conduct their dumps to. Instead of one or two days, it became increasingly common to have anywhere from 3-5 days of database dumps sitting on primary storage being swept up nightly by a filesystem backup agent.

Dump and sweep of course poses problems: in addition to needing large amounts of primary storage, the first backup for the database is on-platform – there’s no physical separation. That means the timing of getting the database backup completed before the filesystem sweep starts is critical. However, the timing for the dump is controlled by the DBA and dependent on the database load and the size of the database, whereas the timing of the filesystem backup is controlled by the backup administrator. This would see many environments spring up where over time the database grew to a size it wouldn’t get an off-platform backup for 24 hours – until the next filesystem backup happened. (E.g., a dump originally taking an hour to complete would be started at 19:00. The backup administrators would start the filesystem backup at 20:30, but over time the database backups would grow and wouldn’t complete until say, 21:00. Net result could be a partial or failed backup of the dump files the first night, with the second night being the first successful backup of the dump.)

Over time backup to disk entered popularity to overcome the overnight operational challenges of tape, then grew, and eventually the market has expanded to include deduplication storage, purpose built backup appliances and even when I’d normally consider to be integrated data protection appliances – ones where the intelligence (e.g., deduplication functionality) is extended out from the appliance to the individual systems being protected. That’s what we get, for instance, with Data Domain: the Boost functionality embedded in APIs on the client systems leveraging distributed segment processing to have everything being backed up participate in its own deduplication. The net result is one that scales better than the traditional 3-tier “client/server/{media server|storage node}” environment, because we’re scaling where it matters: out at the hosts being protected and up at protection storage, rather than adding a series of servers in the middle to manage bottlenecks. (I.e., we remove the bottlenecks.)

Even as large percentages of businesses switched to deduplicated storage – Data Domains mostly from a NetWorker perspective – and had the capability of leveraging distributed deduplication processes to speed up the backups, that legacy “dump and sweep” approach, if it had been in the business, often remained in the business.

We’re far enough into this now that I can revisit the two key schools of thought within data protection:

  • Backup administrators should schedule and control backups regardless of the application being backed up
  • Subject Matter Experts (SMEs) should have some control over their application backup process because they usually deeply understand how the business functions leveraging the application work

I’d suggest that the smaller the business, the more correct the first option is – or rather, when an environment is such that DBAs are contracted or outsourced in particular, having the backup administrator in charge of the backup process is probably more important to the business. But that creates a requirement for the backup administrator to know the ins and outs of backing up and recovering the application/database almost as deeply as a DBA themselves.

As businesses grow in size and as the number of mission critical systems sitting on top of databases/applications grow, there’s equally a strong opinion the second argument is correct: the SMEs need to be intimately involved in the backup and recovery process. Perhaps even more so, in a larger backup environment, you don’t want your backup administrators to actually be bottlenecks in a disaster situation (and they’d usually agree to this as well – it’s too stressful).

With centralised disk based protection storage – particularly deduplicating protection storage – we can actually get the best of both worlds now though. The backup administrators can be in control of the protection storage and set broad guidance on data protection at an architectural and policy level for much of the environment, but the DBAs can leverage that same protection storage and fold their backups into the overall requirements of their application. (This might be to even leverage third party job control systems to only trigger backups once batch jobs or data warehousing tasks have completed.)

Backup Process With Data Domain and Backup Server

Backup Process With Data Domain and Backup Server

That particular flow is great for businesses that have maintained centralised control over the backup process of databases and applications, but what about those where dump and sweep has been the design principle, and there’s a desire to keep a strong form of independence on the backup process, or where the overriding business goal is to absolutely limit the number of systems database administrators need to learn so they can focus on their job? They’re definitely legitimate approaches – particularly so in larger environments with more mission critical systems.

That’s why there’s the Data Domain Boost plugins for Applications and Databases – covering SAP, DB2, Oracle, SQL Server, etc. That gives a slightly different architecture, viz.:

DB Backups with Boost Plugin

DB Backups with Boost Plugin

In that model, the backup server (e.g., NetWorker) still controls and coordinates the majority of the backups in the environment, but the Boost Plugin for Databases/Applications is used on the database servers instead to allow complete integration between the DBA tools and the backup process.

So returning to the initial question – which way is right?

Well, that comes down to the real question: which way is right for your business? Pull any emotion or personal preferences out of the question and look at the real architectural requirements of the business, particularly relating to mission critical applications. Which way is the right way? Only your business can decide.

Here’s a thought I’ll leave you with though: there’s two critical components to being able to make the choice completely based on business requirements:

  • You need centralised protection storage where there aren’t the traditional (tape-inherited) limitations on concurrent device access
  • You need a data protection framework approach rather than a data protection monolith approach

The former allows you to make decisions without being impeded by arbitrary practical/physical limitations (e.g., “I can’t read from a tape and write to it at the same time”), and more importantly, the latter lets you build an adaptive data protection strategy using best of breed components at the different layers rather than squeezing everything into one box and making compromises at every step of the way. (NetWorker, as I’ve mentioned before, is a framework based backup product – but I’m talking more broadly here: framework based data protection environments.)

Happy choosing!

Dec 222015
 

As we approach the end of 2015 I wanted to spend a bit of time reflecting on some of the data protection enhancements we’ve seen over the year. There’s certainly been a lot!

Protection

NetWorker 9

NetWorker 9 of course was a big part to the changes in the data protection landscape in 2015, but that’s not by any means the only advancement we saw. I covered some of the advances in NetWorker 9 in my initial post about it (NetWorker 9: The Future of Backup), but to summarise just a few of the key new features, we saw:

  • A policy based engine that unites backup, cloning, snapshot management and protection of virtualisation into a single, easy to understand configuration. Data protection activities in NetWorker can be fully aligned to service catalogue requirements, and the easier configuration engine actually extends the power of NetWorker by offering more complex configuration options.
  • Block based backups for Linux filesystems – speeding up backups for highly dense filesystems considerably.
  • Block based backups for Exchange, SQL Server, Hyper-V, and so on – NMM for NetWorker 9 is a block based backup engine. There’s a whole swathe of enhancements in NMM version 9, but the 3-4x backup performance improvement has to be a big win for organisations struggling against existing backup windows.
  • Enhanced snapshot management – I was speaking to a customer only a few days ago about NSM (NetWorker Snapshot Management), and his reaction to NSM was palpable. Wrapping NAS snapshots into an effective and coordinated data protection policy with the backup software orchestrating the whole process from snapshot creation, rollover to backup media and expiration just makes sense as the conventional data storage protection and backup/recovery activities continue to converge.
  • ProtectPoint Integration – I’ll get to ProtectPoint a little further below, but being able to manage ProtectPoint processes in the same way NSM manages file-based snapshots will be a big win as well for those customers who need ProtectPoint.
  • And more! – VBA enhancements (notably the native HTML5 interface and a CLI for Linux), NetWorker Virtual Edition (NVE), dynamic parallel savestreams, NMDA enhancements, restricted datazones and scaleability all got a boost in NetWorker 9.

It’s difficult to summarise everything that came in NetWorker 9 in so few words, so if you’ve not read it yet, be sure to check out my essay-length ‘summary’ of it referenced above.

ProtectPoint

In the world of mission critical databases where impact minimisation on the application host is a must yet backup performance is equally a must, ProtectPoint is an absolute game changer. To quote Alyanna Ilyadis, when it comes to those really important databases within a business,

“Ideally, you’d want the performance of a snapshot, with the functionality of a backup.”

Think about the real bottleneck in a mission critical database backup: the data gets transferred (even best case) via fibre-channel from the storage layer to the application/database layer before being passed across to the data protection storage. Even if you direct-attach data protection storage to the application server, or even if you mount a snapshot of the database at another location, you still have the fundamental requirement to:

  • Read from production storage into a server
  • Write from that server out to protection storage

ProtectPoint cuts the middle-man out of the equation. By integrating storage level snapshots with application layer control, the process effectively becomes:

  • Place database into hot backup mode
  • Trigger snapshot
  • Pull database out of hot backup mode
  • Storage system sends backup data directly to Data Domain – no server involved

That in itself is a good starting point for performance improvement – your database is only in hot backup mode for a few seconds at most. But then the real power of ProtectPoint kicks in. You see, when you first configure ProtectPoint, a block based copy from primary storage to Data Domain storage starts in the background straight away. With Change Block Tracking incorporated into ProtectPoint, the data transfer from primary to protection storage kicks into high gear – only the changes between the last copy and the current state at the time of the snapshot need to be transferred. And the Data Domain handles creation of a virtual synthetic full from each backup – full backups daily at the cost of an incremental. We’re literally seeing backup performance improvements in the order of 20x or more with ProtectPoint.

There’s some great videos explaining what ProtectPoint does and the sorts of problems it solves, and even it integrating into NetWorker 9.

Database and Application Agents

I’ve been in the data protection business for nigh on 20 years, and if there’s one thing that’s remained remarkably consistent throughout that time it’s that many DBAs are unwilling to give up control over the data protection configuration and scheduling for their babies.

It’s actually understandable for many organisations. In some places its entrenched habit, and in those situations you can integrate data protection for databases directly into the backup and recovery software. For other organisations though there’s complex scheduling requirements based on batch jobs, data warehousing activities and so on which can’t possibly be controlled by a regular backup scheduler. Those organisations need to initiate the backup job for a database not at a particular time, but when it’s the right time, and based on the amount of data or the amount of processing, that could be a highly variable time.

The traditional problem with backups for databases and applications being handled outside of the backup product is the chances of the backup data being written to primary storage, which is expensive. It’s normally more than one copy, too. I’d hazard a guess that 3-5 copies is the norm for most database backups when they’re being written to primary storage.

The Database and Application agents for Data Domain allow a business to sidestep all these problems by centralising the backups for mission critical systems onto highly protected, cost effective, deduplicated storage. The plugins work directly with each supported application (Oracle, DB2, Microsoft SQL Server, etc.) and give the DBA full control over managing the scheduling of the backups while ensuring those backups are stored under management of the data protection team. What’s more, primary storage is freed up.

Formerly known as “Data Domain Boost for Enterprise Applications” and “Data Domain Boost for Microsoft Applications”, the Database and Application Agents respectively reached version 2 this year, enabling new options and flexibility for businesses. Don’t just take my word for it though: check out some of the videos about it here and here.

CloudBoost 2.0

CloudBoost version 1 was released last year and I’ve had many conversations with customers interested in leveraging it over time to reduce their reliance on tape for long term retention. You can read my initial overview of CloudBoost here.

2015 saw the release of CloudBoost 2.0. This significantly extends the storage capabilities for CloudBoost, introduces the option for a local cache, and adds the option for a physical appliance for businesses that would prefer to keep their data protection infrastructure physical. (You can see the tech specs for CloudBoost appliances here.)

With version 2, CloudBoost can now scale to 6PB of cloud managed long term retention, and every bit of that data pushed out to a cloud is deduplicated, compressed and encrypted for maximum protection.

Spanning

Cloud is a big topic, and a big topic within that big topic is SaaS – Software as a Service. Businesses of all types are placing core services in the Cloud to be managed by providers such as Microsoft, Google and Salesforce. Office 365 Mail is proving very popular for businesses who need enterprise class email but don’t want to run the services themselves, and Salesforce is probably the most likely mission critical SaaS application you’ll find in use in a business.

So it’s absolutely terrifying to think that SaaS providers don’t really backup your data. They protect their infrastructure from physical faults, and their faults, but their SLAs around data deletion are pretty straight forward: if you deleted it, they can’t tell whether it was intentional or an accident. (And if it was an intentional delete they certainly can’t tell if it was authorised or not.)

Data corruption and data deletion in SaaS applications is far too common an occurrence, and for many businesses sadly it’s only after that happens for the first time that people become aware of what those SLAs do and don’t cover them for.

Enter Spanning. Spanning integrates with the native hooks provided in Salesforce, Google Apps and Office 365 Mail/Calendar to protect the data your business relies on so heavily for day to day operations. The interface is dead simple, the pricing is straight forward, but the peace of mind is priceless. 2015 saw the introduction of Spanning for Office 365, which has already proven hugely popular, and you can see a demo of just how simple it is to use Spanning here.

Avamar 7.2

Avamar got an upgrade this year, too, jumping to version 7.2. Virtualisation got a big boost in Avamar 7.2, with new features including:

  • Support for vSphere 6
  • Scaleable up to 5,000 virtual machines and 15+ vCenters
  • Dynamic policies for automatic discovery and protection of virtual machines within subfolders
  • Automatic proxy deployment: This sees Avamar analyse the vCenter environment and recommend where to place virtual machine backup proxies for optimum efficiency. Particularly given the updated scaleability in Avamar for VMware environments taking the hassle out of proxy placement is going to save administrators a lot of time and guess-work. You can see a demo of it here.
  • Orphan snapshot discovery and remediation
  • HTML5 FLR interface

That wasn’t all though – Avamar 7.2 also introduced:

  • Enhancements to the REST API to cover tenant level reporting
  • Scheduler enhancements – you can now define the start dates for your annual, monthly and weekly backups
  • You can browse replicated data from the source Avamar server in the replica pair
  • Support for DDOS 5.6 and higher
  • Updated platform support including SLES 12, Mac OS X 10.10, Ubuntu 12.04 and 14.04, CentOS 6.5 and 7, Windows 10, VNX2e, Isilon OneFS 7.2, plus a 10Gbe NDMP accelerator

Data Domain 9500

Already the market leader in data protection storage, EMC continued to stride forward with the Data Domain 9500, a veritable beast. Some of the quick specs of the Data Domain 9500 include:

  • Up to 58.7 TB per hour (when backing up using Boost)
  • 864TB usable capacity for active tier, up to 1.7PB usable when an extended retention tier is added. That’s the actual amount of storage; so when deduplication is added that can yield actual protection data storage well into the multiple-PB range. The spec sheet gives some details based on a mixed environment where the data storage might be anywhere from 8.6PB to 86.4PB
  • Support for traditional ES30 shelves and the new DS60 shelves.

Actually it wasn’t just the Data Domain 9500 that was released this year from a DD perspective. We also saw the release of the Data Domain 2200 – the replacement for the SMB/ROBO DD160 appliance. The DD2200 supports more streams and more capacity than the previous entry-level DD160, being able to scale from a 4TB entry point to 24TB raw when expanded to 12 x 2TB drives. In short: it doesn’t matter whether you’re a small business or a huge enterprise: there’s a Data Domain model to suit your requirements.

Data Domain Dense Shelves

The traditional ES30 Data Domain shelves have 15 drives. 2015 also saw the introduction of the DS60 – dense shelves capable of holding sixty disks. With support for 4 TB drives, that means a single 5RU data Domain DS60 shelf can hold as much as 240TB in drives.

The benefits of high density shelves include:

  • Better utilisation of rack space (60 drives in one 5RU shelf vs 60 drives in 4 x 3RU shelves – 12 RU total)
  • More efficient for cooling and power
  • Scale as required – each DS60 takes 4 x 15 drive packs, allowing you to start with just one or two packs and build your way up as your storage requirements expand

DDOS 5.7

Data Domain OS 5.7 was also released this year, and includes features such as:

  • Support for DS60 shelves
  • Support for 4TB drives
  • Support for ES30 shelves with 4TB drives (DD4500+)
  • Storage migration support – migrate those older ES20 style shelves to newer storage while the Data Domain stays online and in use
  • DDBoost over fibre-channel for Solaris
  • NPIV for FC, allowing up to 8 virtual FC ports per physical FC port
  • Active/Active or Active/Passive port failover modes for fibre-channel
  • Dynamic interface groups are now supported for managed file replication and NAT
  • More Secure Multi-Tenancy (SMT) support, including:
    • Tenant-units can be grouped together for a tenant
    • Replication integration:
      • Strict enforcing of replication to ensure source and destination tenant are the same
      • Capacity quota options for destination tenant in a replica context
      • Stream usage controls for replication on a per-tenant basis
    • Configuration wizards support SMT for
    • Hard limits for stream counts per Mtree
    • Physical Capacity Measurement (PCM) providing space utilisation reports for:
      • Files
      • Directories
      • Mtrees
      • Tenants
      • Tenant-units
  • Increased concurrent Mtree counts:
    • 256 Mtrees for Data Domain 9500
    • 128 Mtrees for each of the DD990, DD4200, DD4500 and DD7200
  • Stream count increases – DD9500 can now scale to 1,885 simultaneous incoming streams
  • Enhanced CIFS support
  • Open file replication – great for backups of large databases, etc. This allows the backup to start replicating before it’s even finished.
  • ProtectPoint for XtremIO

Data Protection Suite (DPS) for VMware

DPS for VMware is a new socket-based licensing model for mid-market businesses that are highly virtualized and want an effective enterprise-grade data protection solution. Providing Avamar, Data Protection Advisor and RecoverPoint for Virtual Machines, DPS for VMware is priced based on the number of CPU sockets (not cores) in the environment.

DPS for VMware is ideally suited for organisations that are either 100% virtualised or just have a few remaining machines that are physical. You get the full range of Avamar backup and recovery options, Data Protection Advisor to monitor and report on data protection status, capacity and trends within the environment, and RecoverPoint for a highly efficient journaled replication of critical virtual machines.

…And one minor thing

There was at least one other bit of data protection news this year, and that was me finally joining EMC. I know in the grand scheme of things it’s a pretty minor point, but after years of wanting to work for EMC it felt like I was coming home. I had worked in the system integrator space for almost 15 years and have a great appreciation for the contribution integrators bring to the market. That being said, getting to work from within a company that is so focused on bringing excellent data protection products to the market is an amazing feeling. It’s easy from the outside to think everything is done for profit or shareholder value, but EMC and its employees have a real passion for their products and the change they bring to IT, business and the community as a whole. So you might say that personally, me joining EMC was the biggest data protection news for the year.

In Summary

I’m willing to bet I forgot something in the list above. It’s been a big year for Data Protection at EMC. Every time I’ve turned around there’s been new releases or updates, new features or functions, and new options to ensure that no matter where the data is or how critical the data is to the organisation, EMC has an effective data protection strategy for it. I’m almost feeling a little bit exhausted having come up with the list above!

So I’ll end on a slightly different note (literally). If after a long year working with or thinking about Data Protection you want to chill for five minutes, listen to Kate Miller-Heidke’s cover of “Love is a Stranger”. She’s one of the best artists to emerge from Australia in the last decade. It’s hard to believe she did this cover over two years ago now, but it’s still great listening.

I’ll see you all in 2016! Oh, and don’t forget the survey.

Melbourne DPUG and VMware Data Protection

 Backup theory, Data Domain, VBA  Comments Off on Melbourne DPUG and VMware Data Protection
Sep 202015
 

Recently a colleague and I initiated the Melbourne Data Protection User Group (DPUG).

Pug in a pile of backup tapes

If you’re interested in joining and participating and based in Melbourne, you can find details for the user group over at Meetup.

Our first presentation was on Wednesday 9 September, and EMC Melbourne were kind enough to provide the office space for the session. That being said, DPUG is not about EMC products – it’s designed to be a vendor neutral community forum to discuss techniques, strategies and best practices relating to data protection.

Starting DPUG was a healthy reminder that data protection is an overloaded term in the IT industry. To those of us who work within data storage and more broadly, IT infrastructure, data protection covers concepts such as backup and recovery, continuous availability, continuous data protection, replication, snapshots and so on. For people who work at the application layer or communication layer though, data protection is almost invariably interpreted to be something like security, data privacy or intrusion detection/threat mitigation. Data protection is a term we share with other areas of the industry. In the end it’s all data protection, but it has two very different areas of focus.

Our first session was about VMware Data Protection. We’re now seeing a very high percentage of virtualisation within most businesses – it’s not uncommon to see 80% or 90% virtualisation now, and many companies are continuing to pursue a strategy of achieving 100% system and infrastructure virtualisation.

In the VMware Data Protection presentation I walked the audience through a history of how the industry overall has protected virtual machines since their inception in the midrange space. First, we started with treating virtual machines like regular hosts – installing agents on each virtual machine and backing it up as if it were no different from a physical host. That provides a high degree of granularity and flexibility, but as we know, virtualisation is about cooperative resource sharing, whereas traditional backups are about minimising the time it takes to get data from the client into the protection storage. There’s not a lot of compatibility between “cooperative resource sharing” and “minimising the time it takes to get data from the client…”, and a poorly designed backup strategy using in-guest backup agents can bring virtual infrastructure to a screaming halt – even today.

The next attempt to provide a comprehensive solution for backing up virtual machines saw businesses installing backup agent software on the hypervisors, and writing custom scripts to snapshot virtual machines prior to copying them to protection storage. This was usually error prone and when you stop to think about how virtual machines are usually just very big files, it meant that a single change within a virtual machine would trigger a new full backup every time. Once technology such as VMotion became available these techniques became difficult if not impossible to maintain – you could not really predict where a virtual machine would be for backups at any given time. What’s more, hypervisors are a bit like NAS appliances – they’re designed to do one thing really well, and you shouldn’t be trying to install third party software on them.

The solution was an API based approach, of course. While different in practice, you can equate the API approach of VMware backups to the NDMP approach of NAS. The virtualisation system provides an integration point for backup software to use, and leveraging that, backup products are able to streamline the data protection process with image level backups and file level recoveries from those image level backups.

This is something that NetWorker for instance has been doing for some time – most recently with VBA. VBA is something I’ve covered a few times over the last twelve months (Current state of Virtual Machine Backups in NetWorker, NetWorker 8.2 and VBA Instant-Access, and Testing and Debugging an Emergency Restore, for instance).

VMware offers its own version of VBA as well so that businesses (particularly smaller ones) can still protect their environments. It used to be split into VDP and VDP/A, but as of vSphere 6 Essentials, those options have been combined into a single (free) VDP. VDP can’t do everything VBA can do – for example, VDP can’t:

  • Perform instant-access to a virtual machine (powering on from Data Domain storage)
  • Perform tape-out
  • Write to storage other than Data Domain or internal storage

As a means of demonstrating some of the advantages of virtual machine image level backups though, VDP is useful, and that’s what I used in the DPUG session earlier this month. And now, after taking the plunge and investing in some screen recording software, I’ve made three of the demos from the DPUG session available for viewing. If you’re using VBA already you’ll be familiar with all of these. However, if you’ve not yet taken the plunge in utilising VBA for your backup environment, check them out – while the demos show the VMware Data Protection Appliance (VDP) in use, they’re equally applicable and in fact it’s the same process for a VBA install in each situation.

Creating and executing a protection policy:

Executing an image level recovery that makes use of changed block tracking:

Executing a file level recovery from an image level backup:

Don’t forget, if you’re in Melbourne and want to participate in DPUG, you’re more than welcome – regardless of whether you use EMC products or not. We want this to be an open group and look forward to seeing a broad spectrum of regular companies, integrators and vendors participating!

Also, if you’re interested in seeing screencasts for NetWorker related topics on this blog, let me know.

One target to rule them all

 Architecture, Backup theory  Comments Off on One target to rule them all
May 132015
 

IntroductionData Domain

It’s true there are some data types that broadly aren’t suitable to sending to Data Domain – any more than they’re suitable for sending to any other deduplication appliance or system within any environment. Large imaging data and video files will yield minimal deduplication except over successive backups (assuming static data), and compressed and/or encrypted data aren’t all suited either.

But the majority of data within most organisations is suited for writing to Data Domain systems.

Years ago when EMC purchased Data Domain, I don’t think anyone anticipated just what they had in mind for the appliance. I certainly didn’t – and I’d been involved in the backup industry for probably 15 years at that point. Deduplication had been kicking around for several years, but it hadn’t been mainstreamed to the degree EMC has achieved.

The numbers practically speak for themselves. Data Domain represents an overwhelming lions share of the deduplication appliance space – but I’m not going to quote numbers here. I’m going to talk about the architectural vision of Data Domain.

As a target-only appliance, Data Domain represents considerable advantage to any business that deploys it, but that’s just the tip of the iceberg. The real magic happens when we start to consider the simple fact that a Data Domain is not a dumb appliance. EMC have chosen to harness the platform to deliver maximum bang for buck for any company that walks down that path.

May the source be with you

Target based deduplication works brilliantly for drastically reducing the total amount of data stored, but it still results in that data being sent. Avamar demonstrates this overwhelmingly – its source based deduplication backup process is unbelievably efficient and powerful and is a powerfully attractive choice for many businesses, particularly those in the xaaS industry.

Data Domain’s Boost functionality extends its deduplication technology up to the origin of the data. For products like NetWorker, Avamar and VDP/VDPA, this goes right to the source. (For third party products such as NetBackup, it covers the media servers.)

If Boost had stopped at NetWorker and Avamar integration, it would have been a remarkably powerful efficiency hook for many businesses, but there’s more power to be had. The extension of Data Domain Boost to include support for enterprise applications such as Oracle, SQL Server, SAP, etc., provides unparalleled extensibility in the backup space to organisations. It also means that businesses who have deployed other backup technologies but leverage the power of Data Domain deduplication in their data protection strategy can get direct client deduplication performance for what is often their most mission critical systems and applications.

I’m the first to admit that I’ve spent years trying to convince DBAs to hand over control of their application backups to NetWorker administrators. It’s a discussion I’ve won as much as I’ve lost, but the Data Domain plugins for databases have proven one key lesson: when I’ve ‘lost’ that discussion it’s not been through lack of conviction, but through lack of process. DBAs are all for efficiencies in the backup process, but given the enterprise criticality of databases in so many organisations, much of the push back on backup centralisation has been from a lack of control of the process.

The Boost application plugins get past that by allowing a business to make the decision to integrate their application backups into centralised backup storage while allowing for highly granular control of the backup process through the agreed and trusted scheduling methods that offer considerably more granular and flexible controls. Backup products offer scheduling, of course, but they’re not meant to be the bees knees of scheduling that you’ll find in products devoted solely to that purpose. That’s what DBAs have mostly resisted. (This, for what it’s worth, is the difference between app-centric aspects to backup and recovery and a decentralised backup ‘system’.)

Here’s where we’re at with Data Domain – it now sits at a nexus in the Data Centre for data protection and nearline archival storage:

May the source be with you

(Yes, it’s even very well suited for archival workloads.)

NetWorker, Avamar, VDP/VDPA, Client Direct, Enterprise Apps – I could go on – Data Domain sits at the centre ready to receive the data you want to send to it.

But that diagram isn’t quite complete. To truly get the maximised efficiency out of Data Domain, the picture really should look more like this:

Protecting the Protection

That’s right – logically, a Data Domain solution will have at least two Data Domains in it, so that whatever you’re protecting via the Data Domain will itself be protected. Now, by itself, Data Domain offers excellent protection for the data you’re storing, but unlike what most people think of on this front, RAID-6 storage protection is just the tip of the iceberg. RAID-6 is nice – it protects you from two drive failures at any point. On top of that though, you have the Data Invulnerability Architecture that you’ll hear EMC folks talk about quite regularly – that’s the magic sauce. The Data Domain doesn’t just sit there storing your data: it stores it, it checks it, it reads it again, and it checks it as part of regular verification. (If you want to compare it to tape, imagine having a tape library big enough to store every tape you keep for retention and constantly sits there loading all the tapes and confirming all the data can be read back.)

But we all know in the data protection world that you still need that added protection of keeping a second copy of that data, regardless of whether that’s for compliance or for true disaster protection. In terms of absolute efficiency, the absolute best way you’ll get a secondary copy of that data is via the global deduplicated replication offered between two Data Domains. (For what it’s worth, that’s where some companies make the mistake of deploying tape as their secondary copy from an original backup target of Data Domain: what’s the point of deploying efficient deduplication if the first thing you’re going to do is rehydrate all the content again?)

Aside: Coming back to encryption and compression

Earlier I said that compressed and encrypted workloads aren’t necessarily suited to Data Domain. That’s true, but that usually reflects an opportunity to revisit the process and thinking behind those workloads.

Compression is typically used in a data streaming activity for data protection because of a requirement to minimise the amount of data going across the network. Boost eliminates that need by doing something better than compression at the client side – deduplication. Deduplication doesn’t just compress the original data, but it substantially reduces the original data by not even bothering to send data that already exists at the target. For instance, if I turn my attention to Oracle, the two most common reasons why DBAs will create compressed Oracle backups are:

(a) They’re writing them to primary storage and trying to minimise the footprint, or

(b) They’re writing them to NAS or some other form of network storage, and want to minimise the amount of data sent over busy links.

Both of those are squarely addressed by Data Domain:

  • For (a), the footprint is automatically reduced by writing it in uncompressed format to the Data Domain. It handles the deduplication automatically. In fact, it’ll be a lot more space efficient than say, the three most recent database backups being written to Tier-1/Primary storage.
  • For (b), because only unique data is sent over the network, and that data is compressed by Boost before it’s sent over the network, you’re still ending up with a more efficient network transfer than writing a compressed copy over the network.

Encryption might be considered a trickier subject, but it’s not really. There’s two types of encryption a business might require – at rest, or in-flight. Data Domain has supported encryption at rest for quite a long time, and the recent support for in-flight encryption has completed that piece of the puzzle. (That in-flight encryption is integrated in such a way that it still allows for local/source deduplication and associated pre-send compression, too.)

What all this means

When EMC first acquired Data Domain, they acquired a solid product that had already established excellent customer trust built from high reliability and performance. While both of those features have continued to grow (not to mention capacity … have you seen the specs on the Data Domain 9500?), those features alone don’t make for a highly extensible product (just a reliable big bucket of storage). The extensibility comes from the vertical integration right up into the application stack, and the horizontal integration across a multitude of use cases.

Last year’s survey results revealed a very high number of NetWorker environments leveraging Data Domain within their environment, but what we see if we step back a little bit from a single-product focus is that Data Domain is a strategic investment in the enterprise, able to be utilised for a plethora of scenarios across the board.

So there’s two lessons – one for those with Data Domain already, and one for those preparing to jump into deduplication: if you’ve already got Data Domain in your environment, start looking at its integration points and talking to either EMC or your supplier about where else Data Domain can offer synergies, and if you’re looking at deploying, keep in mind that it’s a highly flexible appliance capable of fitting in to multiple workloads.

Either way, that’s how you achieve an excellent return on investment.

Shared Data Domain Access

 Architecture, Data Domain, NetWorker  Comments Off on Shared Data Domain Access
Feb 022015
 

If you’ve come from a tape library environment, you’re well aware the only way to share a tape library between multiple NetWorker datazones – i.e., multiple NetWorker servers – is to use library partitioning. Thus, a 1000 slot library might be presented as 1 x 800 slot library and 1 x 200 slot library to the independent servers.

But what happens when you’re using Data Domain? That’s considerably more flexible.

Data Domain

If we look at basic Virtual Tape Libraries (VTLs), the answer is obvious: create as many VTLs as you need and present each one to a different NetWorker server. VTLs though, like their physical counterparts, suffer limitations that can make them impractical to use. You remain hampered by the number of drives provisioned in the library, and (far more importantly with VTL), you still can’t simultaneously write to a virtual tape and read from it. Looking down to specific technologies, VTLs are also severely limited in a NetWorker environment now when you compare them against Boost functionality. You can’t for instance use a Data Domain VTL for instant access or for granular Exchange recoveries. (Check out this post for more details of VTL vs Boost.)

You don’t have to use VTL to share a Data Domain between two or more NetWorker servers. Once you’ve configured Boost on a Data Domain, you can simply add the Data Domain to each NetWorker server you want to use it with, and a new Mtree will be automatically created for each NetWorker server as you create the first device on each.

For instance, this Data Domain system has 3 NetWorker servers communicating with and using it:

sysadmin@squeezebox# ddboost storage-unit show
Name      Pre-Comp (GiB)   Status
-------   --------------   ------
centaur             74.5   RW    
cyclops            181.8   RW    
medusa               6.6   RW    
-------   --------------   ------
 D    : Deleted 
 Q    : Quota Defined
 RO   : Read Only
 RW   : Read Write

One obvious advantage of using the same Data Domain system for multiple NetWorker servers is the deduplication still happens globally across all Mtrees. Thus, if you’ve got a production NetWorker environment and a test NetWorker environment, you can share a Data Domain between them and achieve a high deduplication rate, assuming your test environment is populated with copies of systems from your production environment.

(With the new multi-tenancy support in DDOS 5.5 and higher, you’ve got the additional advantage of using potentially multiple Boost accounts, thereby segregating access between the NetWorker servers. While a NetWorker server won’t have access to an Mtree for another NetWorker server, a shared Boost user will. Independent Boost users won’t.)

There are of course a couple of caveats to this approach. Including the need to either use multi-tenancy on the Data Domain or make sure you trust everyone who is simultaneously using it, you’ll need to consider the following:

  • Performance:
    • You’ll want to make sure the systems are mutually compatible for performance requirements. Two large-scale production NetWorker environments might for instance have a greater risk of causing performance challenges than say, one large scale production environment and small to medium size lab environment.
  • Stream count:
    • Almost a subset of performance, but it warrants calling out independently – regardless of whether you’ve got one NetWorker server or fifty NetWorker servers using a Data Domain system, you still have to face the recommended hard high limits on stream counts from the Data Domain administration guides for any unit you’re using. With that in mind, you’ll likely want to use NetWorker’s pool based parallelism settings to ensure there are hard limits established to the number of simultaneous savesets NetWorker can write.

Sharing a Data Domain between multiple NetWorker servers won’t be a solution for every environment, but if you’re limited for budget it’s something to keep in mind.