A probe backup typically runs with the following logic:

• Execute command on client
• Does client exit indicating a backup is required?
• If yes: Run backup
• If no: Don’t run backup

All fairly straight forward. Unfortunately, for all of 7.6 SP2 and 7.6 SP3 (base), the probe logic on the server would mistakenly trigger a new backup every time the probe was run. I.e., it wasn’t properly detecting whether the client returns the “no backup required” signal.

This would likely have gone unnoticed in a lot of environments using probe backups, since common uses include:

• Backing up and cleaning up database log files;
• Initiating a once-daily backup in response to a particular condition being met, with short-term probe windows.

However, if your probe was designed to run multiple times during the day and you only wanted one backup, you got a different result.

I’d noticed this behaviour for a while, but never got around to investigating it. So, ironically, when I finally logged a case with EMC about it, I was told that 7.6.3.1 would be out shortly and would resolve the issue. Which, it did.

So, if you’re using probe based backups and you’re currently on either 7.6 SP2 or 7.6 SP3, and the correct behaviour of those probes is important, you need to jump across to 7.6 SP3 CR1 (otherwise known as 7.6.3.1).

Note: The incorrect probe behaviour is actually dictated here by the NetWorker server, not the client.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

A common theme of question asked by people new to NetWorker is whether it supports a push or pull recovery model.

The answer, as you’d expect for an enterprise backup product, is both. However, the recoveries processes aren’t named push and pull.

If you’re not aware of push and pull recovery models, they work thusly:

• A push recovery model is where all recovery requests are handled by the backup administrator, or at least, on the backup server, and the data retrieved is transferred out to the client.
• A pull recovery model has the client that wishes to receive the data initiate the recovery and retrieve the data from the backup server.

NetWorker supports both, and in fact more, but it uses the term directed recoveries.

Technically, all recoveries in NetWorker are directed. They involve three clients, which are:

• Source – the host from where the data was originally backed up;
• Target – the host that the data is to be recovered from;
• Control – the host that initiates the data.

Now, because the backup server has the NetWorker client software on it, it can be any one of those clients. A workgroup style “push” recovery would typically work with clients aligned as follows:

• Source and Target – Host where the data came from originally
• Control – The backup server

On the other hand, a workgroup style “pull” recovery would typically work with clients aligned:

• Source, Target and Control – The host where the data came from originally.

NetWorker’s directed recovery model is more powerful and flexible than the above two examples, though. For example, you can run a recovery where all three hosts are different machines – e.g.:

• Source – Production database server
• Target – Development database server
• Control – Backup server

In this situation the directed recovery would be used to act as a means of getting data from production into the development area.

So the answer to that original question is: yes, NetWorker supports a push recovery model. And yes, NetWorker supports a pull recovery model. But it also supports more.

Share this using...

Related posts:

1. Basics – NetWorker Server Client ID When reviewing a customer’s logs today, I noticed yet another...
2. Basics – Why is NetWorker asking for a Default tape? So you’ve got a pending message sitting there in NMC,...
3. 10 NetWorker Configuration Rules Over the last 15 years, I’ve administered, configured and supported...

Related posts brought to you by Yet Another Related Posts Plugin.

So a new version of NetWorker has come out, or is coming out, and it’s been decided that you’re going to upgrade, but you want a few tips for making that upgrade as painless as possible. Here’s my 5 rules for upgrading NetWorker:

1. Read the release notes. If you’re not going to read the release notes, you are better off staying on your current version, no matter what issues you’re having. I can’t stress enough the importance of reading the release notes and having a thorough grasp of:
• What has changed?
• What are the known issues with the current release?
• What were the resolved issues between the current release and the release you’re currently running?
2. Do a bootstrap and index backup if upgrading between major or minor releases. If going between service packs on the same release, you can skip the index backup so long as your backups have been successful lately, but ensure you still do a bootstrap backup.
3. Unload all tapes (physical or virtual) in jukeboxes before the upgrade. You’ll see why shortly.
4. Upgrade in this order:
• Storage node(s) on the day of the upgrade, before the NetWorker server
• Server on the day of the upgrade, after the storage node(s)
• Client(s) later, at suitable times
5. After the upgrade but before the NetWorker services are restarted on the storage node(s) and server, delete the nsr/tmp directory on those hosts.

Obviously standard caveats, such as following any additional instructions in the release notes or upgrade notes should of course be followed, but sticking to the above rules as well can save a lot of hassle over time. I’ve noticed over the years that a odd, random problems following upgrades can be solved by clearing the nsr/tmp directory on the server and storage nodes. If there’s no tapes in the jukeboxes when the services first start after the upgrade, there’s less futzing for NetWorker to take care of before it’s fully up and running, too.

Share this using...

Related posts:

1. 10 NetWorker Configuration Rules Over the last 15 years, I’ve administered, configured and supported...
2. Basics – NetWorker Server Client ID When reviewing a customer’s logs today, I noticed yet another...
3. New NetWorker Technical Documents (No, not micromanuals – official EMC Technical Notes for NetWorker.)...

Related posts brought to you by Yet Another Related Posts Plugin.

10 – New Micromanual – LinuxVTL and NetWorker

The second micromanual was a step-by-step guide for configuring the open source LinuxVTL system with NetWorker. I had hoped when I started writing micromanuals that I’d get them more frequently delivered, but various factors get in the way of this. Maybe in 2012 I’ll be able to get a couple more out and available.

9 – Killing scheduled cloning operations

When NetWorker’s scheduled clone option was introduced, there were a few bugs relating to stopping a scheduled clone operation from the GUI. Sometimes you could, and sometimes you couldn’t. However, you could always kill a scheduled clone job from the command line, which is what this post explained.

8 – NetWorker Firewall Configuration on Windows

Very early in the year I was doing a lot of work with NetWorker on Windows 2008 R2, and I was noticing a few gaps in the installation process when it came to the process of automated configuration of the Windows Firewall to work with NetWorker daemons. This post explained the lessons I learnt.

7 – Carry a jukebox with you (if you’re using Linux)

This article was my first post about configuring the open source LinuxVTL system with NetWorker. Since then LinuxVTL has evolved quite a lot, and I’ll likely even need to update that micromanual early in the new year as a consequence.

6 – Why I’d choose NetWorker over NetBackup Every Time

Despite the fact that the article was titled “Why I’d choose…”, I had a rather indignant response to this post insisting I was being a jerk by writing it. I stand by every word in that post. I would not, personally, elect to choose NetBackup over NetWorker on the basis that NetBackup only has true image recovery as an option, and that NetBackup doesn’t support dependency chains for backup images. I see both of these factors as critical to a true enterprise backup product, and NetBackup only half supports one of them. That doesn’t make me a jerk, it makes me someone who gives a damn about your data.

5 – Using NetWorker Client with Opensolaris

A guest article written by Ronny Egner, this post covered off getting the NetWorker client working with the OpenSolaris version of Solaris.

4 – Basics – Fixing “NSR peer information” errors

A persistent challenge in NetWorker is when the NSR peer information gets out of whack; usually this can happen when a significant change happens on a client, and the server must have this information reset. I’d still love to see this article become irrelevant by seeing an option appear in NMC to handle it, but until then, this will remain a fairly popular article.

3 – This is wrong

Earlier this year, an Australian hosting service lost thousands of hosted domains and websites due to a “hack attack”. Supposedly the clever hackers destroyed not only the production data, but also all the backups.

What really went wrong was that the company in question had designed a very poor and inadequate backup solution. Rumours were abounding at the time that backups were just simply replicated snapshots. Snapshots may be able to act as backups, but not indefinitely, and not if they’re the only thing configured. (Backups and snapshots are effectively ‘sister’ activities in ILP.)

2 – micromanual: NetWorker Power User Guide to nsradmin

The original micromanual – “NetWorker power user guide to nsradmin” was and remains extremely popular. There’s been thousands of downloads of it since its release, including quite a number from EMC themselves, so it’s clearly a handy resource. If you’ve not downloaded it yourself but you want to boost your NetWorker productivity, it’s a must read.

1 – NetWorker 7.6 SP1

When NetWorker 7.6 SP1 came out, it was a huge release. In my opinion, it should have been numbered NetWorker 7.7 at least; it wasn’t a minor set of changes or a round of bug fixes, it included significant functionality updates (including one of my favourites – support for Boost). As the number one read article of the year, it’s been a big resource for people looking at the functionality of newer releases of NetWorker.

And that, they say, is that

This year has personally been a huge year for me. My partner and I moved state/city in June, going from a regional area just outside of Sydney to the inner west of Melbourne. We also celebrated our 15th anniversary together, surrounded by many of our new friends (who are like family to us) and a few of our old friends. We were even invited to get on the radio to talk about that, not only from the longevity of the relationship and having run the anniversary party up against the monthly Melbourne Den night. (There’s a podcast coming…) It was also the year when I sorted a lot of stuff out, and to boil all this down: it was the year that I spent a lot of time focusing on my personal life and not so much on the blog.

There may still be one or two posts left for 2011, but I’m also starting to get my head around changes and new material for 2012, and I believe 2012 will be a big year for NetWorker users.

Share this using...

Related posts:

1. NetWorker Usage Survey, June 2011 It’s that semi-annual time of the year again when I...
2. NetWorker Usage Survey for June 2011 Results With apologies for the delays, I’m pleased to announce that...

Related posts brought to you by Yet Another Related Posts Plugin.

Rather than continue to vacillate, I’ve decided to do a sample podcast, make it available here for downloading, and decide what to do based on feedback received.

While raw technical posts don’t translate well to podcasts (how do you quote screen output, for instance?), there’s a lot of backup theory related posts I make which can readily converted.

So, please follow the link below to the first podcast, in which I go over a topic near and dear to my heart: What is a zero error policy?

If you’re interested in me producing more podcasts, please let me know. Without feedback, I’ll likely leave it at just this trial. If people are interested though, I’ll setup a proper podcast stream within iTunes and get to work.

Podcast 001: What is a zero error policy?

Cheers!

Share this using...

Related posts:

1. No Zero Error Policy? No backup system In the past I’ve talked about the importance of having...
2. Don’t eliminate the error Long-term blog readers will know that I advocate a zero...
3. You’re not the boss of me Consider the following two questions: Do you manage your backups,...

Related posts brought to you by Yet Another Related Posts Plugin.

As you may imagine, in my time I’ve picked up a few ways of working with NetWorker, and I thought I’d share some of these as my “golden” configuration rules. These are the things that I stick to, regardless of individual design considerations. I.e., don’t consider them to be design rules; they’re different again.

1. Don’t use the default resources

I make it a policy not to use the default resources that NetWorker provides. This isn’t to say that they’re not appropriate at times, but simply that you should own your own configuration. You should establish a naming standard for each of the core configuration items (groups, pools, policies, schedules, etc.) and use that to keep a consistent, uniform configuration, rather than mixing in bits of your own configuration and the default configuration.

Further, there are some default resources that you can’t modify – pools are a classic example. And in those cases, you want to be able to enable certain settings, such as auto media verify. Since you can’t modify bootstrap pools, you may as well start from scratch there.

Exception: Notifications. While there’s a couple of notifications whose alerts you can’t change, for the most part, start by modifying the existing ones for things like savegroup completion, cleaning alerts, etc.

2. Don’t name groups after their start time

I see this time and time again – groups get named after their start time. If you have an entirely static and unchanging configuration, you may sometimes get away with this. However, for the most part, you’re going to need to be flexible on shuffling around group start times from time to time. E.g., you may need to pull a group forward five minutes, or push its start time back ten minutes, etc.

If the group is named after the start time, you’ve got two options:

1. Give it a different start time to the name, making the configuration violate the law of least astonishment; or
2. Create a new group, move the clients across to it, delete the old group, adjust pool configurations, etc.

Either way it’s messy and unpleasant. The best approach is to just not insert the group start time into the name of the group – after all, if you go into NMC and look at a listing of all groups, you’ll see the start time immediately anyway!

If for some reason you really need to include some form of time in the group name, keep it as fuzzy as possible; e.g., “pre midnight” or “post midnight” might be one way of doing it, or even just “Early” and “Late”.

3. Use as few pools as possible

The more pools you have, the more media you’ll need to use (for the most part). It also introduces drive contention and makes performance tuning and tweaking of an environment more challenging. Therefore, keep pools to a minimum, focusing on using them for any/all of the following:

• Segregating backups based on retention periods/frequency of backup (e.g., a “Daily” pool and a “Monthly” pool);
• Segregating backups based on locality (e.g., “Daily Offsite” and “Daily Onsite”).

If you keep the number of pools you use in your environment to a minimum (while still having the number you need), you’ll have a much easier to maintain environment.

4. Avoid adjusting pools while the server is active

In the dim dark days of NetWorker history, you couldn’t edit pools while the server was backing up. Over time that restriction has been lifted. However, there are still all sorts of situations that can trigger NetWorker to log the dreaded message about pools being edited while the server was busy. And if this gets logged, your pool changes won’t take effect until you can stop and restart NetWorker.

The solution? Avoid it. Plan the changes that you need to make to pools, and slot them into change windows where backups will be minimised or not happen. Equally, design your solution around the knowledge that pool modifications while the server is active can be a bit painful – e.g., having clients and/or savesets explicitly specified in a pool selection criteria should be an exception, not a rule.

5. Always enable Monitor RAP

NetWorker has a facility to track changes to the configuration which is called “Monitor RAP” – it’s a server resource setting, and it’s disabled by default. Once you enable it though, a RAP log is generated in the server’s log directory which maintains details of everything that gets changed (either by an administrator, or NetWorker itself) in the configuration. This not only helps in any audit situation, it also lets you back-trace configuration changes and stay appraised of changes to the environment when you have more than one person with administrative privileges in the datazone.

6. Don’t use wildcards for the admin usergroup

No, don’t.

It’s that simple.

7. Use schedule overrides to establish better monthly schedules

When creating schedules where you say, need to have monthly backups that skip all days of the month except the last Friday of the month, switch out of calendar view and use fuzzy time definitions for overrides – e.g., and override of “full last friday every month”. It’ll save you a lot of hassle!

If you want to know how to do this, check out the examples in the Power Users’s Guide to nsradmin micromanual.

8. Give jukeboxes sensible names

When you run a configuration wizard, NetWorker will name the jukebox by the SCSI port that it finds the jukebox on. This is all well and good, but that port isn’t necessarily static – it can be moved around due to various operating system changes, etc. What’s more, it’s usually not all that human-friendly in terms of remembering, etc.

However, you can temporarily disable the jukebox and rename it.

I tend to rename the jukebox to the model type – e.g., “i500″ or something simple along those lines. This is also the case when there’s only one jukebox attached to each storage node – the “rd=hostname” component of the jukebox name will give you some separation between what’s configured on the storage node and what’s configured on the server. If you have multiple jukeboxes in the same location – particularly if they’re the same model, you might append numbers to the end of the names (e.g., DD_VTL1, DD_VTL2, etc.)

If you’ve got multiple similar jukeboxes in disparate locations say, fibre-channel connected to a single host, you might include an abbreviation of the location in the jukebox name – e.g., “DD_VTL_DC1″ and “DD_VTL_DC2″ for “Data Centre 1″ and “Data Centre 2″ – you get the drift…

9. Use the comment field

I’ll sound like an old person here, but let me put it to you this way:

Us old time NetWorker users campaigned long and hard to get a comment field – use it, or you’re being ungrateful!

Seriously though, the comment field is a great way of recording easy-to-use annotations to help make your configuration even easier to understand. Don’t go crazy with it and try to encapsulate the entire configuration for each resource in its comment field, but use it like a good programmer would use code comments.

10. Don’t mix special and filesystem savesets in the same group

Sometimes I’ll see sites where you have the same client in a group twice; the first time the client has savesets of say:

• /
• /opt
• /usr
• /usr/local
• /home

And the second instance of the client is for a module – e.g.,

• RMAN:OracleDB_Details

You see, NetWorker doesn’t let you have two instances of a client in the same group when one of the clients has an “All” saveset; so, in the example above, by rights the first client instance should have an “All” saveset rather than an explicit listing of savesets. But when both clients are shoe-horned into the backups, you move to an inclusive rather than exclusive backup policy, and that’s just dangerous for data protection, and introduces a much higher risk of human error.

Not only that, EMC recommends against doing it anyway – for the reason above, and for the purposes of stability and performance.

11. And a bonus: Don’t have groups that start at the same time

Keep at least five minutes between start times for groups. This is 100% “best practice” and having multiple groups that start at the same time should be absolutely avoided. If you’re in a situation where you don’t have room to configure any more groups with a 5-minute gap between groups, then, well, you’ve got too many groups and you should look at consolidating them.

Share this using...

Related posts:

1. NetWorker Firewall Configuration on Windows I’ve been involved with an increasing number of NetWorker 7.6...

Related posts brought to you by Yet Another Related Posts Plugin.

When I discuss backup and recovery success metrics with customers, the question that keeps coming up is “what are desirable metrics to achieve?” I.e., if you were to broadly look at the data protection industry, what should we consider to be suitable metrics to aim for?

Bearing in mind I preach at the alter of Zero Error Policies, one might think that my aim is a 100% success rate for backups, but this isn’t quite the case. In particular, I recognise that errors will periodically occur – the purpose of a zero error policy is to eliminate repetitive errors, and ensure that no error goes unexplained. It is not however a blanket requirement that no error happens.

So what metrics do I recommend? They’re pretty simple:

• Recoveries – 100% of recoveries should succeed.
  
• Backups – 95-98% of backups should succeed.

That’s right – 100% of recoveries should succeed. Ultimately it doesn’t matter how successful (or apparently) successful your backups are, it’s the recoveries that matter. Remembering that we equate data protection to insurance policies, you can see that the goal is that 100% of “insurance claims” can be fulfilled.

Since 100% of recoveries should succeed, that metric is easy enough to understand – for every one recovery done, one recovery must succeed.

For backups though, we have to consider what constitutes a backup. In particular, if we consider this in terms of NetWorker, I’d suggest that you want to consider each saveset as a backup. As such, you want 95-98% of savesets to succeed.

This makes it relatively easy to confirm whether you’re meeting your backup targets. For instance, if you have 20 Linux hosts in your backup environment (including the backup server), and each host has 4 filesystems, then you’ll around 102 savesets on a nightly basis:

• 20 x 4 filesystems = 80 savesets
• 20 index savesets
• 1 bootstrap saveset
• 1 NMC database saveset

98% of 102 is 100 savesets (rounded), and 95% of 102 is 97 savesets, rounded. I specify a range there because on any given day it should be OK to hit the low mark, so long as a rolling average hits the high mark or, at bare minimum, sits comfortably between the low and the high mark for success rates. Of course, this is again tempered by the zero error policy guidelines; effectively, as much as possible, those errors should be unique or non-repeating.

You might wonder why I don’t call for a 100% success rate with backups – quite frankly much as it may be highly desirable, given the nature of a backup system – to touch on so many parts of an operating IT environment, it’s also one of the most vulnerable systems to unexpected events. You can design the hell out of a backup system, but you’ll still get an error if mid-way through a backup a client crashes, or a tape drive fails. So what I’m actually asserting with that 2-5% failure rate is the “nature of the beast” style failures: hardware issues, Murphy’s Law and OS/software issues.

Those are metrics you not only can depend on, but you should depend on, too.

Share this using...

Related posts:

1. No Zero Error Policy? No backup system In the past I’ve talked about the importance of having...
2. You’re not the boss of me Consider the following two questions: Do you manage your backups,...
3. Don’t eliminate the error Long-term blog readers will know that I advocate a zero...

Related posts brought to you by Yet Another Related Posts Plugin.

I thought I’d run through the release notes new features and provide a bit of a summary of the changes. The full release notes are available via PowerLink, or here.

Slap another GUI on the barbie, will ya?

As a Mac user, I find this a really nice treat. EMC have finally included a recovery GUI for the Mac OS X client. It was always a bit of an omission that the most GUI-centric platform supported by EMC happened to have no client GUI support whatsoever. For what its worth, I’m not overly concerned about a lack of a backup GUI for the platform, since that’s disappeared from other platforms, and most backups should be done via a schedule anyway.

And for what it’s worth, it’s a damn fine GUI:

There were, in beta testing, a few minor glitches: I’ve not yet had the time to run the released version through its paces yet, but as you’ll note from the screen shots above, in “Favorites”, for instance, a few drives are listed twice. Though, now that I think about it, the drives that were listed twice were filesystems that had been rebuilt and/or replaced on my Mac Pro during the beta test period, so perhaps for some reason they were showing up separately.

The other point for the Mac in 7.6 SP3 of course is support for Lion. Don’t however get your hopes up – there’s no mention of 7.6 SP3 supporting backup/recovery of the hidden recovery partition that Lion installs, and given how Apple have done their best to lock down that hidden recovery partition, I’m hardly surprised.

That being said, I’ve had the beta of 7.6 SP3 running on my Macs for some time now, and the support for Lion has been very good thus far.

Default, Default!

There’s been some changes to standard system defaults, and for the most part I really like them. This is minor stuff, to be sure, but it’ll certainly help sites where NetWorker is being deployed for the first time, or where the NetWorker administrators are relatively novice at the product.

For a start, security – at long last, RAP Logging has been enabled by default, and in fact will be enabled on an upgrade, if I’m reading the release notes correctly. This is a simple yet important change; it’s a recognition that auditing and security control is important, and it turns on something that can make back-tracking changes at a site trivial. Every time I do a backup audit on a client, I look to see whether Monitor RAP is enabled. With a bit of luck as companies move to 7.6 SP3 I’ll have less to rabbit on about there.

The devil is in no longer in the details for the start time – I always thought one of the original programmers at Legato must have had a warped sense of humour to have a default group start time of 3.33. Was it because 6.66 is not a valid time? To be sure in a very small environment, a start time of 3.33 was often OK for businesses, but the amount of data to be backed up has grown, and NetWorker is well entrenched as an enterprise product, and a default start time so close to start of business hours always irked me a little. Now it’s been set to 21.00 instead, a much more suitable time.

The nsrmmd polling, control and restart intervals have all been increased; they’re now 15, 10 and 5 minutes respectively. That’s higher than the previous defaults, but actually far more sensible. Those previous intervals were oriented at small LAN networks with a minimal number of backup devices. We’re now increasingly seeing environments with hundreds of backup devices, and the default polling intervals were something that you’d have to change as soon as you went beyond a relatively small number of devices anyway. That’s another bugbear that’s gone.

Max sessions for AFTD/file devices is now set to 32 for all new devices created after the upgrade. If I recall correctly, for most of 7.6 it’s been the case that new AFTD/file devices would get target 1/max 4 by default. I’m not sure I particularly like this change, but I’ll live with it. The challenge with having large numbers of max sessions for disk backup devices is that I’ve seen it encourage sites to not really take time to think about the disk layout. In particular, some sites end up creating a few but large disk backup devices, which results in poor striping of the backup writes and reads across the per-filesystem spindles. That is, a 6TB single disk backup unit made up of 5 x 2TB drives in RAID-6 is not going to give as good a performance as say, 2 x (5 x 1TB drives in RAID 6). I.e., I worry this may leave people thinking they can get away with a higher number of streams to each disk backup unit. Please, make sure you don’t get trapped into that ‘false economy’ thinking. On a similar front, the default max sessions for data domain devices has been set to 10. I suspect again for Boost devices that it may be a little optimistic, but we’ll wait and see.

The Company That Dare Not Speak Its Name

Its safe to say that there’s a great deal of rivalry between NetApp and EMC. If you follow EMC/NetApp employees on Twitter, sometimes the discussions descend into thinly disguised vitriol. There’s bad elements to that of course, but it does speak to the passion that each company has for their own products and their own way of thinking.

That being said, 7.6 SP3 has introduced the ability to do block-level backups of SnapMirror volumes on NetApp, which can make a big difference when you’re backing up highly dense NetApp filesystems.

Ultimately, in talk of “big data” and consolidation, all the vendors have to play together much more nicely, and I’m happy to see additional integration options between NetWorker and NetApp.

NetWorker Health Check

EMC provide a freebie utility called the Health Check Tool, and it’s worth checking out. There’s a new version, 3.1, which supports NetWorker 7.6 SP3. (Of course, you can also get a bunch of diagnostic tools in IDATA Tools, too.)

Still, if you like working within the confines of a GUI, the NetWorker Health Check tool is a handy utility to have in your environment.

I’ll multi-your-plex

This is a nice one – and I’ll be interested to play with it ASAP on a Data Domain host. So long as you’re running DDOS 5.x or higher, NetWorker will work better with deduplication while still multiplexing to a DD VTL. Previously this hasn’t been the case – the way NetWorker multiplexes the data stream somewhat incompatible with most forms of deduplication. Clearly the integration of NetWorker and Data Domain in the BRS family continues to have benefits – Data Domain OS 5.x must have some level of understanding of a multiplexed NetWorker data stream and be able to perform some deduplication on it.

Now, the release notes state clearly you won’t still get optimal deduplication rates, and it will create a performance hit during the backup process – however!, that being said, if your back is up against the wall and you have a choice between stacking on a bunch of storage node enabler codes to support 128 virtual drives on a single host, or using fewer virtual drives with a lower deduplication ratio/speed, this may be beneficial to consider.

vSphere 5 Support

It had to be said, the Oregon State Highway Division not only had a whale of a problem on its hands…

Sorry, I got distracted. It had to be said that it took NetWorker a while to support the VADP backup options under ESX 4.x, etc. While vSphere 5 has been out for a while now, the elapsed time has been much smaller, and NetWorker 7.6 SP3 features vSphere 5 virtual machine/guest support.

Alphawhat?

AlphaStor continues to limp along, and AlphStor 4.0 support is included with 7.6 SP3.

Personally, I think it’s time for AlphaStor to be dropped as independent problem. Fold library virtualisation and the additional media management options directly into the NetWorker product, and ditch AlphaStor as a separate licensed option, please, EMC.

Additional Platform Support

Solaris 11 – remember that champion Unix platform that people used to either love or hate, but now people seem to universally hate since Uncle Larry purchased it? Well, Solaris 11 has official support with NetWorker 7.6 SP3. Don’t get me wrong, I’m not knocking EMC for including support for Solaris 11, but I stand by an earlier piece, RIP Solaris.

CentOS 6.x support … ahem! … RedHat Enterprise Linux 6.x support has been enhanced to include RHEL 6.1. I’m sure this will see a large number of companies currently using CentOS to look at upgrading their release, since they get a better bang for buck quality of support for using CentOS as they do from using other commercial Linux releases.

Unky Larry’s Other Product

Java 7 is now supported with NetWorker.

Upgrading

As always, before you upgrade NetWorker:

• Read the release notes carefully. See what has been fixed from previous releases, and see what are the known issues of the new release;
• Make sure you have a good, reliable backup of your bootstrap and index region before doing the upgrade;
• Have available, ready to install, the version of NetWorker you’re currently using plus any patches you may have applied just in case you need to back-rev for some reason;
• Have a change window big enough to allow you to run some preliminary tests after the upgrade;
• Despite it not being documented anywhere, my honest opinion is that on any NetWorker server upgrade, you should remove the /nsr/tmp directory after shutting it down but before doing the upgrade. This can prevent a lot of random issues.
• Always upgrade your storage nodes at the same time as your backup server. I know EMC says you don’t have to. I disagree, however. (As do my colleagues.)

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

This is elucidated in my posts:

• What is a zero error policy?
• Zero error policy management
• Error lifecycle management
• No zero error policy? No backup system

You could say that those posts are precursors to this post, and if you’re not familiar with what I’ve had to say there, you may want to read those first.

One of the critical mistakes I periodically see when companies try to implement a zero error policy is they focus too much on the errors.

The errors though, are often just the “tip of the iceberg”.

For instance, take the most simple of errors – an open file error. You might run a backup of a Windows filesystem which reports a collection of errors relating to files that were skipped because they were open at the time.

Yet, those open files aren’t really the error. Seeing them as the error is usually a case of mistaking cause and effect. In this scenario, the error is one of:

• The backup software is misconfigured, or
• The backup software is missing modules that allow it to backup open files.

In the first case, it may be that the file(s) which are reported as open and couldn’t be backed up actually don’t need to be backed up. They may be temporary files, or cache files, or some other short-lived collection of files that have no importance in terms of data protection. So the error there isn’t the individual files that failed to backup, but the failure to configure the exclusions for the client appropriately.

In the second case, it may be that those files really do need to be backed up, but to do so requires a special module. They may be database files (e.g., Microsoft SQL Server, Microsoft Exchange, etc.), or some other collection of files that must be quiesced before backup. In this case, the error is that the system is being backed up inconsistently.

Zero error policies aren’t about playing whack-a-mole with errors; they’re about solving problems.

After all, the captain of the Titanic couldn’t have averted the disaster by stopping the ship just short of the iceberg and having someone take a pick axe to the top of it.

The net result of this is that having a zero error policy requires the following two processes/activities:

• Discussion of errors with system owners/nominated key users;
• Root cause analysis.

If either of those are missing, you’re more likely making (at best educated) guesses as to the correct resolution to the errors. However, if you have those in place, you can more confidently review any error as it hits and make an informed (and even documented) decision as to how to resolve the underlaying issue that it represents.

Without it, a zero error policy may actually make the situation worse.

Share this using...

Related posts:

1. No Zero Error Policy? No backup system In the past I’ve talked about the importance of having...

Related posts brought to you by Yet Another Related Posts Plugin.

These are:

1. Architecture – How will it fit together?
2. Rehydration – Can your pipe accommodate the data?
3. Redundancy – Are you putting all your eggs in the one basket?
4. Replicas – How will your copies be handled and recognised by the server?
5. Long term storage – What is your strategy for longer-term backups?

Each of these include factors that you have to consider before you go ahead with data deduplication within an environment, and I’ll go through each one individually.

Architecture

If we look at NetWorker and target based deduplication, we run into an interesting architectural issue. The way NetWorker generates multiplexed savesets can have a direct impact on the compressibility of the datastream. In particular, all VTL based deduplication devices should be configured such that each virtual drive has both target and max sessions set to 1.

In a conventional tape or backup-to-disk environment, it’s common to see configurations where 4 or more sessions are streamed to each device. For physical tape, this may be partly due to the need to keep drives streaming, but it can also be to do with making sure that there’s not a backlog of pending savesets, too – i.e., keeping the backup window as narrow as possible.

If we cut away from that process and move to an architecture that has a 1:1 ratio for streams and virtual drives, the logical solution is to increase the number of virtual drives. Typically I’d suggest that there’s at least a 4:1 ratio of virtual drives to physical drives when a VTL is replacing a PTL. I.e., if you had 4 physical drives, you’ll be configuring a VTL with at least 16 virtual drives.

However, if we look at NetWorker licensing, this has an odd effect. VTLs will either get ‘real’ VTL licenses if they’re of a particular EMC brand, or an alternate VTL license bundle, which grants 3 x Unlimited Autochanger licenses per XTB presented by the VTL.

Neither of those licenses are the issue – the issue is actually with NetWorker’s limitations relating to the number of devices per storage node or server. For NetWorker, Network Edition, you’re entitled to:

• 16 devices on the server;
• 16 devices on each storage node.

For NetWorker, Power Edition, you’re entitled to:

• 32 devices on the server;
• 32 devices on each storage node.

That’s all well and good for physical tape environments – but once you go virtual, those limitations can get very tight, very quickly. (Hint, EMC: Those limitations should be doubled or quadrupled, please.)

The net effect is that if you have say, a 4-drive PTL and a 16-drive VTL, but just a single server, no storage nodes, you’ll need to do one of the following:

• Upgrade from Network edition to Power Edition, or
• Purchase an additional storage node license to ‘stack on’ an extra 16 devices.

Yes – you can purchase and add-on storage node licenses to add to the permitted device count within the environment, without adding an actual storage node. This is handy to know in normal situations, but when it comes to deduplicating VTLs in particular, it’s a must.

Rehydration

It’s all very well to have a fabulous deduplication ratio. Let’s say you’re achieving 10:1 or something along those lines. However, we don’t just deal in deduplicated data. At some point, that data is going to have to be rehydrated. Typically this’ll be for one of the following:

• As part of a recovery, or
• For tape-out functionality.

In either case, you’re no longer concerned about the deduplication ratio you’ve achieved, but the amount of rehydrated data you’ll be streaming out. One immediate consideration is that if you’ve deployed deduplication backups for branch-office scenarios, and you’ve been loving the ‘trickle’ effect of only sending unique data across the WAN, you’re going to be somewhat less enamoured by having to send the entire data stream, rehydrated, back across the WAN.

Unless, of course, you’ve architected for that situation.

If you’re doing tape-out – either cloning or staging, then you need to still factor that actual rehydrated size into any sizing calculations for a physical tape library. In particular, a common mistake I’m seeing is that people think that by implementing deduplication they can substantially reduce the number of physical tape drives in the environment. I would suggest that as a general rule of thumb for most sites, a reduction of between one quarter and one third of the physical devices is the most you can hope to achieve. If you pull out more than that, you’re likely going to suffer serious contention during tape out operations. You’ll also be totally blown out of the water whenever there’s a physical fault.

Redundancy

Deduplication should never be deployed on its own. E.g., you can’t just have a single Avamar RAIN or a single target deduplication unit. It’s putting all your eggs in one basket. You need some form of atomic-unit redundancy, be that a second grid you replicate to, or a second DD you replicate to, or tape-out.

I’ve heard of solutions deployed that have a single Avamar RAIN for instance – and just a few nodes in the grid – with no tape out, and no replication to another site. I personally think that’s data-suicide. Sure, any individual node in a RAIN can fail and the grid will continue, but you’ve still got the fundamental problem – what happens if you lose your grid?

The same applies to target based deduplication. For ease of consideration, any deduplication configuration, be it Avamar, Data Domain, Quantum, FalconStor or anything else should be considered to have one unit per physical location. And if, under those definitions, you’ve only got one unit – well, you’ve got insufficient redundancy.

Replicas

In particular with target based deduplication, if you’re using the replication functionality of the deduplication device (to avoid a NetWorker clone rehydrate+deduplicate again scenario), you introduce a new challenge – how do you get NetWorker to actually know about the replicas? Items for consideration here are:

1. Can both replicas be online at the same time? I.e., does the deduplication environment support this?
2. Will NetWorker perceive the replicas as the same physical media? I.e., do the replicas have the same volume ID? If so, NetWorker won’t permit them to be mounted in two different locations at once.
3. How ‘atomically’ can replicas be brought online? If replicas do have the same volume ID, what is the smallest replica that can be brought online? Typically this will be either a single virtual tape, or a single disk backup unit. For virtual tapes, that’ll be more manageable. For disk backup units, it presents more of a problem.

Newer technology, such as DD Boost, which integrates NetWorker’s cloning facilities with the inherent replication capabilities of the hardware, address this issue. If you’re not using DD Boost though, you need to come up with your own solution.

Long Term Storage

Want deduplication? Want enough deduplication to handle 7 years of backups? 10 years? 15 years? ‘Forever’ years? Long term storage can’t be left by the way-side, you have to plan and architect this into your solution.

Some deduplication vendors (EMC included) are starting to tout new archive credentials in their deduplication arrays, but to be perfectly frank, the long-term cost of maintaining large amounts of either spinning or partially spun down disks with deduplicated storage, vs a batch of tapes with rehydrated storage, is still not at a point that can be entertained by many businesses. Tape is, and shall continue to be cheap for longer term storage and archival storage. Anyone who tries to tell you otherwise likely has a vested interest in dropping more storage on your datacentre floor.

When planning for longer-term storage in a deduplication environment, you have to make a few decisions in advance:

• Do longer term backups go direct to tape (or conventional disk staging areas) instead of ever hitting deduplicated storage?
• If the longer-term backups do sit on deduplicated storage, what will be the additional size requirements?
• Are those size requirements worth it? E.g., if you have to buy a unit that has an additional 20TB of deduplication capabilities in order to hold all the long-term backups that you want to keep ‘nearline’, is it actually worth it, given it’ll always be staged out/relocated to longer-term storage, or do you go for a cheaper initial storage option as well?

Summing up

Between this and other articles, one might think that I’m actually against deduplication. I’m not. However, I am dead-set against the mis-use of technology. Wasteful spending, particularly in the backup environment, just leads to bigger issues – such as artificial and inaccurate budgetary restraints at a later point in time.

When it comes to deduplication, I guess there can only be one rule: eyes wide open.

Share this using...

Related posts:

1. Deduplication and space management Deduplication can create fantastic space saving opportunities within an environment,...
2. 7 common problems with deduplication In an earlier article, I suggested some space management techniques...

Related posts brought to you by Yet Another Related Posts Plugin.