I’ve kept all of the questions from the previous survey, so that I can track adoption of deduplication and interest in open source modules, but I’ve now added a couple of questions about backup encryption, too.

I’d imagine the survey should take less than 5 minutes of your time, and I’d greatly appreciate you taking that time. While the survey system asks you if you want to supply an email address as you submit, you don’t have to – your responses can remain completely anonymous.

I’ll keep the survey running from June 22 2011 through to July 15 2011.

This survey is now closed.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Do you need a backup healthcheck review? This is a short process that:

• Evaluates your current configuration;
• Reviews your logs;
• Gets feedback from IT staff and management on backup policies and requirements;
• Provides a report which covers:
  • Where your environment is meeting your data protection requirements;
  • Gaps in your backup coverage;
  • Areas for process improvement.

The report can be presented in two ways:

• A regular meeting; or,
• A workshop session with appropriate staff (including IT personnel and management responsible for the backups).

Optionally, a filesystem and data profiling assessment can also be conducted, broadening the scope of the engagement.

My move to Melbourne will be starting this weekend, and I’ll be setup and back to work on Tuesday, 14 June. If you’ve been wanting to get some honest, third party feedback on your backup environment from a long-term user (15 years) and consultant (10+ years) in the product, now is your opportunity to do so.

We’ll be offering healthchecks and workshops at an introductory rate initially, so now is the time to touch base with me and start organising an independent review of your environment. Please contact me via my IDATA email address.

(Oh, and if you’re not in Melbourne, never fear! We can run remote healthcheck engagements and present via WebEx, having already done so for companies in the US and the UK.)

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

EMC has been working hard at making very useful technical documents available for NetWorker. There’s two in particular that were  released in April/May 2011 that offer brilliant information – these are:

• NetWorker Probe Overview and Troubleshooting Technical Note – At last. This is worth downloading just for one paragraph in that document, which clearly explains the exact criteria in which clients are probed, new savesets are selected for initiation, etc. You must read this document.
• Configuring Tape Devices for EMC NetWorker Technical Note – A really useful overview of getting tape devices configured in NetWorker. While normally this is straight forward, when there are problems, they can be frustrating to work through. This provides some really comprehensive information regarding the process, including some per-OS details that aren’t to be missed.

Do yourself a favour – make sure to download the above two documents, and keep an eye on the NetWorker Technical Notes – they’re proving to be of excellent value.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

One of the funniest pieces of FUD that I heard from the {Vendor} rep was that “workgroup” (ha!) products like NetWorker will waste time during the recovery process when differential backups have been run. Drawing up a simple table like the following. The argument used ran along these lines:

• Weekend – Full backup (i.e., 100%)
• Monday – Backup 5% change
• Tuesday – Backup 10% change
• Wednesday – Backup 15% change
• Thursday – Backup 20% change
• Friday – Backup 25% change

Now, as I point out in my book, while one must consider the potential that the unique changed files in set of differential backups may be 100% on each day, it’s not always going to be the case. In fact, only in fairly niche areas or situations will this be so. To be more accurate, a differential backup model may look more like:

• Weekend – Full backup (i.e., 100%)
• Monday – Backup 5% change.
• Tuesday – Backup 7% change.
• Wednesday – Backup 9% change.
• Thursday – Backup 10% change.
• Friday – Backup 11% change.

(That is – in most sites where differentials are used, the unique files that change each day will be minimal.)

Now, regardless of which model happens within an environment, the {Vendor} representative bravely then tried to assert that “with NetWorker, that means a full recovery on Friday would need to pull back 125% of the data!”

That statement is of course about as accurate as “croc shoes are cool”.

There are two types of implied FUD In this statement – and both are incorrect. They are:

• The FUD that if you backup the same file in both a full and a differential, NetWorker would recover both files, first the one from the full, then the one from the differential, in order to complete the recovery.
• The FUD that a filesystem recovery from fulls + X might pull back all files that were backed up, rather than a point in time view of the filesystem as of the last backup.

Thankfully, like Elmer, both of these FUDs are relatively easy to put to rest. I’ll do them in reverse order, since disproving the second puts us in an easy position to disprove the first FUD.

Scenario:

• Schedule called “TestDiff”: full, 5, 5, 5, 5, 5, 5
• Group called “TestDiff”: Using schedule “TestDiff”
• Client tara in group “TestDiff” has save set: /root/casestudy

Initial content of /root/casestudy:

[root@tara ~]# ls -al /root/casestudy
total 30796
drwxr-xr-x  2 root root     4096 Feb  2 03:41 .
drwxr-x--- 22 root root    20480 Feb  2 03:34 ..
-rw-r--r--  1 root root 10485760 Feb  2 03:41 full1.dat
-rw-r--r--  1 root root 10485760 Feb  2 03:41 full2.dat
-rw-r--r--  1 root root 10485760 Feb  2 03:41 full3.dat

So that’s 30MB of data. Our first backup will by necessity be a full, and we’ll follow that with an mminfo so we can see how much data has been backed up:

[root@tara ~]# savegrp -l full TestDiff
Feb  2 03:44:35 tara logger: NetWorker media: (waiting) Waiting for 1 writable volume(s) to backup pool 'TestDiff' tape(s) on tara.pmdg.lab
[root@tara ~]# mminfo -q "name=/root/casestudy"
volume        client       date      size   level  name
800844L4       tara.pmdg.lab 02/02/2011 30 MB full  /root/casestudy

Now that we’ve got that initial backup done, we’ll populate a couple more files into the directory, and do 2 level 5 differential backups:

[root@tara ~]# dd if=/dev/zero bs=1024k count=10 of=/root/casestudy/1stdiff-1.dat
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.02353 seconds, 446 MB/s

[root@tara ~]# dd if=/dev/zero bs=1024k count=10 of=/root/casestudy/1stdiff-2.dat
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.080027 seconds, 131 MB/s

[root@tara ~]# ls -al /root/casestudy
total 51308
drwxr-xr-x  2 root root     4096 Feb  2 03:45 .
drwxr-x--- 22 root root    20480 Feb  2 03:34 ..
-rw-r--r--  1 root root 10485760 Feb  2 03:45 1stdiff-1.dat
-rw-r--r--  1 root root 10485760 Feb  2 03:45 1stdiff-2.dat
-rw-r--r--  1 root root 10485760 Feb  2 03:41 full1.dat
-rw-r--r--  1 root root 10485760 Feb  2 03:41 full2.dat
-rw-r--r--  1 root root 10485760 Feb  2 03:41 full3.dat

[root@tara ~]# savegrp -l5 TestDiff
[root@tara ~]# !mminfo
mminfo -q "name=/root/casestudy"
volume        client       date      size   level  name
800844L4       tara.pmdg.lab 02/02/2011 30 MB full  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy

[root@tara ~]# savegrp -l5 TestDiff
[root@tara ~]# mminfo -q "name=/root/casestudy"
volume        client       date      size   level  name
800844L4       tara.pmdg.lab 02/02/2011 30 MB full  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy

All of that looks completely normal. So, now we’ll put a couple of more files in the directory – this time of differing sizes, and run a differential backup as well as the mminfo commands again:

[root@tara ~]# dd if=/dev/zero bs=512k count=10 of=/root/casestudy/3rddiff-1.dat
10+0 records in
10+0 records out
5242880 bytes (5.2 MB) copied, 0.022319 seconds, 235 MB/s
[root@tara ~]# dd if=/dev/zero bs=512k count=10 of=/root/casestudy/3rddiff-2.dat
10+0 records in
10+0 records out
5242880 bytes (5.2 MB) copied, 0.011584 seconds, 453 MB/s

[root@tara ~]# savegrp -l5 TestDiff
[root@tara ~]# !mminfo
mminfo -q "name=/root/casestudy"
volume        client       date      size   level  name
800844L4       tara.pmdg.lab 02/02/2011 30 MB full  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 30 MB    5  /root/casestudy

Right, next step is to delete some files – I’m going to delete the “1stdiff*” files, then run a new backup:

[root@tara ~]# rm /root/casestudy/1stdiff-*
rm: remove regular file `/root/casestudy/1stdiff-1.dat'? y
rm: remove regular file `/root/casestudy/1stdiff-2.dat'? y

[root@tara ~]# ls -l /root/casestudy
total 41032
-rw-r--r-- 1 root root  5242880 Feb  2 03:48 3rddiff-1.dat
-rw-r--r-- 1 root root  5242880 Feb  2 03:48 3rddiff-2.dat
-rw-r--r-- 1 root root 10485760 Feb  2 03:41 full1.dat
-rw-r--r-- 1 root root 10485760 Feb  2 03:41 full2.dat
-rw-r--r-- 1 root root 10485760 Feb  2 03:41 full3.dat

[root@tara ~]# savegrp -l5 TestDiff
[root@tara ~]# !mminfo
mminfo -q "name=/root/casestudy"
volume        client       date      size   level  name
800844L4       tara.pmdg.lab 02/02/2011 30 MB full  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 30 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 10 MB    5  /root/casestudy

Right, {Vendor} FUD – are you with us now? We’ll now delete all the files in the directory and do a recovery and see what we pull back. By rights, it should be only those files in the directory as of the time of backup – full1.dat, full2.dat, full3.dat, 3rddiff-1.dat and 3rddiff-2.dat:

[root@tara casestudy]# cd /root/casestudy
[root@tara casestudy]# rm *
rm: remove regular file `3rddiff-1.dat'? y
rm: remove regular file `3rddiff-2.dat'? y
rm: remove regular file `full1.dat'? y
rm: remove regular file `full2.dat'? y
rm: remove regular file `full3.dat'? y
[root@tara casestudy]# recover -s tara
Current working directory is /root/casestudy/
recover> ls
 3rddiff-1.dat   3rddiff-2.dat   full1.dat       full2.dat       full3.dat
recover> add *
5 file(s) marked for recovery
recover> volumes
Volumes needed (all on-line):
        800844L4 at /dev/nst1
recover> recover
Recovering 5 files into their original locations
Volumes needed (all on-line):
        800844L4 at /dev/nst1
Total estimated disk space needed for recover is 41 MB.
Requesting 5 file(s), this may take a while...
Requesting 1 recover session(s) from server.
./full1.dat
./full2.dat
./full3.dat
./3rddiff-1.dat
./3rddiff-2.dat
Received 5 file(s) from NSR server `tara'
Recover completion time: Wed 02 Feb 2011 03:51:43 AM EST
recover> quit
[root@tara casestudy]# ls -l
total 41032
-rw-r--r-- 1 root root  5242880 Feb  2 03:48 3rddiff-1.dat
-rw-r--r-- 1 root root  5242880 Feb  2 03:48 3rddiff-2.dat
-rw-r--r-- 1 root root 10485760 Feb  2 03:41 full1.dat
-rw-r--r-- 1 root root 10485760 Feb  2 03:41 full2.dat
-rw-r--r-- 1 root root 10485760 Feb  2 03:41 full3.dat

So, {Vendor} FUD #2 is toast. If we do multiple differential backups (or for that matter, incrementals!) with file deletes happening between backups, NetWorker just recovers the filesystem as of the last point it was backed up – it doesn’t try to repopulate files that didn’t exist as of the last backup.

Let’s return now to {Vendor} FUD #1 about differential backups in NetWorker. We’ve got a bunch of files for which we’ve done differential backups with, and so far all of those files have been backed up to a single volume – 800844L4. So, what I’m going to do is unmount that tape, mark it as full, then overwrite the file ‘full3.dat’, which will mean it’ll need a new backup:

[root@tara casestudy]# nsrjb -u 800844L4
Info: Operation `Eject' in progress on device `/dev/nst1'
Jukebox operation finished with status: succeeded
[root@tara casestudy]# nsrmm -o full 800844L4
Mark LTO Ultrium-4 tape 800844L4 as full? y
[root@tara casestudy]# dd if=/dev/zero bs=1024k count=50 of=full3.dat
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.249461 seconds, 210 MB/s
[root@tara casestudy]# !savegrp
savegrp -l5 TestDiff
Feb  2 03:56:29 tara logger: NetWorker media: (waiting) Waiting for 1 writable volume(s) to backup pool 'TestDiff' tape(s) on tara.pmdg.lab
[root@tara casestudy]# !mminfo
mminfo -q "name=/root/casestudy"
 volume        client       date      size   level  name
800843L4       tara.pmdg.lab 02/02/2011 81 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 30 MB full  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 20 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 30 MB    5  /root/casestudy
800844L4       tara.pmdg.lab 02/02/2011 10 MB    5  /root/casestudy

Our new backup is sitting on 800843L4 – a different tape. I’ll now delete and recover full3.dat, and demonstrate that NetWorker doesn’t do anything so stupid as the notion of recovering the file twice:

[root@tara casestudy]# rm full3.dat
rm: remove regular file `full3.dat'? y
[root@tara casestudy]# recover
Current working directory is /root/casestudy/
recover> add full3.dat
/root/casestudy
1 file(s) marked for recovery
recover> volumes
Volumes needed (all on-line):
        800843L4 at /dev/nst2
recover> recover
Recovering 1 file into its original location
Volumes needed (all on-line):
        800843L4 at /dev/nst2
Total estimated disk space needed for recover is 51 MB.
Requesting 1 file(s), this may take a while...
Requesting 1 recover session(s) from server.
./full3.dat
Received 1 file(s) from NSR server `tara.pmdg.lab'
Recover completion time: Wed 02 Feb 2011 03:57:57 AM EST
recover> quit

Now, just to prove that I’m not incorrectly trusting NetWorker, here’s the nsr_render_log output for the daemon.raw from the time of recovery – see if you can spot how many tapes we used:

70920 02/02/2011 03:57:42 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:root browsing
70919 02/02/2011 03:57:51 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:root done browsing
70920 02/02/2011 03:57:51 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:root browsing
70911 02/02/2011 03:57:53 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:/root/casestudy (2/02/11) starting read from 800843L4 of 51 MB
70904 02/02/2011 03:57:57 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:/root/casestudy (2/02/11) done reading 51 MB
70919 02/02/2011 03:57:57 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:root done browsing
70920 02/02/2011 03:57:57 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:root browsing
42506 02/02/2011 03:57:57 AM  2 0 0 2426823904 26733 0 tara.pmdg.lab nsrd recover info: User root on tara.pmdg.lab successfully recovered tara.pmdg.lab's files
70919 02/02/2011 03:58:00 AM  0 0 2 2426823904 26733 0 tara.pmdg.lab nsrd tara.pmdg.lab:root done browsing

Wait for it, wait for it … now let’s see, it used 800843L4. That’s one tape. That was our second backup of the file. Hmmm, but it didn’t pull back the first copy of the file, because that was on 800844L4, and the logs tell us it only read from a single tape.

{Vendor} FUD #1 put to rest too.

The real pity about vendors flinging FUD about other vendors products is that it takes away from time that could be otherwise used productively. In this case, I had been looking forward to getting at least an hour of a {Product} technical briefing. Lamentably, that’s not what I got.

Maybe next time.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Before I wrap up for the year, I wanted to spend a few minutes reintroducing some of the most popular topics of the year on the blog – the top ten articles based on directly linked accesses. Going in reverse order, they are:

• Number 10 – “Why I’d choose NetWorker over NetBackup every time“. I was basically called an idiot by someone in the storage community for writing this, but the fact remains for me that any backup product that fails to support backup dependencies is not one that I would personally choose. Given that a top search that leads people to the blog is of the kind, “netbackup vs networker” or “networker vs netbackup”, clearly people are out there comparing the two products, and I stand by my support of the primacy of backup dependency tracking.
• Number 9 – “A tale of 4 vendors“. A couple of months ago I attended SNIA’s first Australian storage blogger event, touring EMC, IBM, HDS and NetApp. Initially I’d planned to blog a fairly literal dump of the information I jotted down during the event, but I realised instead I was more drawn to the total solution stories being told by the 4 vendors.
• Number 8 – “NetWorker 7.5.2 – What’s it got?“. NetWorker 7.5 represented a big upgrade mark for a lot of sites, particularly those that wanted to jump the v7.3 and v7.4 release trees. I still get a lot of searches coming to the blog based on NetWorker 7.5 features and upgrades.
• Number 7 – “Using NetWorker Client with Opensolaris“. This was written by guest blogger Ronny Egner, and has seen more interest over the last few months as Oracle’s acquisition continues to grind down paid Sun customers. If you’re interested in writing guest blog pieces for the NetWorker Blog in 2011, let me know!
• Number 6 – “Basics – Fixing ‘NSR peer information’ errors“. I’ve said it before, and I’ll say it again: there is no valid reason why the resolution for this hasn’t been built into NMC!
• Number 5 – “NetWorker and linuxvtl, Redux“. The open source LinuxVTL project continues to grow and develop. While it’s not suited for production environments, LinuxVTL is certainly a handy VTL to plug into a NetWorker/Linux system for testing purposes. I know – I use it almost every single day.
• Number 4 and Number 3 – “NetWorker 7.6 SP1“. Interest in NetWorker 7.6 SP1 has been huge, and I had two blog postings about it – a preview posting based on publicly shared information from EMC, and the actual post-release article that covered some key features more in-depth.
• Number 2 – “Carry a Jukebox with you (if you’re using Linux)“. The first article I wrote about the LinuxVTL project.
• Number 1 – “micromanual: NetWorker Power User Guide to nsradmin“. The Power User guide to nsradmin has been downloaded well over a thousand times. I’ve been a fan of nsradmin ever since I started using NetWorker and had to administer a few NetWorker servers over extremely slow links (think dial-up speeds). It’s been very gratifying to be able to introduce so many people to such a useful and powerful tool.

Personally this year has been a pretty big one for me. Probably the biggest single event was that my partner and I made the decision to move from central coast NSW to Melbourne, Victoria during the year. We haven’t moved yet; it’s due for June 2011, but it’s going to necessitate a lot of action and work on our part to get there. It’ll be well worth the effort though, and I’ve already reached that odd point where I no longer think of the place I’m living as “home”. The reasons that led us to that decision are covered on my personal blog here. Continuing the personal front, I was extremely pleased to be able to say goodbye to the mobile “netwont” that is Vodafone in Australia. I’ve been using my personal blog to talk about a lot of varied topics running from internet censorship to invasive information requests to more mundane things, such as what makes a good consultant.

Technically I think the coming few years are going to be fascinating. Deduplication has only just started to make a splash; I think it’ll be a while before it becomes as pervasive as say, plain old disk backup, but it will have a continued and growing effect in the enterprise backup market. I predict that another bevy of dopey analysts will insist that tape is dead, just like they have every year for the last 2 decades, and at the end of the year I predict the majority of companies they interface with will still be using tape in some form or another. However, the use of tape will continue to evolve in the marketplace; as nearline disk storage becomes more regular and cheaper for backup solutions, we’ll see tape continue to be pushed out to longer term retention systems and safety nets – i.e., tape is certainly sliding away from being the primary source for recoveries in an enterprise backup environment.

One last thing – I want to thank the readers of this blog. To those people who subscribe to the mailing list, and those who subscribe to the RSS feed, to those who have the site bookmarked and to those who just randomly stumble across the site – I hope in each case you’re finding something useful, and I’m grateful for your readership.

Happy holidays to those of you celebrating or relaxing over the coming weeks, and peaceful times to those working through.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

It’s time to revisit that survey; I’ve adjusted the questions a little based on some of the responses from the previous survey results, and I’m keen for as many answers as possible to make this a worthwhile report. (Don’t forget, the report will be free to download.)

I’ll be aiming to keep the survey running until the end of November, and publish the results early in December. So please, fill out the survey below and have your say!

~

The survey has now closed. The report will be posted soon.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Titled “Configuring LinuxVTL on CentOS for NetWorker”, this micromanual focuses on providing a start-to-finish description of the process required to install and configure the LinuxVTL software and then subsequently get it configured with NetWorker – in this case, NetWorker 7.6 SP1.

If you’re interested in using the LinuxVTL for lab/testing purposes, but needed a more comprehensive guide on getting it set up and running, this micromanual should get you over the line.

To download, go to the micromanuals page and fill out the registration form.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

I was lucky enough to get to work on the beta testing programme for NetWorker 7.6 SP1. While there are a bunch of new features in NW 7.6 SP1 (with the one most discussed by EMC being the Data Domain integration), I want to talk about three new features that I think are quite important, long term, in core functionality within the product for the average Joe.

These are:

• Scheduled Cloning
• AFTD enhancements
• Checkpoint restarts

Each of these on their own represents significant benefit to the average NetWorker site, and I’d like to spend some time discussing the functionality they bring.

[Edit/Aside: You can find the documentation for NetWorker 7.6 SP1 available in the updated Documentation area on nsrd.info]

Scheduled Cloning

In some senses, cloning has been the bane of the NetWorker administrator’s life. Up until NW 7.6 SP1, NetWorker has had two forms of cloning:

• Per-group cloning, immediately following completion of backups;
• Scripted cloning.

A lot of sites use scripted cloning, simply due to the device/media contention frequently caused in per-group cloning. I know this well; since starting working with NetWorker in 1996, I’ve written countless numbers of NetWorker cloning scripts, and currently am the primary programmer for IDATA Tools, which includes what I can only describe as a cloning utility on steroids (‘sslocate’).

Will those scripts and utilities go away with scheduled cloning? Well, I don’t think they’re always going to go away – but I do think that they’ll be treated more as utilities rather than core code for the average site, since scheduled cloning will be able to achieve much of the cloning requirements for companies.

I had heard that scheduled cloning was on the way long before the 7.6 SP1 beta, thanks mainly to one day getting a cryptic email along the lines of “if we were to do scheduled cloning, what would you like to see in it…” – so it was pleasing, when it arrived, to see that much of my wish list had made it in there. As a first-round implementation of the process, it’s fabulous.

So, let’s look at how we configure scheduled clones. First off, in NMC, you’ll notice a new menu item in the configuration section:

This will undoubtedly bring joy to the heart of many a NetWorker administrator. If we then choose to create a new scheduled clone resource, we can create a highly refined schedule:

Let’s look at those options first before moving onto the second tab:

• Name and comment is pretty self explanatory – nothing to see there.
• Browse and retention – define, for the clone schedule, both the browse and retention time of the savesets that will be cloned.
• Start time – Specify exactly what time the cloning is to start.
• Schedule period – Weekly allows you to specify which days of the week the cloning is to run. Monthly allows you to specify which dates of the month the cloning will run.
• Storage node – Allows you to specify to which storage node the clone will write to. Great for situations where you have say, an off-site storage node and you want the data streamed directly across to it.
• Clone pool – Which pool you want to write the clones to – fairly expected.
• Continue on save set error – This is a big help. Standard scripting of cloning will fail if one of the savesets selected to clone has an error (regardless of whether that’s a read error, or it disappears (e.g., is staged off) before it gets cloned, etc.) and you haven’t used the ‘-F’ option. Click this check box and the cloning will at least continue and finish all savesets it can hit in one session.
• Limit number of save set clones – By default this is 1, meaning NetWorker won’t create more than one copy of the saveset in the target pool. This can be increased to a higher number if you want multiple clones, or it can be set to zero (for unlimited), which I wouldn’t see many sites having a need for.

Once you’ve got the basics of how often and when the scheduled clone runs, etc., you can move on to selecting what you want cloned:

I’ve only just refreshed my lab server, so you can see that a bit of imagination is required in the above screen shot to flesh out what this may look in a normal site. But, you can choose savesets to clone based on:

• Group
• Client
• Source Pool
• Level
• Name

or

• Specific saveset ID/clone IDs

When specifying savesets based on group/client/level/etc., you can also specify how far back NetWorker is to look for savesets to clone. This avoids a situation whereby you might say, enable scheduled cloning and suddenly have media from 3 years ago requested.

You might wonder about the practicality of being able to schedule a clone for specific SSID/CloneID combos. I can imagine this would be particularly useful if you need to do ad-hoc cloning of a particularly large saveset. E.g., if you’ve got a saveset that’s say, 10TB, you might want to configure a schedule that would start specifically cloning this at 1am Saturday morning, with your intent being to then delete the scheduled clone after it’s finished. In other words, it’s to replace having to do a scheduled cron or at job just for a single clone.

Once configured, and enabled, scheduled cloning runs like a dream. In fact, it was one of the first things I tackled while doing beta testing, and almost every subsequent day found myself thinking at 3pm “why is my lab server suddenly cloning? – ah yes, that’s why…”

AFTD Enhancements

There’s not a huge amount to cover in terms of AFTD enhancements – they’re effectively exactly the same enhancements that have been run into NetWorker 7.5 SP3, which I’ve previously covered here. So, that means there’s a better volume selection criteria for AFTD backups, but we don’t yet have backups being able to continue from one AFTD device to another. (That’s still in the pipeline and being actively worked on, so it will come.)

Even this one key change – the way in which volumes are picked in AFTDs for new backups – will be a big boon for a lot of NetWorker sites. It will allow administrators to not focus so much on the “AFTD data shuffle”, as I like to consider it, and instead focus on higher level administration of the backup environment.

(These changes are effectively “under the hood”, so there’s not much I can show in the way of screen-shots.)

Checkpoint Restarting

When I first learned NetBackup, I immediately saw the usefulness of checkpoint restarting, and have been eager to see it appear in NetWorker since that point. I’m glad to say it’s appeared in (what I consider to be) a much more usable form. So what is checkpoint restarting? If you’re not familiar with the term, it’s where the backup product has regular points at which it can restart from, rather than having to restart an entire backup. Previously NetWorker has only done this at the saveset level, but that’s not really what the average administrator would think of when ‘checkpoints’ are discussed. NetBackup, last I looked at it, does this at periodic intervals – e.g., every 15 minutes or so.

Like almost everything in NetWorker, we get more than one way to run a checkpoint:

Within any single client instance you can choose to enable checkpoint restarting, with the restart options being:

• Directory – If a backup failure occurs, restart from the last directory that NetWorker had started processing.
• File – If a backup failure occurs, restart from the last file NetWorker had started processing.

Now, the documentation warns that with checkpoint enabled, you’ll get a slight performance hit on the backup process. However, that performance hit is nothing compared to the performance and potentially media hit you’d take if you’re 9.8TB through a 10TB saveset and the client is accidentally rebooted!

Furthermore, in my testing (which admittedly focused on savesets smaller than 10GB), I inevitably found that with either file or directory level checkpointing enabled, the backup actually ran faster than the normal backup. So maybe it’s also based on the hardware you’re running on, or maybe that performance hit doesn’t come in until you’re backing up millions of files, but either way, I’m not prepared to say it’s going to be a huge performance hit for anyone yet.

Note what I said earlier though – this can be configured on a single client instance. This lets you configure checkpoint restarting even on the individual client level to suit the data structure. For example, let’s consider a fileserver that offers both departmental/home directories, and research areas:

• The departmental/home directories will have thousands and thousands of directories – have a client instance for this area, set for directory level checkpointing.
• The research area might feature files that are hundreds of gigabytes each – use file level checkpointing here.

When I’d previously done a blog entry wishing for checkpoint restarts (“Sub saveset checkpointing would be good“), I’d envisaged the checkpointing being done via the continuation savesets – e.g., “C:\”, “<1>C:\”, “<2>C:\”, etc. It hasn’t been implemented this way; instead, each time the saveset is restarted, a new saveset is generated of the same level, catering to whatever got backed up during that saveset. On reflection, I’m not the slightest bit hung up over how it’s been implemented, I’m just overjoyed to see that it has been implemented.

Now you’re probably wondering – does the checkpointing work? Does it create any headaches for recoveries? Yes, and no. As in, yes it works, and no, throughout all my testing, I wasn’t able to create any headaches in the recovery process. I would feel very safe about having checkpoint restarts enabled on production filesystems right now.

Bonus: Mac OS X 10.6 (“Snow Leopard”) Support

For some time, NetWorker has had some issues supporting Mac OS X 10.6, and it’s certainly caused some problems for various customers as this operating system continues to get market share in the enterprise. I was particularly pleased during a beta refresh to see an updated Mac OS X client for NetWorker. This works excellently for backup, recovery, installation, uninstallation, etc. I’d suggest on the basis of the testing I did that any site with OS X should immediately upgrade those clients at least to 7.6 SP1.

In Summary

The only major glaring question for me, looking at NetWorker 7.6 SP1 is the obvious: this has so many updates, and so many new features, way more than we’d see in a normal service pack – why the heck isn’t it NetWorker v8?

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

As a command line tool, ‘check-clients’ can power through a suite of different tests and data gathering activities against your clients, all with very minimum effort on your part. Let’s look at the tests that are currently available:

[root@nox bin]# check-clients -l
Test Name           Test Description
------------------- ------------------------------------------------------
client_ids          Returns client ID for each configured client
empty               Report clients with empty indices
index               Perform nsrck -L3 on each client
index_rebuild       Perform nsrck -L6 on each client
info                Retrieve client information
list_active         List all configured clients in active groups
list_all            List all clients currently configured
performance         Check backup performance via bigasm
ping                Ping each client
probe               Savgroup probe for each client
resolution          Test/confirm name resolution
rpcinfo             Test rpcinfo/portmapper access
used_space          Calculates used space for backups

Now technically, not all of the above are actually tests as such – for instance, the used_space option was one recently requested by a customer to report on all backups currently held by a backup server for a client. Running it on one of my lab machines, the output looks like the following:

[root@nox bin]# check-clients -g all_active -t used_space
============================================================
Running test: used_space (Calculates used space for backups)
============================================================
        Client                         Used Space (GB)
        ----------------------------   --------------------
        archon                                    362.60783
        faero                                       0.00000
        luyten                                      0.00000
        nox                                       544.40887
        ----------------------------   --------------------
                 Total for 4 clients              907.01669
        ----------------------------   --------------------

To me, that’s a combo test/information gathering option; specifically the customer was after this particular test so that they could spot any newly added clients that hadn’t been backing up (i.e., by having a “Used Space” of 0 GB).

Equally, there’s use in periodically running the “client_ids” test – running and keeping the output of this test will give you help in any sticky situation where you suddenly need access to a previous clients’ host ID:

[root@nox bin]# check-clients -a -t client_ids
=======================================================================
Running test: client_ids (Returns client ID for each configured client)
=======================================================================

        aralathan = 65100d33-00000004-464fcacc-464fcacb-00050000-c0a86404
        archon = 3f33ca7b-00000004-43a4837c-43a484d7-00030000-c0a80006
        asgard = 00b151ed-00000004-43a4837b-43a4837a-00010000-c0a80006
        djwmp = 5560bbf6-00000004-4910cd4b-4910cd4a-01961a00-3d2a4f4b
        faero = 76c06b0a-00000004-453e8e44-453e8e43-00310000-c0a86406
        loki = d3f277da-00000004-4857452f-4857452e-00020000-c0a86404
        luyten = 93166424-00000004-4a2f8cde-4a2f8cdd-01041a00-3d2a4f4b
        nimrod = d6454919-00000004-496aaadc-496aaadb-006f1a00-3d2a4f4b
        nox = 85acae6f-00000004-464fbdd1-464fbdd0-00010000-c0a86404
        valhalla = 61d3ca1e-00000004-495525db-4955299a-00051500-98e71c17

Moving on into actual test territory, multiple tests can be teamed up to do a chunk of information gathering in one command. For instance, combining a ping test and a name resolution test against all active clients is as simple as:

[root@nox bin]# check-clients -g all_active -t ping,resolution
=====================================
Running test: ping (Ping each client)
=====================================
	archon  (0 responses, expected 4)
	faero  (0 responses, expected 4)
	luyten  (4 responses)
	nox.pmdg.lab  (4 responses)

=======================================================
Running test: resolution (Test/confirm name resolution)
=======================================================

	archon
		Name: archon (archon.pmdg.lab) (192.168.100.1) 
		Name: archon.pmdg.lab (archon.pmdg.lab) (192.168.100.1) 
		Addr: 192.168.100.1 (archon.pmdg.lab) 

	faero
		Name: faero (faero.pmdg.lab) (192.168.100.10) 
		Name: faero.pmdg.lab (faero.pmdg.lab) (192.168.100.10) 
		Addr: 192.168.100.10 (faero.pmdg.lab) 

	luyten
		Name: luyten (luyten.pmdg.lab) (192.168.100.18) 
		Name: luyten.pmdg.lab (luyten.pmdg.lab) (192.168.100.18) 
		Addr: 192.168.100.18 (luyten.pmdg.lab) 

	nox.pmdg.lab
		Name: nox.pmdg.lab (nox.pmdg.lab) (192.168.100.4) 
		Name: nox (nox.pmdg.lab) (192.168.100.4) 
		Addr: 192.168.100.4 (balrog.pmdg.lab (unknown))

None of this is re-inventing the wheel of course, but being able to just run a single command that cycles through and tests every active client (or even all clients) is particularly useful.

Even performance testing is catered for with check-clients; reaching out to the clients, the utility can run bigasm tests automatically – a great way for easily testing where performance hits are happening on the network. For example, a quick/basic demo of this option is below:

[root@nox bin]# check-clients -c luyten,nox.anywebdb.com -b Staging -S 50 -t performance
===============================================================
Running test: performance (Check backup performance via bigasm)
===============================================================
        luyten (Solaris/UNIX style test)
                Backup 50 MB to Staging
                50 MB took 12 seconds (4.17 MB/s)
        nox.pmdg.lab (Linux/UNIX style test)
                Backup 50 MB to Staging
                50 MB took 3 seconds (16.67 MB/s)

If you are looking around for a test kit option for NetWorker – and want access to a heap of other goodies at the same time – then ‘check-clients’ out of the IDATA Tools suite may very well be what you need.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Up until a short while ago, I always had to give an answer that differentiated between can and supported. Thankfully a while ago, EMC started supporting NetWorker clients running on CentOS, but servers and storage nodes were a bit of a hold out.

This morning I did a bit of a happy-geek dance (not really, but that description certainly fits my mood) when I read the latest software compatibility guide and saw:

For people who need to run Linux, but dislike the licensing costs of the commercial distributions (or like me, question the quality of support you get from such distributions), this will be a big help.

I’d like to think this is part of a growing trend amongst enterprise vendors – CentOS is in every way an enterprise distribution (and especially appreciated by many administrators thanks to the Yum package management system).

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.