Jul 122012
 

As mentioned in my main NetWorker 8 introduction article, one of the biggest architectural enhancements in NetWorker v8 is a complete overhaul of the backup to disk architecture. This isn’t an update to AFTDs, it’s a complete restart – the architecture that was is no longer, and it’s been replaced by something newer, fresher, and more scalable.

In order to understand what’s involved in the changes, we first need to step back and consider how the AFTD architecture works in NetWorker v7.x. For the purposes of my examples, I’m going to consider a theoretical 10TB of disk backup capacity available to a NetWorker server. Now, under v7.x, you’d typically end up with filesystems and AFTDs that look like the following:

AFTD changes - v7.x AFTD

(In the above diagram, and those to follow, a red line/arrow indicates a write session coming into the AFTD, and a green line/arrow indicates a read session coming out of the AFTD.)

That is, you’d slice and dice that theoretical 10TB of disk capacity into a bunch of smaller sized filesystems, with typically one AFTD per filesystem. In 7.x, there are two nsrmmd processes per AFTD – one to handle write operations (the main/real path to the AFTD), and one to handle read operations from the AFTD device – the shadow or _AF_readonly path on the volume.

So AFTDs under this scenario delivered simultaneous backup and recovery operations by a bit of sleight-of-hand; in some ways, NetWorker was tricked into thinking it was dealing with two different volumes. In fact, that’s what lead to there being two instances of a saveset in the media database for any saveset created on AFTD – one for the read/write volume, and one of the read-only volume, with a clone ID of 1 less than the instance on the read/write volume. This didn’t double the storage; the “trick” was largely maintained in the media database, with just a few meta files maintained in the _AF_readonly path of the device.

Despite the improved backup options offered by v7.x AFTDs, there were several challenges introduced that somewhat limited the applicability of AFTDs in larger backup scenarios. Much as I’ve never particularly liked virtual tape libraries (seeing them as a solution to a problem that shouldn’t exist), I found myself typically recommending a VTL for disk backup in NetWorker ahead of AFTDs. The challenges in particular, as I saw them, were:

  • Limits on concurrency between staging, cloning and recovery from AFTDs meant that businesses often struggled to clone and reclaim space non-disruptively. Despite the inherent risks, this lead to many decisions not to clone data first, meaning only one copy was ever kept;
  • Because of those limits, disk backup units would need to be sliced into smaller allotments – such as the 5 x 2TB devices cited in the above diagram, so that space reclamation would be for smaller, more discrete chunks of data, but spread across more devices simultaneously;
  • A saveset could never exceed the amount of free space on an AFTD – NetWorker doesn’t support continuing a saveset from one full AFTD to another;
  • Larger savesets would be manually sliced up by the administrator to fit on an AFTD, introducing human error, or would be sent direct to tape, potentially introducing shoe-shining back into the backup configuration.

As a result of this, a lot of AFTD layouts saw them more being used as glorified staging units, rather than providing a significant amount of near-line recoverability options.

Another, more subtle problem from this architecture was that nsrmmd itself is not a process geared towards a high amount of concurrency; while based on its name (media multiplexor daemon) we know that it’s always been designed to deal with a certain number of concurrent streams for tape based multiplexing, there are limits to how many savesets an nsrmmd process can handle simultaneously before it starts to dip in efficiency. This was never so much an issue with physical tape – as most administrators would agree, using multiplexing above 4 for any physical tape will continue to work, but may result in backups which are much slower to recover from if a full filesystem/saveset recovery is required, rather than a small random chunk of data.

NetWorker administrators who have been using AFTD for a while will equally agree that pointing a large number of savesets at an AFTD doesn’t guarantee high performance – while disk doesn’t shoe-shine, the drop-off in nsrmmd efficiency per saveset would start to be quite noticeable at a saveset concurrency of around 8 if client and network performance were not the bottleneck in the environment.

This further encouraged slicing and dicing disk backup capacity – why allocate 10TB of capacity to a single AFTD if you’d get better concurrency out of 5 x 2TB AFTDs? To minimise the risk of any individual AFTD filling while others still had capacity, you’d configure the AFTDs to each have target sessions of 1 – effectively round-robbining the  starting of savesets across all the units.

I think that pretty much provides a good enough overview about AFTDs under v7.x that can talk about AFTDs in NetWorker v8.

Keeping that 10TB of disk backup capacity in play, under NetWorker v8, you’d optimally end up with the following configuration:

AFTD changes - v8.x AFTD

You’re seeing that right – multiple nsrmmd processes for a single AFTD – and I don’t mean a read/write nsrmmd and a shadow-volume read-only nsrmmd as per v7.x. In fact, the entire concept and implementation of the shadow volume goes away under NetWorker v8. It’s not needed any longer. Huzzah!

Assuming dynamic nsrmmd spawning (yes), and up to certain limits, NetWorker will now spawn one nsrmmd process for an AFTD each time it hits the target sessions setting for the AFTD volume. That raises one immediate change – for a v8.x AFTD configuration, bump up the target sessions for the devices. Otherwise you’ll end up spawning a lot more nsrmmd processes than are appropriate. Based on feedback from EMC, it would seem that the optimum target setting for a consolidate AFTD is 4. Assuming linear growth, this would mean that you’d have the following spawning rate:

  • 1 saveset, 1 x nsrmmd
  • 2 savesets, 1 x nsrmmd
  • 3 savesets, 1 x nsrmmd
  • 4 savesets, 1 x nsrmmd
  • 5 savesets, 2 x nsrmmd
  • 6 savesets, 2 x nsrmmd
  • 7 savesets, 2 x nsrmmd
  • 8 savesets, 2 x nsrmmd
  • 9 savesets, 3 x nsrmmd

Your actual rate may vary of course, depending on cloning, staging and recovery operations also being performed at the same time. Indeed, for the time being at least, NetWorker dedicates a single nsrmmd process to any nsrclone or nsrstage operation that is run (either manually or as a scheduled task), and yes, you can actually simultaneously recover, stage and clone all at the same time. NetWorker handles staging in that equation by blocking capacity reclamation while a process is reading from the AFTD – this prevents a staging operation removing a saveset that is needed for recovery or cloning. In this situation, a staging operation will report as per:

[root@tara usr]# nsrstage -b Default -v -m -S 4244365627
Obtaining media database information on server tara.pmdg.lab
80470:nsrstage: Following volumes are needed for cloning
80471:nsrstage:         AFTD-B-1 (Regular)
5874:nsrstage: Automatically copying save sets(s) to other volume(s)
79633:nsrstage: 
Starting migration operation for Regular save sets...
6217:nsrstage: ...from storage node: tara.pmdg.lab
81542:nsrstage: Successfully cloned all requested Regular save sets (with new cloneid)
        4244365627/1341909730
79629:nsrstage: Clones were written to the following volume(s) for Regular save sets:
        800941L4
6359:nsrstage: Deleting the successfully cloned save set 4244365627
Recovering space from volume 15867081 failed with the error 'volume 
mounted on ADV_FILE disk AFTD1 is reading'.
Refer to the NetWorker log for details.
89216:nsrd: volume mounted on ADV_FILE disk AFTD1 is reading

When a space reclamation is subsequently run (either as part of overnight reclamation, or an explicit execution of nsrim -X), the something along the following lines will get logged:

nsrd NSR info Index Notice: nsrim has finished crosschecking the media db 
nsrd NSR info Media Info: No space was recovered from device AFTD2 since there was no saveset eligible for deletion. 
nsrd NSR info Media Info: No space was recovered from device AFTD4 since there was no saveset eligible for deletion. 
nsrd NSR info Media Info: Deleted 87 MB from save set 4244365627 on volume AFTD-B-1 
nsrd NSR info Media Info: Recovered 87 MB by deleting 1 savesets from device AFTD1. 
nsrsnmd NSR warning volume (AFTD-B-1) size is now set to 2868 MB

Thus, you shouldn’t need to worry about concurrency any longer.

Another change introduced to the AFTD architecture is the device name is now divorced from the path. This is stored in a different field. For example:

AFTD path/device name separation

In the above example, the AFTD device has a device name of AFTD1, and the path to it is /d/backup1 – or to be more accurate, the first path to it is /d/backup1. I’ll get to the real import of that in a moment.

There is a simpler management benefit: previously if you setup an AFTD and needed to later change the path that it was mounted from, you had to do the following:

  1. Unmount the disk backup unit within NetWorker
  2. Delete the AFTD definition within NetWorker
  3. Create a new AFTD definition pointing to the new location
  4. At the OS, remount the AFTD filesystem at the new location
  5. Mount the AFTD volume from its new location in NetWorker

This was a tedious process, and the notion of “deleting” an AFTD, even though it had no bearing on the actual data stored on it, did not appeal to a lot of administrators.

However, the first path specified to an AFTD in the “Device Access Information” field refers to the mount point on the owner storage node, so under v8, all you’ve got to do in order to relocate the AFTD is:

  1. Unmount the disk backup unit within NetWorker.
  2. Remount the AFTD filesystem in its new location.
  3. Adjust the first entry in the “Device Access Information” field.
  4. Remount the AFTD volume in its new location.

This may not seem like a big change, but it’s both useful and more logical. Obviously another benefit of this is that you no longer have to remember device paths when performing manual nsrmm operations against AFTDs – you just specify the volume name. So your nsrmm commands would go from:

# nsrmm -u -f /d/backup1

to, in the above example:

# nsrmm -u -f AFTD1

The real benefit of this though is what I’ve been alluding to by talking about the first mount point specified. You can specify alternate mount points for the device. However, these aren’t additional volumes on the server – they’re mount points as seen by clients. This allows a bypass of using nsrmmd to perform a write to a disk backup unit from the client, and instead sees the client write via whatever operating system mount mechanism it’s used (CIFS or NFS).

In this configuration, your disk backup environment can start to look like the following:

AFTD changes - v8.x AFTD with NFS

You may be wondering what advantage this offers, given the backup is still written across the network – well, past an initial negotiation with nsrmmd on the storage node/server for the file to write to, the client then handles the rest of the write itself, not bothering nsrmmd again. In other words – so long as the performance of the underlying filesystem can scale, the number of streams you can write to an AFTD can expand beyond the maximum number of nsrmmd processes an AFTD or storage node can handle.

At this point, the “Device Access Information” is used as a multi-line field, with each subsequent line representing an alternate path the AFTD is visible at:

AFTD multimount

So, the backup process will work such that if the pool allows the specified AFTD to be used for backup, and the AFTD volume is visible to a client with “Client direct” setting enabled (a new option in the Client resource), on one of the paths specified, then the client will negotiate access to the device through a device-owner nsrmmd process, and go on to write the backup itself.

Note that this isn’t designed for sharing AFTDs between storage nodes – just between a server/storage node and clients.

Also, in case you’re skeptical, if your OS supports gathering statistics from the network-mount mechanism in use, you can fairly readily see whether NetWorker is honouring the direct access option. For instance, in the above configuration, I had a client called ‘nimrod’ mounting the disk backup unit via NFS from the backup server; before the backup started on a new, fresh mount, nfsstat showed no activity. After the backup, nfsstat yielded the following:

[root@nimrod ~]# nfsstat
Client rpc stats:
calls      retrans    authrefrsh
834        0          0
Client nfs v3:
null         getattr       setattr      lookup       access       readlink 
0         0% 7          0% 3         0% 4         0% 9         1% 0         0% 
read         write         create       mkdir        symlink      mknod 
0         0% 790       94% 1         0% 0         0% 0         0% 0         0% 
remove       rmdir         rename       link         readdir      readdirplus 
1         0% 0          0% 0         0% 0         0% 0         0% 0         0% 
fsstat       fsinfo        pathconf     commit 
10        1% 2          0% 0         0% 6         0%

NetWorker also reports in the savegroup completion report that a client successfully negotiated direct file access (DFA) for a backup, too:

* nimrod:All savefs nimrod: succeeded.
* nimrod:/boot 86705:save: Successfully established DFA session with adv_file device for save-set ID '1694225141' (nimrod:/boot).
V nimrod: /boot                    level=5,          0 KB 00:00:01      0 files
* nimrod:/boot completed savetime=1341899894
* nimrod:/ 86705:save: Successfully established DFA session with adv_file device for save-set ID '1677447928' (nimrod:/).
V nimrod: /                        level=5,         25 MB 00:01:16    327 files
* nimrod:/ completed savetime=1341899897
* nimrod:index 86705:save: Successfully established DFA session with adv_file device for save-set ID '1660670793' (nox:index:65768fd6-00000004-4fdfcfef-4fdfd013-00041a00-3d2a4f4b).
 nox: index:nimrod                 level=5,        397 KB 00:00:00     25 files
* nimrod:index completed savetime=1341903689

Finally, logging is done in the server daemon.raw to also indicate that a DFA save has been negotiated:

91787 07/10/2012 05:00:05 PM  1 5 0 1626601168 4832 0 nox nsrmmd NSR notice Save-set ID '1694225141' 
(nimrod:/boot) is using direct file save with adv_file device 'AFTD1'.

The net result of all these architectural changes to AFTDs is a significant improvement over v7.x AFTD handling, performance and efficiency.

(As noted previously though, this doesn’t apply just to AFTDs, for what it’s worth – that client direct functionality also applies to Data Domain Boost devices, which allows a Data Domain system to integrate even more closely into a NetWorker environment. Scaling, scaling, scaling: it’s all about scaling.)

In order to get the full benefit, I believe sites currently using AFTDs will probably go through the most pain; those who have been using VTLs may have a cost involved in the transition, but they’ll be able to transition to an optimal architecture almost immediately. However, sites with multiple smaller AFTDs won’t see the full benefits of the new architecture until they redesign their backup to disk environment, increasing the size of backup volumes. That being said, the change pain will be worth it for the enhancements received.

Basics – Cloning vs Staging

 Basics, NetWorker  Comments Off on Basics – Cloning vs Staging
Mar 212011
 

When people are just starting to get into NetWorker, a common situation is that they get confused about the difference between cloning and staging. (This isn’t helped given NetWorker can report in-progress staging operations as cloning – a perennial source of annoyment.)

So, what’s the difference?

  • A clone operation is where NetWorker duplicates a saveset. It makes a registered copy of the saveset, and at the conclusion of the operation is aware that it has an additional copy.
  • A stage operation is where NetWorker moves a saveset. It first makes a registered copy of the saveset, and then at the conclusion of the operation removes reference to the instance that it copied from.

Typically when we talk about staging, we talk about moving from a disk (media type ‘FILE’ or media type ‘ADV_FILE’) volume to tape. In such a situation where NetWorker stages from an actual FILE/ADV_FILE volume, it not only removes reference to the original saveset, but it actually removes it from the source volume as well. That’s to be expected – it’s a real disk filesystem that NetWorker is accessing, and removing a saveset is as simple as just running an operating system ‘delete’ command.

While it’s not often done, NetWorker does support staging from tape – but obviously when it’s done reading from the source volume, it can’t then selectively erase chunks of savesets from the source tape. Instead, all it does in that situation is delete, from the media database, the reference to the saveset having been on the source volume.

(In case you’re new to NetWorker and intend to now run off and try some staging – make sure, please, before you do, to read the second article ever posted on the NetWorker Blog – “Instantiating Savesets“. It has some very cautionary information about staging operations.)

One final thing – while I said that a clone or stage operation is done against a saveset, it’s not always as simple as that. If you tell NetWorker to clone or stage an individual saveset/saveset instance, that’s exactly what it will do. However, if you tell NetWorker to clone or stage a NetWorker tape volume, it will clone/stage the entire volume, with multiplexing left intact.

Regardless of those caveats though, remember the simple rule – a clone is a copy operation, and a stage is a move operation.

Mar 032010
 

While I touched on this in the second blog posting I made (Instantiating Savesets), it’s worthwhile revisiting this topic more directly.

Using ADV_FILE devices can play havoc with conventional tape rotation strategies; if you aren’t aware of these implications, it could cause operational challenges when it comes time to do recovery from tape. Let’s look at the lifecycle of a saveset in a disk backup environment where a conventional setup is used. It typically runs like this:

  1. Backup to disk
  2. Clone to tape
  3. (Later) Stage to tape
  4. (At rest) 2 copies on tape

    Looking at each stage of this, we have:

    Saveset on ADV_FILE deviceThe saveset, once written to an ADV_FILE volume, has two instances. The instance recorded as being on the read-read only part of the volume will have an SSID/CloneID of X/Y. The instance recorded as being on the read-write part of the volume will have an SSID/CloneID of X/Y+1. This higher CloneID is what causes NetWorker, upon a recovery request, to seek the “instance” on the read-only volume. Of course, there’s only one actual instance (hence why I object so strongly to the ‘validcopies’ field introduced in 7.6 reporting 2) – the two instances reported are “smoke and mirrors” to allow simultaneous backup to and recovery from an ADV_FILE volume.

    The next stage sees the saveset cloned:

    ADV_FILE + Tape CloneThis leaves us with 3 ‘instances’ – 2 physical, one virtual. Our SSID/CloneIDs are:

    • ADV_FILE read-only: X/Y
    • ADV_FILE read-write: X/Y+1
    • Tape: X/Y+n, where n > 1.

    At this point, any recovery request will still call for the instance on the read-only part of the ADV_FILE volume, so as to help ensure the fastest recovery initiation.

    At some future point, as disk capacity starts to run out on the ADV_FILE device, the saveset will typically be staged out:

    ADV_FILE staging to tapeAt the conclusion of the staging operation, the physical + virtual instances of the saveset on the ADV_FILE device are removed, leaving us with:

    Savesets on tape only

    So, at this point, we end up with:

    • A saveset instance on a clone volume with SSID/CloneID of: X/Y+n.
    • A saveset instance on (typically) a non-clone volume with SSID/CloneID of: X/Y+n+m, where m > 0.

    So, where does this leave us? (Or if you’re not sure where I’ve been heading yet, you may be wondering what point I’m actually trying to make.)

    Note what I’ve been saying each time – NetWorker, when it needs to read from a saveset for recovery purposes, will want to pick the saveset instance with the lowest CloneID. At the point where we’ve got a clone copy and a staged copy, both on tape, the clone copy will have the lowest CloneID.

    The net result is that NetWorker will, in these circumstances, when both tapes aren’t online, request the clone volume for recovery – even though in an extreme number of cases, this will be the volume that’s offsite.

    For NetWorker versions 7.3.1 and lower, there was only one solution to this – you had to hunt down the actual clone saveset instances NetWorker was asking for, mark them as suspect, and reattempt the recovery. If you managed to mark them all as suspect, then you’d be able to ‘force’ NetWorker into facilitating the recovery from the volume(s) that had been staged to. However, after the recovery you had to make sure you backed out of those changes, so that both the clones and the staged copies would be considered not-suspect.

    Some companies, in this situation, would instigate a tape rotation policy such that clone volumes would be brought back from off-site before savesets were likely to be staged out, with subsequently staged media sent offsite. This has a dangerous side-effect of temporarily leaving all copies of backups on-site, jeapordising disaster recovery situations, and hence it’s something that I couldn’t in any way recommend.

    The solution introduced around 7.3.2 however is far simpler – a mminfo flag called offsite. This isn’t to be confused with the convention of setting a volume location field to ‘offsite’ when the media is removed from site. Annoyingly, this remains unqueryable; you can set it, and NetWorker will use it, but you can’t say, search for volumes with the ‘offsite’ flag set.

    The offsite flag has to be manually set, using the command:

    # nsrmm -o offsite volumeName

    (where volumeName typically equals the barcode).

    Once this is set, then NetWorker’s standard saveset (and therefore volume) selection criteria is subtly adjusted. Normally if there are no online instances of a saveset, NetWorker will request the saveset with the lowest CloneID. However, saveset instances that are on volumes with the offsite flag set will be deemed ineligible and NetWorker will look for a saveset instance that isn’t flagged as being offsite.

    The net result is that when following a traditional backup model with ADV_FILE disk backup (backup to disk, clone to tape, stage to tape), it’s very important that tape offsiting procedures be adjusted to set the offsite flag on clone volumes as they’re removed from the system.

    The good news is that you don’t normally have to do anything when it’s time to pull the tape back onsite. The flag is automatically cleared* for a volume as soon as it’s put back into an autochanger and detected by NetWorker. So when the media is recycled, the flag will be cleared.

    If you come from a long-term NetWorker site and the convention is still to mark savesets as suspect in this sort of recovery scenario, I’d suggest that you update your tape rotation policies to instead use the offsite flag. If on the other hand, you’re about to implement an ADV_FILE based backup to disk policy, I’d strongly recommend you plan in advance to configure a tape rotation policy that uses the offsite flag as cloned media is sent away from the primary site.


    * If you did need to explicitly clear the flag, you can run:

    # nsrmm -o notoffsite volumeName

    Which would turn the flag back off for the given volumeName.

    Feb 222010
     

    The scenario:

    • A clone or stage operation has aborted (or otherwise failed)
    • It has been restarted
    • It hangs waiting for a new volume even though there’s a partially written volume available.

    This is a relatively easy problem to explain. Let’s first look at the log messages that happens. To generate this error, I started cloning some data to the “Default Clone” pool, with only one volume in the pool, then aborted. Shortly thereafter I tried to run the clone again, and when NetWorker wouldn’t write to the volume I unmounted and remounted it – a common thing that newer administrators will try in this scenario. This is where you’ll hit the following error in the logs:

    media notice: Volume `800829L4' ineligible for this operation; Need a different volume
    from pool `Default Clone'
    media info: Suggest manually labeling a new writable volume for pool 'Default Clone'

    So, what’s the cause of this problem? It’s actually relatively easy to explain.

    A core component in NetWorker’s media database design is that a saveset can only ever have one instance on a piece of media. This applies as equally to failed as complete saveset instances.

    The net result is that this error/situation will occur because it’s meant to – NetWorker doesn’t permit more than one instance of a saveset to appear on the same piece of physical media.

    So what do you do when this error comes up?

    • If you’re backing up to disk, an aborted saveset should normally be cleared up automatically by NetWorker after the operation is aborted. However, in certain instances this may not be the case. For NetWorker 7.5 vanilla and 7.5.1.1/7.5.1.2, this should be done by expiring the saveset instance – using nsrmm to flag the instance as having an expiry date within a few minutes or seconds. For all other versions of NetWorker, you should just be able to delete the saveset instance.
    • When working with tape (virtual or physical), the most recommended approach would be to move on to another tape, or if the instance is the only instance on that tape, relabel the tape. (Some would argue that you can use nsrmm to delete the saveset instance from the tape and then re-attempt the operation, but since NetWorker is so heavily designed to prevent multiple instances of a saveset on a piece of media, I’d strongly recommend against this.)

    Overall it’s a fairly simple issue, but knowing how to recognise it lets you resolve it quickly and painlessly.

    The top 10 for 2009

     Architecture, Basics, NetWorker, Quibbles, Site, Support  Comments Off on The top 10 for 2009
    Jan 062010
     

    Looking at the stats both for this new site and the previous site, I’ve compiled a list of the top 10 read articles on The NetWorker Blog for 2009. The top 3 of course match the three articles that routinely turn out to be the most popular on any given month, which speaks something of their relevance to the average NetWorker administrator.

    (Note: I’ve excluded non-article pages from the top 10.)

    Number 10 – Instantiating Savesets

    The very first article on the blog, Instantiating Savesets detailed the importance of distinguishing between all instances of a saveset and a specific instance of a saveset.

    This distinction between using just the saveset ID, and using a saveset ID/clone ID combination becomes particularly important when staging from disk backup units. If clones exist and you stage using just the saveset ID, when NetWorker cleans up at the end of the staging operation it will remove reference to the clones as well as deleting the original from the disk backup unit. (Something you really don’t want to have happen.)

    Recommendation to EMC: Perhaps it would be worthwhile requiring a “-y” argument to nsrstage if staging savesets from disk backup units and specifying only the saveset ID.

    Recommendation to NetWorker administrators: Always be careful when staging that you specify both the saveset and the clone ID.

    Number 9 – Basics – Important mminfo fields

    In May I wrote about a few key mminfo fields – notably:

    • savetime
    • sscreate
    • ssinsert
    • sscomp
    • ssaccess

    Sadly, I didn’t get the result I wanted with EMC on ssaccess. Documented as being updated whenever a saveset fragment is accessed for backup and recovery, the most I could get was an acknowledgement that it was currently broken and to lodge an RFE to get it fixed. (The alternative was to have the documentation changed to take out reference to read operations – something I didn’t want to have happen!)

    Recommendation to EMC: ssaccess would be a particularly useful mminfo field, particularly when analysing recovery statistics for NetWorker. Please fix it.

    Number 8 – Basics – Listing files in a backup

    Want to know what files were backed up as part of the creation of a saveset? If you do, you’re not unique – this has remained a very popular article since it was written in January.

    Recommendation to EMC: This information can be retrieved via a combination of mminfo/nsrinfo, but it would be handy if NMC supported drilling down into a saveset to provide a file listing.

    Number 7 – Using yum to install NetWorker on Linux

    NetWorker’s need for dependency resolution on Linux for installation of the client packages in particular drew a lot of people to this article.

    Number 6 – Basics – mminfo, savetime, and greater than/less than

    This article explained why NetWorker uses the greater than and less than signs in mminfo in a way that newcomers to the product might find backwards. If you’re not aware of why mminfo works the way it does for specifying savetimes, you should be.

    Number 5 – 7.5(.1) changed behaviour – deleting savesets from adv_file devices

    This was a particularly unpleasant bug introduced into NetWorker 7.5, thankfully resolved now in the cumulative service releases and NetWorker 7.6

    The gist of it is that in NetWorker 7.5/7.5.1 (aka 7.5 SP1), if you deleted a saveset on a disk backup unit, NetWorker would suffer a serious failure where it would from that point have issues cleaning regular expired savesets from the disk backup unit and insist that the disk backup unit had major issues. The primary error would manifest as:

    nsrd adv_file warning: Failed to fetch the saveset(ss_t) structure for ssid 1890993582

    This was fixed in 7.5.1.2, thankfully.

    Recommendation to EMC: Never let this bug see the light of day again, please. (So far you’re doing an excellent job, by the way.)

    Number 4 – NetWorker 7.5.1 Released

    I’ve recently noticed a disturbing trend among many vendors, EMC included, where once a new release is made of a product, sales and account staff become overly enthusiastic about recommending new releases. This comes on top of not really having any technical expertise. (Please be patient, I’m trying to put this as diplomatically as possible.)

    One of the worst instances I’ve seen of this in the last few years was the near-hysterical pumping of 7.5 thanks to some useful features to do with virtualisation in particular. I’ll admit that my articles on the integration between Oracle Module 5 and NetWorker 7.5, as well as Probe Based Backups may have added to this. However, there was somewhat of a stampede to 7.5 when it came out, and consequently, when it had some issues, there was strong enthusiasm for the release of 7.5.1.

    This is why, by the way, that IDATA maintains for its support customers a recommended versions list that is not automatically updated when new versions of products come out.

    Recommendation to EMC: Remind your sales staff that existing users already have the product, and not to just go blindly convincing them to upgrade. Otherwise you’ll eventually start sounding like this.

    Number 3 – Carry a jukebox with you (if you’re using Linux)

    During 2009, Mark Harvey’s LinuxVTL project first got the open source LinuxVTL working with NetWorker in a single drive configuration, then eventually, in multi-drive configurations. (Mark assures me, by the way, that patches are coming real soon to allow multiple robots on the same storage node/server.)

    Lesson for me: With the LinuxVTL configured on multiple lab servers in my environment, I’ve really taken to VTLs this year, and considerably changed my attitude on using them. (I’ll say again: I still resent that they’re needed, but I now respect them a lot more than I previously did.)

    Lesson for others: Even Mark himself says that the open source VTL shouldn’t be used for production backups. Don’t be cheap with your backup system, this is an excellent tool for lab setups, training, diagnostics, etc., but it is not a replacement to a production-ready VTL system. If you want a VTL, buy a VTL.

    Number 2 – Basics – Parallelism in NetWorker

    Some would say that the high popularity of an article about parallelism in NetWorker indicates that it’s not sufficiently documented.

    I’m not entirely convinced that’s the case. But it does go to show that it’s an important topic when it comes to performance tuning, and summary articles about how the various types of parallelism interact are obviously popular.

    Lesson for everyone: Now that the performance tuning guide has been updated and made more relevant in NetWorker 7.6, I’d recommend people wanting an official overview of some of the parallelism options checking that out in addition to the article above.

    Number 1 – Basics – Fixing “NSR peer information” errors

    Goodness this was a popular article in 2009 – detailing how to fix the “NSR peer information” errors that can come up from time to time in the NetWorker logs. If you’re not familiar with this error yet, it’s likely you will eventually as a NetWorker administrator see an error such as:

    39078 02/02/2009 09:45:13 PM  0 0 2 1152952640 5095 0 nox nsrexecd SYSTEM error: There is already a machine using the name: “faero”. Either choose a different name for your machine, or delete the “NSR peer information” entry for “faero” on host: “nox”

    Recommendation for EMC: Users shouldn’t really need to be Googling for a solution to this problem. Let’s see an update to NetWorker Management Console where these errors/warnings are reported in the monitoring log, with the administrator being able to right click on them and choose to clear the peer information after confirming that they’re confident no nefarious activity is happening.

    Wrapping Up

    I have to say, it was a fantastically satisfying year writing the blog, and I’m looking forward to seeing what 2010 brings in terms of most useful articles.

    Nov 132009
     

    Something that continues to periodically come up is the need to remind people running manual staging to ensure they specify both the SSID and the Clone ID when they stage. I did some initial coverage of this when I first started the blog, but I wanted to revisit and demonstrate exactly why this is necessary.

    The short version of why is simple: If you stage by SSID alone, NetWorker will delete/purge all instances of the saveset other than the one you just created. This is Not A Good Thing for 99.999% of what we do within NetWorker.

    So to demonstrate, here’s a session where I:

    1. Generate a backup
    2. Clone the backup to tape
    3. Stage the saveset only to tape

    In between each step, I’ll run mminfo to get a dump of what the media database says about saveset availability.

    Part 1 – Generate the Backup

    Here’s a very simple backup for the purposes of this demonstration, and the subsequent mminfo command to find out about the backup:

    [root@tara ~]# save -b Default -LL -q /etc
    save: /etc  106 MB 00:00:07   2122 files
    completed savetime=1258093549
    
    [root@tara ~]# mminfo -q "client=tara.pmdg.lab,name=/etc" -r volume,ssid,cloneid,
    savetime
     volume        ssid          clone id  date
    Default.001    2600270829  1258093549 11/13/2009
    Default.001.RO 2600270829  1258093548 11/13/2009

    There’s nothing out of the ordinary here, so we’ll move onto the next step.

    Part 2 – Clone the Backup

    We’ll just do a manual clone to the Default Clone pool. Here we’ll specify the saveset ID alone, which is fine for cloning – but is often what leads people to being in the habit of not specifying a particular saveset instance. I’m using very small VTL tapes, so don’t be worried that in this case I’ve got a clone of /etc spanning 3 volumes:

    [root@tara ~]# nsrclone -b "Default Clone" -S 2600270829
    [root@tara ~]# mminfo -q "client=tara.pmdg.lab,name=/etc" -r volume,ssid,cloneid,
    savetime
     volume        ssid          clone id  date
    800843S3       2600270829  1258094164 11/13/2009
    800844S3       2600270829  1258094164 11/13/2009
    800845S3       2600270829  1258094164 11/13/2009
    Default.001    2600270829  1258093549 11/13/2009
    Default.001.RO 2600270829  1258093548 11/13/2009

    As you can see there, it’s all looking fairly ordinary at this point – nothing surprising is going on at all.

    Part 3 – Stage by Saveset ID Only

    In this next step, I’m going to stage by saveset ID alone rather than specifying the saveset ID/clone ID, which is the correct way of staging, so as to demonstrate what happens at the conclusion of the staging. I’ll be staging to a pool called “Big”:

    [root@tara ~]# nsrstage -b Big -v -m -S 2600270829
    Obtaining media database information on server tara.pmdg.lab
    Parsing save set id(s)
    Migrating the following save sets (ids):
     2600270829
    5874:nsrstage: Automatically copying save sets(s) to other volume(s)
    
    Starting migration operation...
    Nov 13 17:34:00 tara logger: NetWorker media: (waiting) Waiting for 1 writable
    volume(s) to backup pool 'Big' disk(s) or tape(s) on tara.pmdg.lab
    5884:nsrstage: Successfully cloned all requested save sets
    5886:nsrstage: Clones were written to the following volume(s):
     BIG991S3
    6359:nsrstage: Deleting the successfully cloned save set 2600270829
    Successfully deleted original clone 1258093548 of save set 2600270829
    from media database.
    Successfully deleted AFTD's companion clone 1258093549 of save set 2600270829
    from media database with 0 retries.
    Successfully deleted original clone 1258094164 of save set 2600270829
    from media database.
    Recovering space from volume 4294740163 failed with the error
    'Cannot access volume 800844S3, please mount the volume or verify its label.'.
    Refer to the NetWorker log for details.
    6330:nsrstage: Cannot access volume 800844S3, please mount the volume
    or verify its label.
    Completed recover space operation for volume 4177299774
    Refer to the NetWorker log for any failures.
    Recovering space from volume 4277962971 failed with the error
    'Cannot access volume 800845S3, please mount the volume or verify its label.'.
    Refer to the NetWorker log for details.
    6330:nsrstage: Cannot access volume 800845S3, please mount the volume
    or verify its label.
    Recovering space from volume 16550059 failed with the error
    'Cannot access volume 800843S3, please mount the volume or verify its label.'.
    Refer to the NetWorker log for details.
    6330:nsrstage: Cannot access volume 800843S3, please mount the volume
    or verify its label.

    You’ll note there’s a bunch of output there about being unable to access the clone volumes the saveset was previously cloned to. When we then check mminfo, we see the consequences of the staging operation though:

    [root@tara ~]# mminfo -q "client=tara.pmdg.lab,name=/etc" -r volume,ssid,cloneid,
    savetime
     volume        ssid          clone id  date
    BIG991S3       2600270829  1258095244 11/13/2009

    As you can see – no reference to the clone volumes at all!

    Now, has the clone data been erased? No, but it has been removed from the media database, meaning you’d have to manually scan the volumes back in order to be able to use them again. Worse, if those volumes only contained clone data that was subsequently removed from the media database, they may become eligible for recycling and get re-used before you notice what has gone wrong!

    Wrapping Up

    Hopefully the above session will have demonstrated the danger of staging by saveset ID alone. If instead of staging by saveset ID we staged by saveset ID and clone ID, we’d have had a much more desirable outcome. Here’s a (short) example of that:

    [root@tara ~]# save -b Default -LL -q /tmp
    save: /tmp  2352 KB 00:00:01     67 files
    completed savetime=1258094378
    [root@tara ~]# mminfo -q "name=/tmp" -r volume,ssid,cloneid
     volume        ssid          clone id
    Default.001    2583494442  1258094378
    Default.001.RO 2583494442  1258094377
    [root@tara ~]# nsrclone -b "Default Clone" -S 2583494442
    
    [root@tara ~]# mminfo -q "name=/tmp" -r volume,ssid,cloneid
     volume        ssid          clone id
    800845S3       2583494442  1258095244
    Default.001    2583494442  1258094378
    Default.001.RO 2583494442  1258094377
    [root@tara ~]# nsrstage -b Big -v -m -S 2583494442/1258094377
    Obtaining media database information on server tara.pmdg.lab
    Parsing save set id(s)
    Migrating the following save sets (ids):
     2583494442
    5874:nsrstage: Automatically copying save sets(s) to other volume(s)
    
    Starting migration operation...
    
    5886:nsrstage: Clones were written to the following volume(s):
     BIG991S3
    6359:nsrstage: Deleting the successfully cloned save set 2583494442
    Successfully deleted original clone 1258094377 of save set 2583494442 from
    media database.
    Successfully deleted AFTD's companion clone 1258094378 of save set 2583494442
    from media database with 0 retries.
    Completed recover space operation for volume 4177299774
    Refer to the NetWorker log for any failures.
    
    [root@tara ~]# mminfo -q "name=/tmp" -r volume,ssid,cloneid
     volume        ssid          clone id
    800845S3       2583494442  1258095244
    BIG991S3       2583494442  1258096324

    The recommendation that I always make is that you forget about using saveset IDs alone unless you absolutely have to. Instead, get yourself into the habit of always specifying a particular instance of a saveset ID via the “ssid/cloneid” option. That way, if you do any manual staging, you won’t wipe out access to data!

    Sep 082009
     

    In NetWorker, staging refers to moving savesets from one piece of media to another. The move operation is two-fold, consisting of:

    • A clone operation (source -> target)
    • A delete operation (source)

    Both of these operations are done automatically as part of the staging process, with the delete being defined as the maximum supported operation for the media that the source was on; for tape based media, this means deleting the media database entries for the saveset(s) staged, and for disk this means both the media database delete and the filesystem delete of the saveset from the disk backup unit.

    There’s a few reasons why you use staging policies in NetWorker:

    1. To free up space on disk backup units.
    2. To move backups/clones from a previous media type to a new media type.
    3. To move backups/clones from older, expiring media to new media for long-term retention.

    The second two options usually refer to tape -> tape staging, which these days is the far less common use of staging in NetWorker. The most common use it now for managing used space on disk backup units, and that’s what we’ll consider here.

    There’s two ways you can stage within NetWorker – either as a scheduled task, or as a manual task.

    Scheduled Staging

    Scheduled staging occurs by creating one or more staging policies. Typically in a standard configuration for disk backup units, you’ll have one staging policy per disk backup unit. For example:

    Creating a staging policy

    Creating a staging policy

    The staging policy consists of settings that define:

    • Name/Comment – Identification details for the staging policy, as you’d expect.
    • Enabled – Default is no, set to Yes if you want the staging policy to run. (Note that you can’t start a disabled staging policy manually – or you couldn’t, last time I checked.)
    • Which disk backup units (devices) will be read from. Choose both the read-only and the read-write version of the disk backup units. (Unless there are significant issues, NetWorker will always read from the read-only version of the disk backup unit anyway.)
    • Destination pool – where savesets will be written to.
    • High water mark – this is expressed as a percentage of the total filesystem capacity that the disk backup resides on (which is why each disk backup unit should be on its own filesystem!). It basically means “if the occupied space in savesets reaches <nominated> percent, then start staging data off”.
    • Low water mark – again, a percentage of the total filesystem capacity that the disk backup resides on. If staging is initiated due to a high watermark value, then staging will continue until the disk backup unit can be freed up such that the used space at the end of the staging is equal to or less than the low water mark.
    • Save set selection – will be one of oldest/largest/youngest/smallest. For most disk backup units, the choice is normally between oldest saveset or largest saveset.
    • Max storage period / period unit – defines the maximum amount of time, in either days or hours, that savesets can remain on disk before they must be staged out. This will occur irrespective of any watermarks (and the watermarks, similarly, will occur irrespective of any maximum storage period).
    • Recover space interval / interval unit – defines how frequently NetWorker will check to see if there are any recyclable savesets that can be removed from disk. (Aborted savesets, while checked for in this, should be automatically cleaned up when they are aborted.)
    • Filesystem check interval / interval unit – defines how frequently NetWorker will check to see whether it should actually perform any staging.

    While there’s a lot of numbers/settings there in a small dialog, they actually all make sense. For instance, let’s consider the staging policy defined in the above picture. It shows a policy called “Daily” that will move savesets out from /d/nsr/01 to the Daily pool, with the following criteria:

    • If the disk backup unit becomes 85% full, it will commence staging until it has moved enough savesets to return the disk backup unit to 50% capacity.
    • Any saveset that is older than 7 days will be staged.
    • Whenever savesets are staged, they will be picked in order of oldest to newest.
    • It will check for recyclable/removable savesets every 4 hours.
    • It will check to see if staging can be run every 7 hours.

    Note that if the disk backup unit becomes full, staging by watermark will be automatically kicked off, even if the 7 hour wait between staging checks hasn’t been reached.

    One important note here – If you have multiple disk backup units with volumes labelled into the same pool, you can choose to either have them all in one staging policy, have one staging policy per disk backup unit, or have a series of staging policies with one or more disk backup unit in each policy. There’s risks/costs associated with each option. If you have too many defined under a single staging policy, then staging becomes very “single-threaded” in terms of disks read from; this can significantly slow down the staging policy. Alternatively, if you have one staging policy per disk backup unit, but the number of disk backup units exceeds the number of tape drives, you can end up with significant contention between staging, cloning and tape-recovery operations. It’s a fine balancing act.

    Manual staging

    Manual staging is accomplished by running the nsrstage command. That, incidentally, is what happens in the background for scheduled staging – the staging policy runs, evaluates what needs to be staged, then runs nsrstage.

    The standard way of invoking nsrstage is:

    # nsrstage -b destinationPool -m -v -S ssid/cloneid

    or

    # nsrstage -b destinationPool -m -v -S -f /path/to/file.txt

    In the first, you’re staging a single saveset instance. NOTE: You must always specify ssid/cloneid; if you don’t – if you just specify the saveset ID, then when NetWorker cleans up at the end of the staging operation, it will delete all other instances of the saveset. So if you’ve got a clone, you’ll lose reference to the clone!

    In the second instance, you’re staging multiple sssid/cloneid combinations, specified one per line in a plain text file.

    (There are alternate mechanisms for calling nsrstage to either clean the disk backup unit, or stage by volume. These aren’t covered here.)

    As with all pools these days in NetWorker, you can either have the savesets staged with their original retention period in place, or stage them to a pool with a defined retention policy, in which case their retention policy will be adjusted accordingly when they are staged.

    Scripting

    You can of course script staging operations; particularly when running manually you’ll likely first run mminfo, then run nsrstage against a bunch of savesets. Alternatively, you may want to check out dbufree, a utility within the IDATA Tools utility suite; this offers considerable enhancements over regular staging, including (but not limited to):

    • Stage savesets selected on a disk backup unit by any valid mminfo query (e.g., stage by group…)
    • Specify how much space you want to free up, rather than watermark based
    • Stage in chunks – stage enough space to free up a nominated amount of capacity, stop to allow reclamation to take place, then start staging again.
    • Only stage savesets that have clones.
    • Enhanced saveset order options – in addition to biggest/smallest/oldest/newest, there’s options to order savesets by client (to logically assist in speeding up recovery from multiple backups) in any of the ‘primary’ sort methods as a sub-order.

    Cloning

    Talking about staging wouldn’t be complete if we also didn’t mention cloning. In an ideal configuration for disk backup units, you should:

    • Backup to disk
    • Clone to tape
    • (Later), stage to tape.

    This can create some recovery implications. I’ve covered that previously in this post.

    Quibbles – Cloning and Staging

     Quibbles  Comments Off on Quibbles – Cloning and Staging
    Aug 132009
     

    For the most part cloning and staging within NetWorker are pretty satisfactory, particularly when viewed from a combination of automated and manual operations. However, one thing that constantly drives me nuts is the inane reporting of status for cloning and staging.

    Honestly, how hard can it be to design cloning and staging to accurately report the following at all times:

    Cloning X of Y savesets, W GB of Z GB

    or

    Staging X of Y savesets, W GB of Z GB

    While there have been various updates to cloning and staging reporting, and sometimes it at least updates how many savesets it has done, it continually breaks when dealing with the total amount staged/cloned in as much as it resets whenever a destination volume is changed.

    Oh, and while I’m begging for this change, I will request one other – include in daemon.raw whenever cloning/staging occurs, the full list of ssid/cloneids that have been cloned or staged, and the client/saveset details for each one – not just minimal details when a failure occurs. It’s called auditing.

    Apr 252009
     

    Yesterday I wanted to delete a few savesets from a lab server I’d upgraded from 7.4.4 to 7.5.1.

    Wanting to go about it quickly, I did the following:

    • I used “nsrmm -dy -S ssid” for each saveset ID I wanted to delete, to erase it from the media database.
    • I used “nsrstage -C -V volumeName” for the disk backup unit volumes to run a cleaning operation.

    Imagine my surprise when, instead of seeing a chunk of space being freed up, I got a lot of the following notifications:

    nsrd adv_file warning: Failed to fetch the saveset(ss_t) structure for ssid 1890993582

    I got one of these for every saveset I deleted. And since I’d run a lot of tests, that was a lot of savesets. The corresponding result was that they all remained on disk. What had been a tried and true version of saveset deletion under 7.4.x and below appears to not be so useful under 7.5.1.

    In the end I had to run a comparison between media database content and disk backup unit content – i.e.:

    # mminfo -q "volume=volName" -r "ssid(60)"

    To extract the long saveset IDs, which are in effect the names of the files stored on disk, then:

    # find /path/to/volume -name -print

    Then for each filename, check to see whether it existed in the media database, and if it didn’t, manually delete it. This is not something the average user should do without talking to their support people by the way, but, well, I am support people and it was a lab server…

    This change is worrying enough that I’ll be running up a couple of test servers using multiple operating systems (the above happened on Linux) to see whether its reproducible or whether there was just say, some freaky accident with the media database on my lab machine.

    I’ll update this post accordingly.

    [Update – 2009-04-27]

    Have done some more tests on 7.5.1 on various Linux servers, comparing results to 7.4.4. This is definitely changed behaviour and I don’t like it, given that it’s very common for backup administrators to delete one or two savesets here and there from disk. Chatting to EMC about it.

    In the interim, here’s a workaround I’ve come up with – instead of using nsrmm -d to delete the saveset, instead run:

    # nsrmm -w now -e now -S ssid

    To mark the saveset as immediately recyclable. Then run “nsrim -X” to force a purge. That will work. If you have scripts though that manually delete savesets from disk backup units, you should act now to update them.

    [Update – 2009-04-30]

    It would appear as well that if you delete then attempt to reclaim space, NetWorker will flag the “scan required” flag for a volume. Assuming you’re 100% OK with what you’ve manually deleted and then purged from disk using rm, you can probably safely clear the flag (nsrmm -o notscan). If you’re feeling paranoid, unmount the volume, scan it, then clear the flag.

    [Update – 2009-05-06]

    Confirmed this isn’t present in vanilla 7.5. It seemed to occur in 7.5.1.

    [Update – 2009-06-16]

    Cumulative patches for 7.5.1 have been released; according to EMC support these patches include the fixes for addressing this issue, allowing a return to normal operations. If you’re having this issue, make sure you touch base with EMC support or your EMC support partner to get access to the patches. (Note: I’ve not had a chance to review the cumulative patches, so I can’t vouch for them yet.)

    [Update 2009-08-11]

    I forgot to update earlier; the cumulative patches (7.5.1.2 in the case of what I received) did properly incorporate the patch for this issue.

    %d bloggers like this: