Mar 302016

At the start of the week we saw NetWorker 8.2 SP3 released. Now, you might think given NetWorker 9 is out there’s no new features in NetWorker 8.2 SP3, but you’d be wrong.

NetWorker 9 is a jump – it’s a change of processes and it’s a new way of going about configuring your backups. I’m seeing more details every day of people having great experiences with NetWorker 9, but backup is one of those areas where change can often come slowly, so 8.2 still gets a lot of attention. So if you’re the sort of business that needs the features in NetWorker 9 you can dive in, but if you want to hang back for a little while yet, 8.2 will have you covered for a while yet.

nsrwatch in 8.2.3

The all new nsrwatch

OK, I admit I’m a bit of an nsrwatch junkie. Unless I have to setup a Windows NetWorker server I’ll setup NetWorker on Linux every time. (But at heart I’m still a Unix system administrator. It was the Unix integration that drove me to Mac OS X, after all.)

There’s a lot of good new features in NetWorker 8.2 SP3 but I have to admit given my CLI-junkie status, I just love the update to nsrwatch. For me this handy little utility has saved me thousands or more times from having to launch a full GUI, and if you’ve ever seen how many windows I end up having active on my screens at the same time you’ll understand why that’s a good thing.

The good old nsrwatch utility now gives you a lot more control over what you see on-screen. You can resize panels or even turn them off and setup an environment variable to make that your default view. You can switch between different views – e.g., all devices (seen above), mounted devices and active devices:

Mounted Devices

nsrwatch showing mounted devices only

nsrwatch showing active devices

nsrwatch showing active devices

You also get control options directly embedded into nsrwatch now:

nsrwatch with control options

nsrwatch with control options

All up, a great set of changes. I was lucky enough to try out some of the options while they were under development, so I’ve been looking forward to talking about it for some time now!

That’s not the only features in NetWorker 8.2 SP3 though – but it did really appeal to my I’ve-been-using-NetWorker-for-20-years inner-geek – so it’s time to move on to the rest of the enhancements!

Server Capability

There’s big changes under the hood in SP3 – the media catalogue has been migrated to SQLite to take advantage of the huge performance increases this gave in NetWorker 9 – and it’ll make the migration path to NetWorker 9 a little more streamlined as well. This may sound like a minor change, but the switch to SQLite is really important; the old format media database was great and stable, but it had limits on the amount of concurrent operations you could do. SQLite is great and stable and a lot more capable of supporting a number of concurrent operations.

The server daemons have had some tweaks as well – a bunch of issues that could lead to a server hang situation have been quashed, and the number of DNS reverse lookups performed has been pared down. The DNS caches used in a bunch of NetWorker daemons are now populated from nsrd to improve lookup performance as well. Also if you’ve got a lot of storage nodes in your environment, there are options to do a staggered start of the storage node manager daemons to improve startup performance.

Data Domain

8.2 SP3 includes support for DDOS 5.7 with an update of Data Domain libraries to 3.1. This will align it to some new options coming out soon, not to mention the Data Domain High Availability option introduced in the last month for the DD9500. (One of the other things it’ll align to I can blog about in a few days, hopefully.)

There’s performance enhancements for Clone Controlled Replication (CCR) as well, allowing for boosts (no pun intended) in the performance of cloning operations between two Data Domain systems under NetWorker control.

SP3 also introduces support for Distributed Segment Processing and all other Boost goodness into the Mac OS X client. That means if you’ve got some Mac clients within your NetWorker environment they’ll now get all of the Boost advantages you see everywhere else.

Updated Support

There’s a whole bunch of platforms and options that have had support added in this release. Check out the new VBA appliances if you’re backing VMware, too – you’ll definitely want to take advantage of updates there. But it’s not just VMware backups. This version of NetWorker also adds support for:

  • LTO7 tape drives
  • Mac OS X 10.11 El Capitan
  • Snapshot Management for NetApp SnapVault and SnapMirror ‘C-mode’ operations – creation, replication, restore and rollover
  • Hitachi NAS token based backups
  • Isilon Fast Incremental – Making backup of really large filesystems a whole lot easier
  • SQL Server AlwaysOn availability groups in a Failover Cluster (great way of offloading backups in SQL Enterprise Server environments)
  • MySQL 5.7.9/MySQL Enterprise Backup 4

In Summary

You won’t see the same sorts of massive features lists in a service pack release as you do in a full new release, but that being said 8.2 SP3 packs some wallop for your environment if you’re still in the 8.2 tree – or using an earlier version still. In addition to all the standard fixes that go into any service pack, rolled up from previous service packs and cumulative releases, 8.2 SP3 has been fine tuned for performance and scaleability and will ensure those customers not yet ready to upgrade to NetWorker 9 have an excellent platform to settle onto.

You can find the 8.2 SP3 binaries in the downloads section of the NetWorker product support page, and you can access the release notes directly from this link.

NetWorker Scales El Capitan

 Basics, NetWorker  Comments Off on NetWorker Scales El Capitan
Dec 202015

When Mac OS X 10.11 (El Capitan) came out, I upgraded my personal desktop and laptop to El Capitan. The upgrade went pretty smoothly on both machines, but then I noticed overnight my home lab server reported backup errors for both systems.

When I checked the next day I noticed that NetWorker actually wasn’t installed any more on either system. It seemed odd for NetWorker to be removed as part of the install, but hardly an issue. I found an installer, fired it up and to my surprise found the operating system warning me the installer might cause system problems if I continued.

Doing what I should have done from the start, I set up an OS X virtual machine to test the installation on, and it seemed to go through smoothly until the very end when it reported a failure and backed out of the process. That was when I started digging into some of the changes in El Capitan. Apple, it turns out, is increasing system security by locking down third party access to /bin, /sbin, /usr/bin and /usr/sbin. As NetWorker’s binaries on Unix systems install into /usr/bin and /usr/sbin, this meant NetWorker was no longer allowed to be installed on the system.

El Capitan

Fast forward a bit, and as of NetWorker 8.2 SP2 Cumulative Release 2 (aka, was released a week or so ago including a relocated NetWorker installer for Mac OS X – now the binaries are located in /usr/local/bin and /usr/local/sbin instead. (Same goes for NetWorker 9.) Having run on my home Macs for a couple of weeks, with backup and recovery testing, the new location works.

If you’ve got Mac OS X systems being upgraded to El Capitan, be sure to download NetWorker

Oh, and don’t forget to fill in the 2015 NetWorker Usage Survey!

Jul 202010

I have, on a few occasions, been puzzled as to how to downgrade NetWorker on Mac OS X. There’s a couple of distinct issues that I’ve come up against, and I thought I’d outline them here now that I’ve fully resolved how to do it.

The first is that when NetWorker installs, it’s meant to install uninstall utilities into /Library/Receipts/NetWorker.pkg. However, on Snow Leopard, NetWorker doesn’t write this uninstall information, meaning that technically it’s not possible to uninstall the product. There is, thankfully, a way around this.

First, open up your NetWorker.dmg file, but then drop into the command line and change directory into the NetWorker.pkg/Contents directory within the dmg:

Important Directory Listings - NetWorker Mac OS X Package

In the above screen shot, I’ve shown the two directories you need to be aware of; NetWorker.pkg/Contents, and NetWorker.pkg/Contents/Resources.

You’ll note in the Resources directory that there’s a NetWorkerUninstall script, which needs to be run as root. However, the script depends on there being some content in /Library/Receipts/NetWorker.pkg, so you’ll need to do the following:

$ sudo bash
# cd /Volumes/NetWorker<<version>>/NetWorker.pkg/Contents
# mkdir -p /Library/Receipts/NetWorker.pkg/Contents
# cp /Library/Receipts/NetWorker.pkg/Contents
# cp Resources/NetWorkerUninstall /Library/Receipts/NetWorker.pkg
# /Library/Receipts/NetWorker.pkg/NetWorkerUninstall

Once you run the NetWorkerUninstall script, there’ll be a brief pause before you see a flash of lines with entries such as:

Removing: /usr/share/man/man8/tur.8

and so on.

At the end of this, you theoretically should be able to run the NetWorker installer for the version you want to install. However, you’re likely to still end up with the following output from the installer:

NetWorker Installer - Newer Version Exists

It was this step that had been frustrating me. Thankfully though, I finally started to think like a combined Mac + Unix user, and released there was probably a plist style file hanging around somewhere that wasn’t being cleaned up by the uninstaller, and that if it followed Apple’s naming conventions, it would be com.emc.*.plist. So I did:

# find -xdev / -name "com.emc.*" -print

Lo and behold, I found the following:


Removing them was the final piece of the puzzle – without them hanging around, the NetWorker installer utility didn’t pick up there was a newer version of the software installed, and I was finally able to downgrade NetWorker for testing purposes.

May 032010

Less than a month ago, Apple released service pack 3 to Snow Leopard – i.e., 10.6.3. A few days after that they released which was apparently only needed in a few instances, but I downloaded and applied anyway due to some irregularities I’d noticed with my OS after installing the vanilla 10.6.3.

It’s recently occurred to me that NetWorker (7.6) has been a heck of a lot more reliable since going to 10.6.3 / As always, it’s a bit of a grey zone, since it’s not officially supported (and there’s definitely some patching required) – hence the wait for 7.6 SP1, but overall I’m now noticing that the client process remains contactable by the server across multiple sleep/wake and/or location transitions, something that it wouldn’t do before. There’s still some other behavioural oddities, but overall, I realised that I’ve not reinstalled the client on my laptop now for over 2 weeks, which is a bit of a record since I installed Snow Leopard. If you’re in a situation where you absolutely have to be running 10.6 and backing up with NetWorker, and knowing it’s not currently supported, I’d suggest you make sure you’re on

Mar 112010

While initially I had some success with Snow Leopard and Mac OS X, I’m increasingly finding that it’s just boiling down to being too random for reliable backups. So far problems mainly seem to occur after a machine has gone to sleep and woken up multiple times – or had its network location changed multiple times. Thus it mainly seems (for the moment) to affect laptops or machines that frequently sleep.

The net result is that you’ll get into situations where several errors will start to happen and you’ll need to eventually reinstall the NetWorker client, reboot, and then potentially reinstall the NetWorker client another time. Note that complete cold restarts do not seem to as reliably fix (or temporarily offer a workaround to the) issues as does the reinstall/reboot/reinstall method.

Error 1

Attempts to connect from the server to the client will fail – e.g.,

[root@nox ~]# nsradmin -p 390113 -s archon
39078:nsradmin: RPC error: Remote system error
There does not appear to be a NetWorker nsrexecd server running on archon.

Error 2

Stopping and restarting the NetWorker services on the client fails:

root@archon ~
$ SystemStarter stop NetWorker
Stopping NetWorker Client.
root@archon ~
$ ps -eaf | grep nsr
0  5381  5230   0   0:00.00 ttys001    0:00.00 grep nsr
root@archon ~
$ SystemStarter start NetWorker
Starting NetWorker Client.
/Library/StartupItems/NetWorker/NetWorker: line 10:  5389 Illegal instruction     /usr/sbin/nsrexecd

Error 3

I’m finding that directives are getting confused over directories and paths too:

* archon:/ 70340:savepnpc: ignoring directory specification for `/Users/preston/Library/Application Support/Yojimbo/' in
* archon:/ `/Users/preston/Library/Application Support/Yojimbo/.nsr' - not contained within directory `/users/preston/Library/Application Support/Yojimbo/'
* archon:/ 70340:savepnpc: ignoring directory specification for `/Users/preston/Library/Parallels/' in
* archon:/ `/Users/preston/Library/Parallels/.nsr' - not contained within directory `/users/preston/Library/Parallels/'

It seems to be a spurious error – usually when this happens the directives are still processed.

Error 4

On some backups – usually full, I get hundreds of malloc errors in the savegroup completion – e.g.,

* archon:/ savepnpc(668,0xa0a01500) malloc: *** error for object 0x20: pointer being freed was not allocated
* archon:/ *** set a breakpoint in malloc_error_break to debug
* archon:/ savepnpc(668,0xa0a01500) malloc: *** error for object 0x20: pointer being freed was not allocated
* archon:/ *** set a breakpoint in malloc_error_break to debug

What I’m doing

I’ve currently got a question case open with EMC asking when we’ll get official support for Snow Leopard. I’ll update this blog with details when I can.

[Update] There are existing escalations to get Snow Leopard support. The current tentative schedule, I’m told, is for support in NetWorker 7.6 SP1. There’s apparently escalations against 7.5.x as well – personally, if I were a betting person, I’d be betting we’ll more likely get support in just 7.6 via SP1 rather than both 7.6 and the 7.5 tree.

NetWorker 7.6 and Mac OS X 10.6 (Snow Leopard)

 NetWorker  Comments Off on NetWorker 7.6 and Mac OS X 10.6 (Snow Leopard)
Nov 232009

While still not appearing on the NetWorker compatibility lists, it would certainly appear from testing that NetWorker 7.6 plays a whole lot nicer with Mac OS X 10.6 (Snow Leopard) better than its predecessors did.

Previously when Snow Leopard came out, I posted that NetWorker 7.5 was able to work with it, but did qualify that if you were backing up laptops or other hosts that periodically changed location, NetWorker would get itself into a sufficient knot that it would become necessary to reinstall and/or reboot in order to get a successful backup. (In actual fact, it was rare that a reboot would be sufficient.) Fixed-point machines – e.g., servers and desktops – typically did not experience this problem.

It would seem that 7.6 handles location changes considerably better.

If you’re experiencing intermittent problems with NetWorker 7.5 not wanting to work properly on Mac OS X 10.6 hosts, I’d suggest upgrading to NetWorker 7.6 as a test to see whether that resolves your problems.

The usual qualifiers remain – read the release notes before doing an upgrade.

Aside – Parallels Desktop 5 for Mac OS X

 Aside  Comments Off on Aside – Parallels Desktop 5 for Mac OS X
Nov 132009

I’ve been using Parallels Desktop for the Mac for several years now – in fact, when I originally started using it, VMware were only talking about doing a desktop virtualisation product for the Mac.

That’s partly why I stay loyal to Parallels – they supported the platform sooner. The other reason is that after years of hearing tripe from VMware employees about why you couldn’t mix windows from both operating systems, Parallels went ahead and did it with Coherence. (Yes, VMware Fusion’s Unity functionality went there too, around the same time, but the amount of times I heard it wasn’t a feature they were interested in within Workstation, as an example, drove me nuts.)

So when Parallels Desktop v5 for the Mac came out, I jumped on the upgrade bandwagon. It’s given me one major positive – I can now run Solaris 10 AMD 64-bit on my Mac Pro; that was the one remaining OS I absolutely need to periodically run that I was being blocked from doing previously. After about 3 days of installing, reinstalling, downloading more recent versions of Solaris 10 AMD, I even finally managed to get networking operational within the VM too, which meant it was useful.

Other than that, Parallels v5 has been a bit of a disappointment. You see, currently for me, Coherence mode only works if I’ve got a single monitor attached to my machine. The only time that happens is when I’m using my laptop away from my desk, or when I’m traveling – and those are times I’m less likely to run VMs. Since Coherence doesn’t work for me 99% of the time, that means I can’t really try out the Crystal mode – though I’ll admit, I’m unlikely to use it heavily; I dislike the “shared apps” approach offered by Parallels, and the new Crystal view mode seems to rely heavily on this.

It does continue to annoy me that I don’t have the option of virtually turning the monitor off – why can’t I close the console for a running VM? Surely that’s not so huge a thing that it can only be limited to enterprise class virtualisation – aka ESX, VMware Server or Parallels Server. When you have 10+ VMs running at once, even minimised all those consoles start to get annoying.

While overall I’m pretty happy with Parallels performance in terms of memory and CPU, recent support cases have highlighted to me that when it comes to translated IO, Parallels struggles – it seems to peak at about 20MB/s per VM, regardless of the throughput capabilities of the attached device. I first noticed that under v4, and was unhappy to see no change in v5.

If you’re currently a Parallels Desktop for Mac user, and you haven’t yet upgraded, I’d suggest holding off until the next build of v5, rather than jumping into the initial build released. Hopefully by then they’ll at least have Coherence reliably working again.

[Edit, 2009-11-14]

I’d like to take back my comments about Parallels 5 still having mediocre IO performance. I just realised, a short while ago, that one of the VMs I’d been testing with had failed in its VMware Tools update. Now, with both VMs in this particular test config updated to Parallels Tools v5, instead of getting 14-17MB/s transfer speed between them as I’d been getting under Paralles v4, I’m now getting 45-57MB/s. Now that’s a performance improvement.

NetWorker and Snow Leopard, Redux

 NetWorker  Comments Off on NetWorker and Snow Leopard, Redux
Sep 182009

When Snow Leopard first came out, I was reasonably impressed with how easily NetWorker continued to operate with it – and for desktop users and administrators of fixed-location servers, that should remain the case.

For laptop users though, it’s turning out to be slightly different story. My ongoing experience now is that if I switch locations repeatedly (e.g., home to work, work to customer site, customer site to work), the NetWorker client daemons eventually get so bogged down that it’s necessary to reboot to get back to working backups. In fact, a couple of times I’ve needed to go so far as to reinstall NetWorker on my laptop in order to get it running again smoothly. (That’s using NetWorker 7.5.1.)

If you’ve got mobile users upgraded to Snow Leopard who are now experiencing backup problems, a reboot (unfortunately) may be your first point of call – the nature of the daemon hang-up seems to prevent proper process shutdown, which in turn prevents the daemons from properly restarting. If the reboot fails, a client reinstall should fix it.

From my experience so far, it seems to only happen when locations are changed multiple times.

Aug 282009

I was rather pleased this morning to have a friendly FedEx courier drop off Mac OS X 10.6 – Snow Leopard.

Of course, my first thought was “will this work with NetWorker?”

I’m pleased to report – yes, yes it does. All of the following worked for me using NetWorker 7.5.1:

  • Recoveries from prior to the upgrade
  • Backups following the upgrade
  • Recoveries from new backups only
  • Recoveries that mixed old backups and new backups

(That’s just standard filesystem recoveries – I don’t use NetWorker for complete disaster recovery of Mac OS X as I feel that Time Machine is a much more appropriate system for those styles of recoveries.)

So, now we just need to wait for the software compatibility guide to be updated…

Note: The installer is very different under Snow Leopard, and no longer supports the old “Archive and Install” option; thus, when you finish the installation, there’s no need to actually re-install NetWorker; it remains there, and working, from the previous installation.

[Edit, 2009-09-04]

I can report one bit of odd experience with NetWorker under Snow Leopard. I was presenting a training course earlier in the week and wanted to change which hosts could backup the NetWorker client on my laptop. After editing the /nsr/res/servers file I found that the NetWorker client processes wouldn’t properly restart on the laptop. In the end, I did a new install of NetWorker onto the laptop, which fixed the problem. FYI, in case you notice similar things, reinstalling seems to fix the problem.

Wishlist – Server/Storage Node for Mac OS X

 Architecture, NetWorker  Comments Off on Wishlist – Server/Storage Node for Mac OS X
Aug 132009

For some time I’ve wished NetWorker would support both storage node and server functions on Mac OS X. When I had a PPC 17″ PowerBook, this mostly came from the glacially slow performance of running Linux within VirtualPC so as to run up a NetWorker server for testing. (Windows-within-Virtual PC was a dead-loss: the then-current version of NetWorker would not even start within VirtualPC.)

Since Apple made the jump to Intel machines, running a NetWorker server for lab work within a virtual machine has been far more efficient, given that now it’s just virtualisation rather than emulation. However, I’ve been thinking for a while that given the performance options available on Mac OS X, and the amount of data frequently stored on Mac OS X machines, not supporting at least a storage node is foolish.

Now that I have a Mac Pro, my personal belief is that it’s crazy not to support Mac OS X both as server and storage node.

Why, you may ask, would I think this? Is it just some weird combination of the “Mac Fan Boy” and “NetWorker Fan Boy” that I want them joined at the hip like some bizarre Doctor Frankenstein experiment?

[Here’s an aside. Why is it that people who defend Apple, and Macs, are immediately declared to be Apple Fan Boys, when PC/Windows users just as vehemently defend their own platforms declare themselves ‘realistic’? There’s only one answer: sad hypocrisy. Defending one platform is “hysterial frothing at the mouth buy-in to the reality distortion effect of Steve Jobs”, whereas equally defending another platform is “logical”. Please, spare me.]

So, jumping off that little soap box, I do actually have a method to my madness here. I honestly think, bang for buck, that the Mac Pro (using Apple’s high end machines as a reference point) represent the sort of significant processing and expansion capability often sought in backup servers. I snapped up a bargain previous generation Mac Pro that features that Intel Xeon 5400 CPUs rather than the current top-of-the-line Nehalem based processors, but this machine has serious processing power. The reason it’s called a workstation by Apple is because of it’s ability to handle complex graphics – but in reality it’s basically a server in a nice shell. With 8 x 3.2GHz cores and (currently) 12GB of RAM, this is a machine that just absolutely flies at data throughput. With expansion of up to 32GB of RAM, Mac Pros represent in one shiny shell more than enough processing power to run a backup server/storage node for any sized business*.

For companies that are space-conscious, there’s the “server” version of the Mac Pro, the Xserve, which is quite a powerful host in a 1RU enclosure.

Given the client software has already been ported to Mac OS X, the hard work has effectively already been done; server and storage node options are not going to take a significant amount of development effort.

Is there justification in porting server and storage node to Mac OS X? The cynical part of me wants to answer that there’s a hell of a lot better justification in porting server/storage node to Mac OS X than there was in porting the client to Linux PPC, but undoubtedly that would have been done to service some large-scale deal for EMC – i.e., there would have been significant business-incentive to do so.

Is there a business incentive for supporting more than client capabilities on Mac OS X? Well, market share is continuing to grow, as evidenced by Microsoft breaking what is almost universally acknowledged as the golden rule of advertising**. Then there’s the high frequency of use of Mac OS X systems in academia. This may not seem a compelling business case for EMC now, but let’s think a little longer-term – as more and more people become exposed to Macs again during their education (either secondary or tertiary), that exposure is going to influence them in their buying decisions as they move into employment. I.e., short of some catastrophic collapse***, Apple is going to see market share continue to increase – not flatten, not drop, but continue to increase.

In the short term though, another compelling reason is where Apple’s market share is at its highest – multimedia: graphic design, advertising, etc., all feature large amounts of data storage. While there’s some support for client software within Mac OS X, the backup server market in that arena is owned almost exclusively by Retrospect. (Retrospect is a good product, but it is still reasonably limited – definitely a workgroup, rather than an enterprise product.) In short, it seems mad that machines that routinely store tens or more terabytes of storage are denied storage node/dedicated storage node capabilities.

Now, some might argue that the lack of support for Sybase DBAnywhere (which powers GST, the back-end to NMC) would be sufficient cause to stop at the client (or at most, the storage node); after all, if you can’t run the GST/NMC server on the backup server, what’s the point? I have two (I believe valid) responses to this: first, it’s reasonably common to see separation between NMC/GST services and the NetWorker server, not only in environments that have multiple backup servers, but also just for reducing the potential for one service impacting the other. Secondly, there’s already examples of NetWorker server platforms that don’t have an accompanying NMC/GST server option – Solaris/AMD springs to mind immediately, and I know there’s other examples as well.

I do honestly think the point is rapidly approaching (if it is not already here) where there are more compelling reasons to port NetWorker server and storage node to Mac OS X than there are for not doing so. Architecturally, data storage volumes, increasing market share and an existing client all point to this having solid reasons.

* Note that I’m not saying that they are capable of being the sole backup server for a massive company; just like any other platform, in larger environments, the three-tier approach is always required.

** That rule is that the number one company in an industry should never refer to the number two company in the industry in their advertising.

*** With a market cap that now periodically bounces above Google’s, this seems somewhat unlikely now. (Apple’s market cap now exceeds the combined market cap of HP and Dell.)