A very blue upgrade

I recently encountered one of those has-to-be-environmental errors that turned out to be a NetWorker issue. You know the one: the situation being explained is so straight-forward that there’s no chance NetWorker could be the actual culprit.

Then you investigate.

Then you investigate a little more.

Then you blink slowly and realise it really is a NetWorker error.

The case in point was a customer with a VMware environment who reported that simply trying to upgrade NetWorker would result in servers irretrievably blue-screening to the point where it was necessary to recover from an image-level backup. That there were other odd issues happening in the environment was obvious: the NetWorker server would at times lose access to its own nsrdb directory, locked out with permissions errors. Virtual machines reported corrupt sectors on disk, and then there was the blue-screening upgrade.

But after testing against clean images in the customer environment and then setting up a test environment in my lab, this was something wrong with NetWorker.

Here’s the scenario:

• Windows 2008 R2 server, patched to current patch levels.
• Install NetWorker 8.1.0.0 (i.e., 8.1 vanilla release).
• Install NMM 3.0.0 Build 282.
• (Use the machine for a while – or don’t)
• Uninstall NMM 3.0.0 Build 282 in order to install 3.0.1 Build 280.
• (Reboot)
• While you’re at it, Upgrade NetWorker to 8.1.1.[2,3,4].
• Install NMM 3.0.1 Build 280, which triggers a reboot
• Server bluescreens – STOP: 0x0000007a error.

In actual fact, you don’t even need to install NMM 3.0.1 – the problem is caused by the upgrade of 8.1.0.0 to a newer 8.1.x.y release.

My guess (and that of a colleague, too), was that this has something to do with the Block Level Backup option in 8.1. In the original 8.1 release, enabling the BLB option triggered a reboot requirement. Under newer releases this doesn’t seem to be the case.

The solution turned out to be reasonably straight forward once the culprit was identified: uninstall 8.1.0.0 first (leaving metadata in place), and then install the newer 8.1.x.y release.

The problem seems to be limited specifically to those hosts running the original 8.1 release … subsequent upgrades of the client software don’t trigger the issue.

If you installed the base 8.1 release on your Windows 2008 R2 servers … watch out for this one.