The always astute Martin Glassborow has an article on his blog titled “This Changes Everything … Honest!” In it, Martin bemoans vendors who push technologies as miraculous solutions – the notion that if you’ve got a problem, all you need to do is buy a widget and bingo! the problem is gone. The meat of Martin’s argument is as follows:

Actually if you try to do this without a huge technology change; you may reap more benefits. The problem with large technology changes is that you often fail to do anything with the legacy tail; you simply deploy new stuff onto the new technology. The legacy just sits and moulders, costing money to maintain and manage. Of course you could simply just decide not to maintain the legacy, which will save you money in the short-term but when it breaks, you could find yourself spend even more money trying to fix a system which no-one really understands. If you try to change the process before rolling in the new technology, you will probably stand a greater chance of dealing with the legacy tail.

So don’t expect miracles from a change in technology!

Without process change, technology changes are probably just cost!

People Trump Processes Trump Technology

It’s the final sentence that I find most important in this – “People Trump Processes Trump Technology”; i.e., people trump processes, and processes trump technology.

It’s like rock, paper, scissors, but with a trump card.

In my book, I spend an entire chapter talking about the human and technical layers within a backup solution, and so Martin’s argument struck very true with me: it doesn’t matter how excellent the purchased technology is – if you don’t have people and processes established to integrate with the technology in a synergistic function, you won’t get anywhere.

Or to put it another way, there is no silver bullet. There’s no one piece of technology that you can deploy which will magically solve your issues if you can’t integrate it properly within your operating processes. In fact, I’d suggest that technology at most only ever forms 40% of the total solution – and that’s an absolute maximum percentage. The rest comes from people, and processes.

This is why so many large scale technology purchases fail to actually achieve anything within organisations. Companies that pursue an aggressive technology change strategy without addressing core issues – personnel and processes – repeatedly fail to achieve anything, and lurch from one silver bullet to the next, cursing each failed solution. This isn’t always their fault – it’s easy to believe a slick salesperson who suggests that their technology will resolve all your issues when you’re reluctant to face those core concerns.

If you want to take some honest advice from someone who has worked in the systems integration space for a decade, consider this: there are no silver bullets. You can’t solve a problem just by buying a specific piece of technology, and anyone trying to tell you this is either lying or misguided. Instead, if someone is pitching a piece of technology, or a suite of technologies, as able to solve your problems, and they don’t understand your business, they can’t be believed.

The important question is: do you understand your business? You need to – you have to, in order to understand what part people and processes need to play in solving any issue you’re currently having. Remember: they’re 60% of the solution. The technology is certainly the easy part of the solution, but it’s not a silver bullet.

 

On my personal blog, where I normally babble on about anything that strikes my interest at any given time, I recently wrote a piece about the coming internet censorship storm in Australia. If you’re not aware, it would seem that the Federal government is only weeks away from introducing into parliament a law mandating internet censorship for all in Australia. This is dangerous and repressive, and primarily uses the specious argument of “protecting the children”.

Americans may have at least had a vigorous debate about the notion of universal health care recently, but Australia is struggling to get any real debate about internet censorship going for one key reason: those leading the charge against the censorship are focusing too much on the individuals at the heart of the debate, and not the points of the debate, as discussed in this Search Networking article here. As a result, the real points of the debate are often going unheard.

As I’m a member of a minority group in Australia with rights not yet equalling those of the majority, and where those rights have been only starting to come anywhere close to parity in the last 5 or so years, I’m well aware of the danger of a government secretly censoring the topics it currently considers to be inappropriate for its population. Or to be blunt, as I write in the piece:

Had this internet censorship bill been introduced even 20 years ago, I most likely would still be largely living in fear, and laws would still vastly discriminate against me and others like me.

If you don’t live in Australia and think this bill doesn’t matter to you, you could very well be wrong. Many western democracies tend to look at what governments in different countries get away with and use those actions as precedents. If you want to read what I had to say on this topic in full, or mistakenly believe that mandatory internet censorship is probably a good idea, then please jump across to “An argument against internet censorship“.

 

In the debut episode of Doctor Who for Matt Smith (“The Eleventh Hour”), the Doctor, having only just regenerated is faced with needing to work out what sort of food suits him. He churns through apples, yoghurt (to great comedic effect), sandwiches, bacon (much to the amusement of a young Scottish girl sourcing each demand) before settling on fish fingers dipped in custard.

The day to day operations of a backup administrator are fairly typical – diagnose any overnight issues, manage media and systems capacity, facilitate recoveries, add any new hosts, and ensure that maintenance/reporting functions are still in place. That’s the apples and the yoghurt, the sandwiches and the bacon of normal operations.

As I’ve said in the past though, enterprise backup is one of those rare domains where it not only touches on almost all other areas of IT, but is impacted by all other areas of IT. You don’t have the luxury of ignoring storage, or networking, or at least a basic understanding of all operating systems and databases being backed up – you have to be jack of all trades and master of some.

Because of that, backup administrators must always be ready for fishy custard. What’s fishy custard? That’s the unexpected, the completely novel situation – when your manager comes out of a meeting saying that you need to cater for another 20% of clients by the end of the weekend, or that you need to come up with a strategy for backing up a ten terabyte MySQL database running on SCO Unix connected by an ISDN line via a double-NAT firewall. Or when as a result of automated overnight patching, suddenly half of the clients failed their backups overnight.

We can’t literally prepare for fishy custard in that as much as anything it’s about the unknown and the completely unexpected. A lecturer I had at Uni once described a rather OCD colleague when he taught in the UK who carried around a large book with him. In that book he recorded contingency plans for all manner of emergency situations – such as what he might do if a meteorite struck in a field near to where he was riding a bike and lit a grassfire, etc. That’s not how you prepare for the unexpected. It’s how you get an odd-ball reputation.

Preparing for the unexpected (the fishy custard) is about entering each day willing to accept that the daily routine might have to be thrown out the window. Among the worst backup sites I’ve dealt with are those where the daily routine is maintained with such rigidity by the administrators and the managers that new and unexpected requirements get pushed out to “sometime later”, where “sometime later” equals “an entirely inappropriate time”. A balance has to be struck, and it has to be accepted that unexpected things will happen in the backup realm, regardless of whether you want them to or not. Sometimes it’s something unexpected that you need to file away and action later, and sometimes it’s something unexpected that you need to drop everything and action straight away.

So remember, like The Doctor, you need to start each day prepared for the possibility of fishy custard. It’ll make your job less stressful, and your backups more successful.

 

The top article(s) for March related to the highly successful NetWorker Usage survey run. With over 200 respondents, this survey asked questions covering:

  • NetWorker server operating system types and versions
  • Storage node operating system types and versions
  • Number of clients backed up
  • Client operating system types
  • Modules used
  • Open source databases in use

The results were quite interesting; with over 65% of respondents using open source databases in their environments, it perhaps most a clear message to EMC that support for some form of backup integration with these modules are important, something I was lucky enough to be able to speak directly to product management about last week as well.

It also spoke clearly to EMC’s competitors that FUD regarding NetWorker’s place in enterprise backup is just that. With a high percentage of respondents reporting using NetWorker to backup environments of 500+ clients, NetWorker is not (nor was it ever) a “small” backup product. It’s there in major enterprises ensuring that they stay up and running.

You can download the survey report from here.

 

There’s been some discussion in the EMC Community Forum lately as to whether it would be handy to have a directive preview option in NMC. I’d suggest it is, but if you need this now then you don’t have anything to wait for, so long as you’re willing to work on the command line.

The way you do this is via the save command, invoking with the switches “-n” and “-v”. Of course, these can be used in the same argument (-nv or if you prefer, -vn). The “-n” option tells NetWorker not to actually perform the backup, while the “-v” option tells NetWorker to be verbose as it walks the filesystem. This verbosity will tell you what application specific module (ASM) NetWorker is going to invoke against each file/directory encountered.

For instance, let’s consider a save against the /root folder on a backup server where I’ve got a directive that’s configured to skip everything in there. The start of the output looks like the following:

[root@tara ~]# save -b Default -nv /root
3817:save: Using tara.pmdg.lab as server
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /dev/
70342:save: save: got prototype for /
70342:save: save: got prototype for /dev/
70342:save: save: got prototype for /
70342:save: save: got prototype for /opt/
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /
70342:save: save: got prototype for /media/
70342:save: save: got prototype for /proc/sys/fs/
70342:save: save: got prototype for /var/lib/nfs/
chdir(/root)
Name=`/root/', name=`/root/', fname=`./'
32240:save: found protofile spec for /:
mntasm : backup4
70342:save: save: got prototype for /
66135:save: NSR directive file (/.nsr) parsed
70342:save: save: got prototype for /root/
66135:save: NSR directive file (/root/.nsr) parsed
walk(/root/, ./)
walk(/root/.nsr, .nsr)
uasm -s /root/.nsr
walk(/root/.odbc.ini, .odbc.ini)
uasm -s /root/.odbc.ini
walk(/root/backup.log, backup.log)
32246:save: matched internal `skip' on `backup.log' for `/root/backup.log'
walk(/root/micromanual-nsradmin, micromanual-nsradmin)
32246:save: matched internal `skip' on `micromanual-nsradmin' for `/root/micromanual-nsradmin'
walk(/root/lose-files.sh, lose-files.sh)
32246:save: matched internal `skip' on `licenses.sh' for `/root/lose-files.sh'

As you can see by the above, we get a file-by-file analysis of the walk process, and that includes the ASM that will be invoked against that file. For instance, looking at the ‘backup.log’ file, we see:

walk(/root/backup.log, backup.log)
32246:save: matched internal `skip’ on `backup.log’ for `/root/backup.log’

So the first line tells us that it’s encountered a file called ‘backup.log’, with a full path of ‘/root/backup.log’. The second line tells us that it’s matched the (internal) ASM, ‘skip’, on the file ‘backup.log’ for the path ‘/root/backup.log’ – i.e., it’s effectively telling us that it will skip that file.

While it may be messy, it is a valid and usable technique to see what your directives are going to do.

 

There’s been much speculation as to whether Sun under Oracle would retain the EMC NetWorker OEM arrangement.

Finally there’s some details on Sun’s website under the banner “Sun and EMC“. In it, they state:

Sun will continue to OEM the EMC NetWorker software for backup and recovery which enables Sun to continue offering the EMC software as Sun StorageTek Enterprise Backup Software.

So, business as usual?

Unfortunately it looks like “yes”. Don’t get me wrong, I teethed on Solstice Backup, as it was called then – in fact I used Solstice Backup for 4 years before I even installed NetWorker as a non-OEM product.

Here’s the rub: Sun have been woeful at (a) supporting “NetWorker” in the rebadged form, and (b) providing patches in a timely manner. Again and again, I get complaints from Sun OEM customers that Sun takes ages to update their releases in sync with EMC’s releases. I also hear frequent tales of OEM NetWorker support cases with Sun that take forever. Both of these factors well and truly gel with my experience as a Sun customer in the late 90′s, and I’m still hearing the same stories in the hear and now.

Disclaimer: I sell and support EMC NetWorker native. That should have been obvious but I don’t want to be accused of hiding this.

I didn’t think Sun’s statement went far enough. I don’t want to hear that they’re going to continue to OEM NetWorker, I want to hear that they’re going to OEM NetWorker and pick up their game. Release cycles should be much closer tied to NetWorker, support needs to be considerably improved, and patches need to come out sooner to OEM NetWorker as EMC actually release.

If they’re not, then when you factor in the changes that Oracle are making to Solaris OS licensing, I’m expecting that the reasons for people remaining with the OEM version of NetWorker to shrink considerably.

 

A short while ago I had a chat with some senior EMC folks about NetWorker 7.6 Service Pack 1.

Obviously being an NDA partner I can’t talk about what’s coming.

I can share my opinion though: this is going to be a big, and very important release, and it’s going to be well worth the wait for a lot of users. I can’t be drawn into discussion over what’s in it, but I do know I’ll need to do a lot of blog articles about features once it’s released!

(I feel compelled given the date I’m publishing this piece: No, this isn’t April Fools, and nor were EMC playing an April Fools on me.)

© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha