What I’ve learnt this year…

As the year draws to a close, I thought I’d spend a few minutes jotting down the major (technology) related lessons I’ve learnt this year. I’m not talking about training lessons, but rather, real world lessons that have real implications for my day to day work and what I’d recommend to customers (or anyone else who takes the time to listen to me).

It seems fittingly appropriate that this post is not only one of the first ones on this new site, but is also the 300th post on the blog.

So, without further introduction, here’s my key technology lessons from this year.

ext3 sucks

I’m still kicking myself that it took so long to actually get around to starting to use XFS on Linux. Since that point, I won’t go back. Regardless of whether it’s for regular filesystems or disk backup filesystems, I’m not likely to back to the ext tree, and it’s equally likely that I won’t try another Linux filesystem until btrfs comes out. I can’t believe it’s not offered by default on RedHat (and therefore CentOS).

I really like VTLs

I’ll be the first to admit that I still rail against the principle of VTLs: to be perfectly blunt, we should not need them in an ideal world. But regardless of whatever backup product you’re currently using, there are practical limitations with the way conventional disk backup is implemented. The level of “suck” will vary depending on the product, but the fact remains that we need VTLs (and I’m growing to like VTLs) not because they represent a compelling evolutionary architectural step in data protection, but because they’re frequently better than the alternative of conventional backup to disk.

When it comes to tape, logic struggles to defeat stupidity

Every year we see a plethora of stories enthusiastically predicting the death of tape. Despite all the logical arguments to the contrary, these stories just keep on coming out again, and again, and again, and again. It continues to be tiresome.

My long term avoidance of working with MySQL seems well founded

In terms of administration, MySQL seems sadly lacking in comparison to PostgreSQL. (Obviously this entry is biased in favour of PostgreSQL, but some of the arguments are compelling.) I just have to say, I really find that the MySQL way of doing things comes across as very counter-intuitive. Maybe it’s because before I used PostgreSQL, the databases I most used were Oracle and Ingres, but PostgreSQL just seems to make more sense to me. I’ll continue to use MySQL where I need to, but PostgreSQL will remain my favourite open source database.

Be prepared for slapfests when reading vendor blogs or twitter postings

You have to take the bad with the good, it seems. When they’re good (i.e., detailing their products and how they envisage the market moving forward), they’re very good. But when they’re engaged in petty point scoring slapfests with one another, it’s cringeworthy school-ground name calling at best, and downright unprofessional at worst.

There are actually merits to Twitter

This one took me completely by surprise. I had long railed against Twitter as being incomprehensibly useless. I hereby admit I was entirely wrong. I’m starting to grok that it’s a remarkably useful tool for train of thought collaboration in a way email could never achieve (particularly with verbose people like myself – impose a character limit and it forces a rethink of how to get your ideas across).

Everyone likes Drobo

For a period in November it seemed like the technical world was ablaze with lust for Drobo devices. I’m already starting to be afraid of my next electricity bill, given that “intelligent” metering is being introduced – and I do have an awful lot of drives attached to my Mac Pro. Whether I buy a Drobo or not will largely depend on whether my electricity goes up sufficiently high enough that there’s a 3 quarter cost-neutrality between the two costs 🙂

Mac Pros are about the most awesome computers you could ever possibly hope to own

No debate, no discussion, throw out every other desktop computer/tower you’ve ever had. If you want sheer performance and scaleability, do yourself the favour and go buy a Mac Pro now. Seriously, now – stop reading, and go buy one now. That’s me talking from the previous generation Mac Pro, not even the Nahelem series, which is practically twice as fast again.

(Why, just why EMC doesn’t at least port the NetWorker storage node to Mac OS X is also beyond me. My Mac Pro, only moderately specced, smashes the performance capabilities of many of my customers’ backup servers.)

People don’t always do everything I tell them to

Otherwise you’d be too busy buying a Mac Pro to read this point, hmmm? I guess I’d better come up with a few more points then…

Leopards do change their spots

Moving from Mac OS X 10.5 (Leopard) to 10.6 (Snow Leopard) was one of the biggest performance boosts I’ve had in years on computers.

A big name does not guarantee a reliable product

Think Cloud, then think Microsoft/Danger and the T-Mobile sidekick debacle.

Actually, this didn’t really take me by surprise, but it did seem to take quite a few other people by surprise, so maybe what took me by surprise was that it took others by surprise.

Apparently you can just keep rolling the same tired old turd in glitter

Think Microsoft and Live Search, oops I mean Windows Live Search, oops I mean MSN Search, oops I mean Bing. There must obviously reach a point where all you see is the glitter and suddenly it’s good (at least that’s how it seems to go in the media).

More recently, Microsoft seems to have decided that in order go get more people to use Bing, they need to come up with exclusivity arrangements for search forms on mobile devices. It would seem that Microsoft are still trying the same tired old arrangements of buying market share: the problem is that people are getting increasingly aware of this trick and will rebel rather than silently put up with it.

The problem with Microsoft is Ballmer

People may think that I’m just anti-Microsoft through and through. I used to, as well. These days I’ve determined I’m anti-Ballmer, and think the best thing that company could do would be to replace him and actually put someone in charge who can drives the company up rather than down. This was after all the decade that Microsoft missed. (Also check this article over at CNET.)

Apparently Avatar is only good because of NetApp

If you believe the incessant twittering, blogging and raving from various folks from NetApp, Avatar is the most amazing movie of all time not really because of the quality of the effects, rendering, story and acting, but because the files for Avatar were stored on NetApp fileservers. (Sorry NetApp folks, I don’t buy this. You don’t have any right to hitch your commercial wagon to this. The type of storage used shouldn’t make or break a movie so long as it’s been appropriately planned and deployed, and going on like a broken record gets a little tiresome.)

That being said, Avatar is, in my opinion, the greatest blockbuster of all time.

2009 was the year DeDupe entered mainstream

Sure, deduplication has been around for a while, but in whispers and bleeding edge implementations. I still maintain that in its current state, deduplication is mostly bleeding edge, but it’s definitely entered the mainstream. Nothing emphasised this more than the outright bidding war between EMC and NetApp for Data Domain.

Data deduplication can no longer be ignored, and given the logical challenges of having a total data lifecycle data deduplication process that keeps the data deduplicated throughout, 2010 will be the year when companies start to pick (in backup) whether they need to implement source or target deduplication. (My preference for a variety of reasons remains target based deduplication, for what it’s worth.)

Michael Dell doesn’t get the computer industry

When Apple purchased NeXT, Michael Dell was famously quoted as saying that as far as he was concerned, Apple should have been closed and the money given back to shareholders.

Perhaps, given that Apple’s market capitalisation massively exceeds that of Dell, they have almost double the assets of Dell, significantly higher net income, yet almost half the employees, Michael Dell should forget what he learnt at school and try some fresh business approaches. Unfortunately for Dell, they made their name on shipping a similar product, but being cheaper and faster. Now that their competitors have finally pulled their pants up and are able to negate those advantages, Dell has continually struggled to find true competitive advantages. In short, Dell has forgotten the crucial business lesson that everyone can imitate: in order to succeed you have to learn how stay unique.

I’m losing my tolerance for Solaris

It used to be that when customers said to me “should I deploy NetWorker on Solaris or Linux?” I’d 100% of the time recommend Solaris.

I’m struggling to reach that same conclusion any more. This isn’t due to costs, or potential issues to do with Oracle’s acquisition of Sun, but due to incredibly weird timeout issues that occur on Solaris compared to Linux.

Take for instance an issue I’ve been working on for the last several months which is now “resolved” in NetWorker 7.5.1.8 on Solaris. Well, I say “resolved” because I think it’s more that weird Solaris timeouts prevent it from happening – even though it does still happen on Linux.

The scenario is that if you’re staging from a disk backup unit on a storage node to a server, then have extremely flaky communications between the two hosts that cause NetWorker to:

• Detect the storage node nsrmmd’s have stopped responding
• Attempt to restart them
• Fail
• Timeout and attempt to restart them
• Comms are restored
• Restart nsrmmd’s

Here’s the odd thing: in this scenario it appears that from the time that the nsrmmd’s are restarted, it takes approximately 1 hour and 45 minutes for NetWorker on Solaris to finally kill the staging operation. On Linux? 3 minutes or less.

This is on top of a variety of other timeout issues – inactivity timeouts being ignored when certain network failures occur, etc.

So when someone asks me now if they should deploy NetWorker on Solaris or Linux, I’m likely to counsel fibre channel device connectivity and Linux all the way.

Coming to this conclusion having used Solaris in various forms since 1992 is greatly disappointing.

I prefer the cumulative patch cluster package release style

When EMC started offering cumulative patch clusters as whole new packages/installers rather than just issuing hot fixes, I railed against it, feeling that it wasn’t all that logical an idea. Now, I’m rather sold on it. I do however continue to think that not easily offering the cumulative patch builds within PowerLink is a tad painful.

dbgcommand

For some time, dbgcommand was a mysterious and pseudo-magical command that was whispered about and occasionally handed out by EMC engineering. Since we managed to get it included in the standard distribution, I’ve grown to absolutely love this command, and think that all NetWorker administrators should ensure they’re acquainted with it.

The RFE system works

Many would argue that EMC’s RFE system, like that of all other vendors, sucks. Certainly there remains room for improvement, but 2009 was a record year for me in getting bugs and RFEs alike in NetWorker actually fixed and integrated into new releases. I’m very impressed with the efforts that EMC has gone to in order to improve the NetWorker RFE system, and I’d encourage those who have previously given up on it to revisit it. (Alternatively, in case I’m having more luck because of this blog, contact me and convince me to make the argument for a bug fix or RFE resolution. Obviously I can’t promise anything, but something is working for me.)

You can’t use compress and encryption directives

Earlier in the year I’d blogged to say that you could. What it turned out was that NetWorker silently ignored both directives in this case, and I learnt this some time later.

I like Wizards

I don’t mean the Harry Potter kind. (Well, I like Harry Potter, but that’s not my point). I’d long scoffed at Wizards in NetWorker, but the updates in NetWorker 7.5 and Oracle Module 5 led to the first time I’ve found something easier and more efficient to do in the GUI than I normally do from the command line.

Pseudo-disorganised tools can help people anal retentive about filing

I love Yojimbo. I rarely, if ever go a day without using it. End of point.

Don’t always trust the man pages

I refer to nsr_render_log and the lack of security in its security feature (now removed in 7.6).

In summary

When I was at University, a philosophy lecturer remarked that University is the last place you learn for the sake of learning. After that, you learn for work or learn to survive. I thought at the time this sounded like sage advise. Now I think while he may have been technically accurate, he wasn’t correct. People who only learn because they need to for their job don’t really learn, they assimilate the required information and nothing more – and rarely the comprehension required to use it out of the exact reason they learnt it for.

It’s possible to learn for the sake of learning for your job as well. That’s called being enthusiastic, and being passionate about what you do.

I’d like to think that my blog, started in January this year, helped others learn for the sake of learning.