I periodically see indignant tweets and comments by people that if you sell something to a client, then you’re at worst being unethical, or at best being idiotic to say that you like to consider customer relations as partnerships.

This has reached the point where I’ll no longer sit back and listen to cynics who think that as soon as you start selling you either cease being human, or cease being unable to think symbiotically.

Insisting that companies cannot, and should not, refer to clients as partners, is at worst toxic and at best, demeaning to all parties.

Now, I’m not going to say that there are instances where some companies jump on the bandwagon and like to insinuate a partnership but stick to a traditional “stick whatever badge you need on that widget to sell it” sales approach. Of course that is going to happen.

But to tar all companies that sell, or integrators with that brush? Pah! Think again.

I’ve worked in some form of consulting pretty much all my career. I started as a trainee consultant, and when that programme was dying I transferred across to a Unix system administration team. Even as an “end customer” I still had my own customers, and as the company I was working for started taking on outsourcing contracts, I started being a consultant again. That was followed by a brief stint in the less than compatible world of finance, and since then I’ve remained in consulting.

Consult! Consult! Consult!

Consulting, systems integration, however you want to think about it, does not work well when customers are treated as meat – as paying clients to service the next bill. That leads to a succession of one-off engagements and implementations. Rape a company of budget, move on to the next and pillage that, too. It’s not a sustainable model. Or rather, unless you’re a global company and trade on some pre-established name, that model doesn’t get you very far. Pretty soon you get a crap name in the market and you start driving yourself out of business. You’ll blame the technology you’re using, and switch to another product, or another vendor, exhaust a new set of customers, and move on again.

There’s only one sustainable model in consulting and systems integration, and that’s the model where you engage with clients in a partnership. I’m not talking about looking for joint ventures; I’m talking about basic recognition of fundamental business cooperation, viz:

  • I want to help you succeed at what you do;
  • If you succeed at what you do, you’ll be able to help me succeed by buying things from me.

Symbiotic? Or parasitic? A cynic would say parasitic, and they’d be wrong. Or they’d come from the “everything should be free except for what I do” school of business. You know – the people who think that the only company entitled to put markup on a widget, or make a profit, is themselves.

It’s actually a symbiotic relationship, because it recognises that a relationship can actually be of mutual benefit to both parties. It doesn’t have to be about one “winning” and one “losing”, or “one making money” and “one spending money”.

The absolute basis of my belief in this is covered in my “13 traits of a great consultant” post. In particular, point 11 sums up exactly why a customer/client relationships should become a partnership:

Solve the problem, don’t answer the question – From an IT perspective, I use this example: an engineer, if asked a question by a customer, will do his or her utmost to answer the question as exactingly as possible. A consultant will look past the direct question and aim to solve the problem that led the customer to ask the question. Or in other words: if it doesn’t have a yes/no answer, no question is asked in isolation.

If you just have a customer/client relationship, then all you get is an engineering relationship. “Yes we can sell you widget X? What, you thought widget X did Y? But you didn’t ask? Thankyou for shopping, no refunds!” Do you really want that sort of relationship? Going down that path, you get a plethora of situations where technology is blamed for non-technical issues – and indeed, it happens at both the client and the sales side.

Form a symbiotic partnership though, and the relationship is far more wholesome and useful. From the sales side of it, satisfied customers whom you consistently deliver expected results to are repeat customers; repeat customers form the basis of predictable sales and earnings, and as time goes on provide valuable feedback to your growth as a company, too. From the client side, you get solutions that are tailored to your needs by people who you know and trust – and you know and trust them because they’re very much aware of your business requirements, constraints and operational models. A partner in fact will be able to help you through the rougher times – regardless of whether that’s unexpected staff changes without handover, or simply when needing a leaner approach that sacrifices scope only, rather than quality and scope. A partner will have the experience of working within your organisation and be able to deliver faster, more efficiently, and with less impact to your operational processes.

So, the next time someone suggests to you that you can’t have a partnership in a sales/client model, or that consultants/system integrators can’t form symbiotic relationships with your business, consider this one question:

Do you want a supplier you can trust, or a box dropper?

Rarely, if ever, will the answer be the latter.

 

One of the settings that can be made within a group is the ‘inactivity timeout’ setting. This refers to client inactivity. This is often erroneously considered to be a group timeout setting, but it’s not.

Now, to start with, the architecture of having a client inactivity timeout setting is, I believe, flawed, and should be addressed by adding heartbeat functionality between the NetWorker server and the client backup process*.

There are a plethora of situations that don’t fall into client inactivity. These include:

  • Blocking IO call failing (can happen to just about any product)
  • Saveset initiation request sent but not responded to. (This is a tricky one to define – that seems to be the point where the failure happens, but it’s almost impossible to diagnose.)
  • Backup server’s bootstrap/index:server saveset waiting for media on the backup server.

There have been various attempts to fix these situations over the years – for instance, most recently there were patches introduced into the 7.5 service pack stream to try to prevent a situation where a group would hang on startup probe. As is always the case with hanging situations, it’s difficult to say for sure whether those potential issues were well and truly dealt with.

What it remains clear though, and it’s really important to remember, is that just because you’ve set a “client inactivity” timeout within a group doesn’t guarantee that the group will timeout after a certain period of inactivity. I.e., it doesn’t excuse you from confirming on a daily basis whether your groups have finished or not.

Monitoring can be achieved a few different ways:

  • Literally checking each group that is still running in NMC at a certain point in the day and determining whether it should be running or if it is hung.
  • Paying special attention to savegroup completion reports that tell the group is “aborted, already running” (though that means missing a hung group for around 24 hours).
  • Scripting a check and alert for still-running groups – like the NMC option, but automated.

It would be great to say that there should never be a case where a group hangs and doesn’t complete, but I recognise this is one of those things that’s difficult to program, and in actual fact is almost impossible to guarantee. Could it be handled better? Undoubtedly; it’s just I’m enough of a pragmatist to know that it’s never going to be perfect.

The catch-cry of the backup administrator should be “constant vigilance!” As I’ve discussed previously in posts about enacting zero error policies, it’s not about trying to configure a “set and forget” system where there’ll never be an issue, it’s about always having your finger on the pulse and never, ever accepting that there will be regular alerts for “events-that-look-like-errors-but-you-know-they’re-not”.

So while the client inactivity timeout in a group will save you from some mundane aspects of group administration, it won’t let you ignore monitoring your groups for unexpected states.

__________
* By flawed, I mean:

Currently the backup process works as follows:

  1. Server instructs client to start backing up
  2. Client starts sending data to appropriate storage node/nsrmmd process
  3. If client fails to send any data for ‘inactivity timeout’ minutes, backup is considered to have failed, and restart is run if necessary.

This doesn’t suit situations where there’s a dense filesystem walk taking place, and in fact it really, really should work as follows:

  1. Server instructs client to start backing up
  2. Client starts sending data to appropriate storage node/nsrmmd process
  3. Every X (e.g., 90) seconds or so when no data has been sent, the storage node/nsrmmd process asks the client if the save is still running.
  4. If the client responds within X seconds, keep waiting.

That’s the sort of heartbeat mechanism that should be used…

 

Are you still backing up Novell NetWare hosts? If you are, I hope you’re actively considering what you’re going to do in relation to NetWare recoveries in March 2010, when NetWare support ceases from both Novell and EMC.

I still have a lot of customers backing up NetWare hosts, and I’m sure my customer set isn’t unique. While Novell still tries to convince customers to switch from traditional NetWare services to NetWare on OES/SLES, a lot of companies are continuing to use NetWare until “the last minute”.

The “last minute” is of course, March 2010, when standard support for NetWare finishes.

Originally, NetWare support in NetWorker was scheduled to finish in March 2009, but partners and customers managed to convince EMC to extend the support to March 2010, to match Symantec and co-terminate with Novell’s end of standard support for NetWare as well.

Now it’s time we start considering what happens when that support finishes. Namely:

  1. How will you recover long term NetWare backups?
  2. How will you still run NetWare systems?
  3. How will you manage NetWorker upgrades?

These are all fairly important questions. While we’re hopeful we might get some options for recovering NetWare backups on OES systems (i.e., pseudo cross-platform recoveries), there’s obviously no guarantees of that as yet.

So the question is – if you’re still using NetWare, how do you go about guaranteeing you can recover NetWare backups once NetWare has been phased out of existence?

The initial recommendation from Novell on this topic is: keep a NetWare box around.

I think this is a short-sighted recommendation on their part, and shows that they haven’t properly managed (internally) the transition from traditional NetWare to NetWare on OES/SLES. This is perhaps why there isn’t a 100% transition from one NetWare platform to the other. Being faced with unpalatable transition options, some Novell customers are instead considering alternate transitionary options.

Unfortunately, in the short term, I don’t see there being many options. I’m therefore inclined to recommend that:

  1. Companies backing up traditional NetWare who only need to continue to recover a very small number of backups consider performing an old-school migration – recover the data to a host, and backup on an operating system that will continue to enjoy OS vendor and EMC support moving forward.
  2. Companies backing up larger amounts of traditional NetWare should consider virtualising at least one, preferably a few more NetWare systems before end of support, and keeping good archival VM backups (to avoid having to do a reinstall), using those systems as recovery points for older NetWare data.

The longer-term concern is that the NetWare client in NetWorker has always been … interesting. Once NetWare support vanishes, the primary consideration for newer versions of NetWorker will be whether those newer versions actually support the old 7.2 NetWare client for recovery purposes.

With this in mind, it will become even more important to carefully review release notes and conduct test upgrades when new releases of NetWorker come out to confirm whether newer versions of the server software actually support communicating with the increasingly older NetWare client until such time as recovery from those NetWare backups is no longer required.

You may think this is a bit extreme, but bear in mind we don’t often see entire operating systems get phased out of existence, so it’s not a common problem. To be sure, individual iterations or releases may drop out of support (e.g., Solaris 6), but the entire operating system platform (e.g., Solaris, or even more generally, Unix) tends to stay in some level of support. In fact, the last time I think I recall an entire OS platform slipping out of NetWorker support was Banyan Vines, and the last client version released for that was 3 point something. (Data General Unix (DGUX) may have ceased being supported more recently, but overall the Unix platform has remained in support.)

If you’re still backing up NetWare servers and you’re not yet considering how you’re going to recover NetWare backups post March 2010, it’s time to give serious consideration to it.

 

For what it’s worth, I believe that the continuing lack of support for renaming clients as a function within NMC, (as opposed to the current, highly manual process), represents an annoying and non-trivial gap in functionality that causes administrators headaches and undue work.

For me, this was highlighted most recently when a customer of mine needed to shift their primary domain, and all clients had been created using the fully qualified domain name. All 500 clients. Not 5, not 50, but 500.

The current mechanisms for renaming clients may be “OK” if you only rename one client a year, but more and more often I’m seeing sites renaming up to 5 clients a year as a regular course of action. If most of my customers are doing it, surely they’re not unique.

Renaming clients in NetWorker is a pain. And I don’t mean a “oops I just trod on a pin” style pain, but a “oh no, I just impaled my foot on a 6 inch rusty nail” style pain. It typically involves:

  • Taking care to note client ID
  • Recording the client configuration for all instances of the client
  • Deleting all instances of the client
  • Rename the index directory
  • Recreate all instances of the client, being sure on first instance creation to include the original client ID

(If the client is explicitly named in pool resources, they have to be updated as well, first clearing the client from those pools and then re-adding the newly “renamed” client.)

This is not fun stuff. Further, the chance for human error in the above list is substantial, and when we’re playing with indices, human error can result in situations where it becomes very problematic to either facilitate restores or ensure that backup dependencies have appropriate continuity.

Now, I know that facilitating a client rename from within a GUI isn’t easy, particularly since the NMC server may not be on the same host as the NetWorker server. There’s a bunch of (potential pool changes), client resource changes, filesystem changes and the need to put in appropriate rollback code so that if the system aborts half-way through it can revert at least to the old client name.

As I’ve argued in the past though, just because something isn’t easy doesn’t mean it shouldn’t be done.

 

A few days ago a customer was having a rather odd problem. They’re currently running NetWorker 7.3.3 and getting ready to jump directly to NetWorker 7.5.1, but to do so they wanted to first run up a NetWorker 7.5.1 server and confirm current client types, databases, etc., will backup without issue*.

So the customer installed NetWorker 7.5.1 on a new Linux host, created some devices and pools, but then encountered a particularly odd problem when they went to create the clients. NMC would allow them to fill in all the properties for the client, but when they clicked OK in the new client dialog box, nothing would happen. No errors were produced, but nor were any clients actually created.

When they raised this with me I was a little puzzled for a few minutes, then asked if they were using the NMC that comes with NetWorker 7.5.1, or the NMC that comes with 7.3.3 and had just added the new server to the control zone.

The answer was that the control zone for the existing NMC that came with NetWorker 7.3.3 was just extended to include the 7.5.1 server.

For pools, devices and groups this was not a problem – these were all successfully created on the 7.5.1 server using the 7.3.3 NMC. However, when it came to clients, it wouldn’t work.

The reason is quite simple – as new features and functions are added to NetWorker over time, different fields within a configuration resource may or may not become mandatory. Some of the time this is obvious, because we’re required to fill in certain fields – e.g., client names, schedules, etc. However, in other instances, NetWorker has predefined defaults that it slots into place if a value isn’t entered – e.g., parallelism, priority, browse/retention time, etc. Just because defaults are put into place however doesn’t mean that fields are any less mandatory – it’s all about allowing you to create resources quicker.

So, what’s all this got to do with differing NMC/NSR versions? In short, everything!

You see, what happened for this customer is that between NetWorker 7.3 and 7.5, there has been a raft of client based functionality added – e.g., data deduplication, support for defining a client as being virtual, etc.

Undoubtedly some of these new features have mandatory values – so that if the server is probing details for the clients, it can safely request say, dedupe status or virtual status without worrying about getting an (undefined!) style response. Each version of NetWorker is “aware”, via base configuration, what fields must be supplied when creating a new resource, and thus, the scenario for this customer would have been:

  1. Fill in client properties in NMC 7.3.3
  2. Attempt “client create” 7.3.3 -> 7.5.1.
  3. The 7.5.1 server reviews the proposed client resource and,
  4. The 7.5.1 server rejects the proposed client resource as not having all the mandatory fields filled in.

Should NetWorker/NMC have provided an error to explain what was going wrong? Undoubtedly that would have been good, and I’d suggest that NMC/NSR should be able to better communicate resource creation/update failure in these circumstances. However, that being said, the fundamental problem remained the same – the version of NMC in use couldn’t create new clients because it wasn’t supplying all the mandatory details to the more recent version of NetWorker.

In many small sites, the NMC server and the NetWorker server are on the same host, and are thus upgraded in lock-step. However, for sites where the NMC server is installed on another host, this is a valuable lesson – unless you have a very valid reason, don’t run a version of NMC that matches an earlier version of NetWorker than the current server version. It may work (mostly), but if it does fail, it’s unlikely to be immediately obvious why it’s failing.


* This is what I’d call an excellent upgrade policy – you can read the release notes until they’re 100% memorised, but nothing quite beats actually running up your own test server.

 

Introduced in the 7.4 series of NetWorker was an option for the client resources called “Scheduled backup”. Normally set to “Enabled”, this can be changed to “Disabled” to allow an administrator to leave the client in all its current groups, but to not be backed up.

The purpose of course is that if a client is going to be temporarily unavailable for a few days (e.g., maintenance operations, etc.), an administrator can turn the backups off for that client without having to later remember what groups it has to go back into.

This is a really good idea, with one exception. Humans forget.

What this option needs is an automated reset function. That is, when an administrator turns off scheduled backups, the administrator should also be able to set a date at which point NetWorker will automatically re-enable scheduled backups for the client. NetWorker should, on a daily basis, scan all clients that have scheduled backups disabled to see whether their “reset” date is up, at which point it should automatically re-enable the backups for the client without any further administrator intervention.

Seeing this added to NetWorker would be awesome.

© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha