Basics – dbgcommand

Note: Please understand I’m not suggesting you play around with this feature randomly or without good reason. You shouldn’t make use of this unless you’re advised to by support – or unless you really, really know what you’re doing.

In the past if you’ve had a fairly serious issue with NetWorker you’ll probably have been advised to temporarily stop NetWorker, and restart it in debug mode. For example, support may have asked you to run, say:

# nsrd -D9

Which launches the main NetWorker server process (the “master” process) in debug mode level 9. There’s 99 levels of debug, ranging from 1 to 99, with the special level of “0” being “off”. Debug mode level 1 is not really all that interesting; debug model level 3 can be useful sometimes for tracing name resolution issues. Level 9 produces fairly interesting details for people in support roles – I frequently make use of it when trying to debug issues customers may be encountering. (After that, the levels start to get less useful to anyone except for EMC engineers, and level 99 falls into the what, are you insane? category.)

There’s a problem with this strategy of stopping NetWorker and rerunning it though, as you may have already guessed – sometimes stopping and starting effectively kills off the problem you’re having. What is needed is the ability to put NetWorker into debug mode on the fly.

To address this, there’s been a support tool floating around for a while that EMC support would occasionally pass on to customers to help out with enabling debug mode without having to restart NetWorker. It’s called dbgcommand. However, dbgcommand wasn’t provided by default, and was typically only handed out either (a) when it was explicitly requested or (b) when engineering asked support to get the customer to run it.

As you can imagine, this has been a tad frustrating for just about everyone involved. Actually, I’ll rephrase that this has been quite frustrating for everyone involved.

Thanks in part to various people (e.g., my colleagues and I) continuing to ask for it, the pseudo-magical handing out of dbgcommand has been dropped, and it’s now included in NetWorker as of 7.5. [Note: I think it’s also in 7.4.4…]

So, how do you use it?

Surprisingly easily:

# dbgcommand -p pid command

Where pid is the process ID of the NetWorker daemon you want to interact with.

Typically your usage would be say:

# dbgcommand -p pid Debug=X

Where X is a value between 0 and 99, or perhaps:

# dbgcommand -p pid FlushDnsCache

For example, if the NetWorker server process, /usr/sbin/nsrd, were running with a PID of 23278, you could put it in debug mode level 9 without a restart by typing the command:

# dbgcommand -p 23278 Debug=9

When you adjust a processes debug level, it gets logged in the appropriate daemon log – for example:

0 1234248435 2 0 0 3051682432 23277 0 nox nsrd 2 %s 1 0 54 Changing Debug level of process (id 23278) from 0 to 9
0 1234248446 2 0 0 3051682432 23277 0 nox nsrd 2 %s 1 0 54 Changing Debug level of process (id 23278) from 9 to 0

Or:

0 1234249625 2 0 0 3051682432 23277 0 nox nsrd 2 %s 1 0 56 Changing FlushDnsCache of process (id 23278) from 0 to 1

So there you have it – no longer do you have to stop and restart NetWorker to run in debug mode, you can alter the debug level of whatever NetWorker process you need to on the fly with minimum fuss.