Debugging nsrmmd

 Linux, NetWorker, Scripting  1 Response »
Oct 282010
 

Sometimes it’s helpful to run NetWorker in debug mode – but sometimes, you just want to throw the nsrmmd processes into debug mode, and depending on your site, there may be a lot of them.

So, I finally got around to writing a “script” to throw all nsrmmd processes into debug mode. It hardly warrants being a script, but it may be helpful to others. Of course, this is Unix only – I’ll leave it as an exercise to the reader to generate the equivalent Windows script.

The entire script is as follows:

#!/bin/sh

PLATFORM=`uname`

if [ "$PLATFORM" = "Linux" ]
then
	PROCLIST=`ps -C nsrmmd -o pid | grep -v PID`
elif [ "$PLATFORM" = "SunOS" ]
then
	PROCLIST=`ps -ea -o pid,comm | grep 'nsrmmd$' | awk '{print $1}'`
fi

DBG=$1

for pid in $PROCLIST
do
	echo dbgcommand -p $pid Debug=$DBG
	dbgcommand -p $pid Debug=$DBG
done

The above is applicable only to Solaris and Linux so far – I’ve not customised for say, HPUX or AIX simply because I don’t have either of those platforms hanging around in my lab. To invoke, you’d simply run:

# dbgnsrmmd.sh level

Where level is a number between 0 (for off) and 99 (for … “are you insane???”). Running it on one of my lab servers, it works as follows:

[root@nox bin]# dbgnsrmmd.sh 9
dbgcommand -p 4972 Debug=9
dbgcommand -p 4977 Debug=9
dbgcommand -p 4979 Debug=9
dbgcommand -p 4982 Debug=9
dbgcommand -p 4991 Debug=9
dbgcommand -p 4999 Debug=9
Note that when you invoke dbgcommand against a sub-daemon such as nsrmmd (as opposed to nsrd itself), you won’t get an alert in the daemon.{raw|log} file to indicate the debug level has changed.
 Posted by Preston de Guise at 06:37  Tagged with: , , , , , , ,

Cautions for running NetWorker daemons manually

 NetWorker  No Responses »
Jun 252009
 

(An alternate title for this entry might be, “Now that dbgcommand is available, make sure you use it”.)

The command dbgcommand, as previously discussed, has been recently added to the standard distributions for NetWorker (as of 7.5 and 7.4.4). This utility, in addition to a variety of other functions, is particularly useful at enabling an administrator to place one or more NetWorker daemons into debug mode without having to restart the services.

Recently we had an issue where a reasonably secured site experienced a variety of issues. To trace what was happening, NetWorker was placed into debug mode. As NetWorker had been stopped and it needed to be put into debug mode, the expedient option seemed to be to manually start nsrexecd, then manually run nsrd.exe -D3.

However, this had a side effect that was not quite anticipated. While the user account used to run nsrexecd and nsrd from had sufficient administrator priveleges to perform backups, email access was locked down sufficiently that the running user couldn’t send emails.

The result of course was that savegroup completion notifications, sent by nsrd running under the given user account were blocked. So were bootstrap notifications, for that matter.

The lesson? Now that dbgcommand is available, don’t mess around with manually running NetWorker in debug mode – make use of the tool that can do it all for you while preserving all other running options, including the account the services are run under.

 Posted by Preston de Guise at 13:39  Tagged with: , ,

Basics – dbgcommand

 Basics, NetWorker, Scripting  1 Response »
Feb 102009
 

Note: Please understand I’m not suggesting you play around with this feature randomly or without good reason. You shouldn’t make use of this unless you’re advised to by support – or unless you really, really know what you’re doing.

In the past if you’ve had a fairly serious issue with NetWorker you’ll probably have been advised to temporarily stop NetWorker, and restart it in debug mode. For example, support may have asked you to run, say:

# nsrd -D9

Which launches the main NetWorker server process (the “master” process) in debug mode level 9. There’s 99 levels of debug, ranging from 1 to 99, with the special level of “0″ being “off”. Debug mode level 1 is not really all that interesting; debug model level 3 can be useful sometimes for tracing name resolution issues. Level 9 produces fairly interesting details for people in support roles – I frequently make use of it when trying to debug issues customers may be encountering. (After that, the levels start to get less useful to anyone except for EMC engineers, and level 99 falls into the what, are you insane? category.)

There’s a problem with this strategy of stopping NetWorker and rerunning it though, as you may have already guessed – sometimes stopping and starting effectively kills off the problem you’re having. What is needed is the ability to put NetWorker into debug mode on the fly.

To address this, there’s been a support tool floating around for a while that EMC support would occasionally pass on to customers to help out with enabling debug mode without having to restart NetWorker. It’s called dbgcommand. However, dbgcommand wasn’t provided by default, and was typically only handed out either (a) when it was explicitly requested or (b) when engineering asked support to get the customer to run it.

As you can imagine, this has been a tad frustrating for just about everyone involved. Actually, I’ll rephrase that this has been quite frustrating for everyone involved.

Thanks in part to various people (e.g., my colleagues and I) continuing to ask for it, the pseudo-magical handing out of dbgcommand has been dropped, and it’s now included in NetWorker as of 7.5. [Note: I think it's also in 7.4.4...]

So, how do you use it?

Surprisingly easily:

# dbgcommand -p pid command

Where pid is the process ID of the NetWorker daemon you want to interact with.

Typically your usage would be say:

# dbgcommand -p pid Debug=X

Where X is a value between 0 and 99, or perhaps:

# dbgcommand -p pid FlushDnsCache

For example, if the NetWorker server process, /usr/sbin/nsrd, were running with a PID of 23278, you could put it in debug mode level 9 without a restart by typing the command:

# dbgcommand -p 23278 Debug=9

When you adjust a processes debug level, it gets logged in the appropriate daemon log – for example:

0 1234248435 2 0 0 3051682432 23277 0 nox nsrd 2 %s 1 0 54 Changing Debug level of process (id 23278) from 0 to 9
0 1234248446 2 0 0 3051682432 23277 0 nox nsrd 2 %s 1 0 54 Changing Debug level of process (id 23278) from 9 to 0

Or:

0 1234249625 2 0 0 3051682432 23277 0 nox nsrd 2 %s 1 0 56 Changing FlushDnsCache of process (id 23278) from 0 to 1

So there you have it – no longer do you have to stop and restart NetWorker to run in debug mode, you can alter the debug level of whatever NetWorker process you need to on the fly with minimum fuss.

 Posted by Preston de Guise at 18:08  Tagged with: , ,
© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha