(This, for what it’s worth, is why you can’t have another process, or another person, playing with the jukebox as well as NetWorker. For instance, a customer once had their jukebox accessible to all the developers on-site. They found on average the jukebox got into a terrible state several times a day, and thought they had a lemon of a product (either NetWorker or the STK L700) until they found out that having developers open the library door, arbitrarily pull tapes out and put new tapes in was not a good idea.)

Coming back to jukeboxes though, there are times when the cache is out of sync with reality. A few of the more common scenarios where this will happen are:

• In disaster recovery situations
• In situations where someone has manually moved around media
• In situations where NetWorker has lost track of state due to a lengthy timeout on an error

In situations such as these, there’s an invaluable tool called sjirdtag that can come to the rescue. Instead of checking with the NetWorker cached contents of the library, sjirdtag instead delves down into what the library describes as its own content. I.e., it’s like peeking inside the library without having to leave your desk.

In order to use sjirdtag, you need to know the SCSI control port of the library; this is reported in the library properties in NetWorker management console, or you can find it out relatively quickly via inquire:

[root@tara ~]# inquire -l

-l flag found: searching all LUNs, which may take over 10 minutes per adapter
 for some fibre channel adapters.  Please be patient.

scsidev@0.0.0:STK     L700            5500|Autochanger (Jukebox), /dev/sg1
                                           S/N:    XYZZY     
                                           ATNN=STK     L700            XYZZY     
                                           WWNN=5123456003030303
scsidev@0.1.0:QUANTUM SDLT600         5500|Tape, /dev/nst0
                                           S/N:    ZF7584364
                                           ATNN=QUANTUM SDLT600         ZF7584364
                                           WWNN=5123456003030303

In this case, our library (a VTL presenting itself as an STK L700) is on scsidev@0.0.0. So, when we want to check the contents of the library, we run the command sjirdtag 0.0.0 – which looks like the following:

[root@tara ~]# sjirdtag 0.0.0
Tag Data for 0.0.0, Element Type DATA TRANSPORT:
        Elem[001]: tag_val=0 pres_val=1 med_pres=0 med_side=0
Tag Data for 0.0.0, Element Type STORAGE:
        Elem[001]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800843S3                       >
        Elem[002]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800844S3                       >
        Elem[003]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800845S3                       >
        Elem[004]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800846S3                       >
        Elem[005]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800847S3                       >
        Elem[006]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800848S3                       >
        Elem[007]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800849S3                       >
        Elem[008]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800850S3                       >
        Elem[009]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800851S3                       >
        Elem[010]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800852S3                       >
        Elem[011]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800853S3                       >
        Elem[012]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800854S3                       >
        Elem[013]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800855S3                       >
        Elem[014]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800856S3                       >
        Elem[015]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800857S3                       >
        Elem[016]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800858S3                       >
        Elem[017]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800859S3                       >
        Elem[018]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800860S3                       >
        Elem[019]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800861S3                       >
        Elem[020]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800862S3                       >
        Elem[021]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG990S3                       >
        Elem[022]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG991S3                       >
        Elem[023]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG992S3                       >
        Elem[024]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG993S3                       >
        Elem[025]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG994S3                       >
        Elem[026]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG995S3                       >
        Elem[027]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG996S3                       >
        Elem[028]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG997S3                       >
        Elem[029]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG998S3                       >
        Elem[030]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG999S3                       >
        Elem[031]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<CLN001L1                       >
        Elem[032]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<CLN002L1                       >
Tag Data for 0.0.0, Element Type MEDIA TRANSPORT:
        Elem[001]: tag_val=0 pres_val=1 med_pres=0 med_side=0
Tag Data for 0.0.0, Element Type IMPORT/EXPORT:
        Elem[001]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1
        Elem[002]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1
        Elem[003]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1
        Elem[004]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1

For those who are unfamiliar with sjirdtag, let’s break this up into the four sections presented (using the capitalisation in the output – not shouting):

• DATA TRANSPORT – Refers to the tape drives within the library – i.e., the units responsible for transporting the data.
• STORAGE – The slots used by the library for storage of cartridges. This does not refer to the slot(s) in the CAP/MAS.
• MEDIA TRANSPORT – The robot head(s). There’ll be one per robot head.
• IMPORT/EXPORT – The contents of the slots in the CAP/MAS.

If you’re wondering about those element numbers, they’re essentially the positions or numbers of the units as assigned by the library. In particular, for the drives (DATA TRANSPORT) section, these refer to the drives in order as they are presented by the tape library; this means that if your operating system drive mappings don’t match the library sequence, the output here also won’t match the operating system sequence of devices.

Now for each element other than the CAP/MAS areas, we get the following selection of information:

tag_val=[0|1] pres_val=[0|1] med_pres=[0|1] med_side=[0|1]

Each of these items mean:

• tag_val – Indicates that there’s SCSI tag data for that element. 1 for yes, 0 for no.
• med_pres – Jukebox state indicates that there is media present in this location. 1 for yes, 0 for no.
• pres_val – A bit of an airy-fairy value; if set to 1, then it means that the med_pres value should be fairly believable. If set to 0 but the med_pres value is 1, then while there may be media present, there may also be an error condition. If set to 0, and med_pres is set to 0, then it also means that the med_pres value should be fairly believable.
• med_side – For jukeboxes/media that supports double-sided media (e.g., older optical disk libraries), this indicates which side of the media is in use; for tape based libraries, this will always be 0.

For any element that has a volume with a barcode, this will be shown on the line underneath the element details with the format:

VolumeTag=<PCL                 >

For our import/export regions, the additional options, inp_enab, exp_enab, access, full and imp_exp are effectively undocumented, but my assumption on these items are:

• inp_enab – Slot can be used for import.
• exp_enab – Slot can be used for export.
• access – Slot is accessible.
• imp_exp – Slot is an import/export slot.

(The other option, “full”, most definitely indicates whether the slot is occupied or not.)

As can be evidenced by the “airy-fairy” nature of the pres_val tag, there’s no 100% guarantee that this information is physically accurate. However, it is an accurate reflection of the state that the library thinks it’s in, and thus is an accurate reflection of how the library will behave in response to requested operations. Furthermore, if the state shown by sjirdtag differs from the state shown by nsrjb, then it’s a good indication that it’s time to reset/reinventory the library. I.e., time to run:

# nsrjb -HEvvv
# nsrjb -II

(The reset instructs NetWorker to throw away its state information, tell the library to reinitialise itself, and then refreshes the volume state.The inventory command specified is assuming a barcode-supported library with barcoded volumes.)

Things that I routinely use (or get customers to use) sjirdtag for include:

• Checking to see if there is a tape in a drive that NetWorker thinks is empty.
• Checking to see if the tape NetWorker thinks is in a drive really is in the drive.
• Checking to see if operators at a remote library have loaded media into the CAP/MAS.
• Checking to see if there is a tape stuck in the robot gripper.
• Finding the bootstrap volume when a disaster recovery (mmrecov) is required.

If you’ve not used sjirdtag before, it’s worthwhile scheduling a time where there’s minimal activity in the library so you can check it out.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Getting around this behaviour is quite easy, and just requires a bit of typing on the command line.

Let’s look first at relabelling, since this is arguably the most common scenario. Say you’ve got a volume in slot 21 of your tape library that you want to relabel and have it remain mounted so you can immediately start using it. For a normal relabel operation you’d consider something like:

# nsrjb -LRYvvv -S 21

Note 1: I always put in the ‘-vvv’ option whenever dealing with a jukebox. These days I practically consider it to be ‘best practices’.

Note 2: In the examples in this article I’m using the -Y switch, which means NetWorker does not prompt for any confirmation on the operation (it assumes Yes in response to any question it may have); this is done only for the purposes of keeping example output simplified, and I don’t recommend you get in the habit of using it.

Instead of using the -L option here, we switch to -l (for load); thus the command becomes:

[root@tara ~]# nsrjb -lRYvvv -S 21
setting verbosity level to `3'
Info: Preparing to load volume `BIG990S3' from slot 21 into device `/dev/nst0'.
Info: Loading volume `BIG990S3' from slot `21' into device `/dev/nst0'.
Info: Load sleep for 5 seconds.
Info: Performing operation `Verify label' on device `/dev/nst0'.
Info: Operation `Verify label' in progress on device `/dev/nst0'
Info: Performing operation `Label' on device `/dev/nst0'.
Info: Operation `Label' in progress on device `/dev/nst0'
Info: Recycling volume `BIG990S3'

That’s it – those of you familiar with highly verbose nsrjb output will recognise that there’s no “Unmount in progress” style message; the volume remains mounted and instantly ready for use once the relabel operation is complete.

Now, moving on to a tape that hasn’t previously been labelled, we’d usually use a command such as:

# nsrjb -LYvvv -b poolName -S x

However, to keep the tape mounted after labelling, we need to include the ‘-m’ option; thus, if we wanted to label the tape in slot 1 into the “Default Clone” pool and keep it mounted after labelling, our command would look like the following:

[root@tara ~]# nsrjb -mLYvvv -b "Default Clone" -S 1
setting verbosity level to `3'
Info: Preparing to load volume `800843S3' from slot 1 into device `/dev/nst0'.
Info: Loading volume `800843S3' from slot `1' into device `/dev/nst0'.
Info: Load sleep for 5 seconds.
Info: Performing operation `Verify label' on device `/dev/nst0'.
Info: Operation `Verify label' in progress on device `/dev/nst0'
Info: Expected volume `800843S3' in slot `1'. The actual volume is `<NULL>'.
Info: Cannot read the current volume label `no tape label found'.
Info: nsrmmgd assumes the volume is unlabeled and will write a new label.
Info: Performing operation `Label' on device `/dev/nst0'.
Info: Operation `Label' in progress on device `/dev/nst0'
Info: Label: `800843S3', pool: `Default Clone', capacity: `<NULL>'.

There you go … and don’t forget Note 2 above! It’s not wise to get into the habit of throwing a -Y into nsrjb commands; the examples only show it to keep the examples simpler.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

When performing command line operations with nsrjb, this is relatively straight forward – just add on a few “-v” options to the operation. Indeed, my customers sometimes probably tire of hearing me start every nsrjb command with:

# nsrjb -vvv

It’s become such a force of habit that I literally don’t consider running it without those three verbose flags any more*. If you’re not sure why you would do this, or have never run nsrjb with verbose flags, you should go and do it – you get a lot more information about the progress of the operation, which can actually be very important if you’re debugging or otherwise trying to resolve hanging hardware operations.

Being able to run nsrjb in verbose mode is all well and good, but it doesn’t help you diagnose periodic or random failures if they’re only happening during backup operations. To keep track of progress on these, you need to use a setting within the jukebox resource – the debug trace level option.

This updates the daemon.raw log file with extra details about autochanger operations, in a similar way to the additional details you get when you run nsrjb with additional verbosity flags. It can be set to a number, and like the debug option for a lot of NetWorker commands, defaults to 0 (for no additional information), and can be set to ever larger numbers that generate a lot of extra information. It doesn’t generate the same information as running nsrjb with multiple “-v” flags, but it still helps.

Now, before I go further, I’d suggest that this is probably something you shouldn’t leave enabled all the time. That is, if you’re having an intermittent problem with a tape library, consider turning this on so that you have the option of reviewing log files from overnight backups to see if there’s any relevant/helpful information from the time of failures. However, when you finish your debugging, turn it back off by setting it to zero.

There’s two ways you can set it – either from the command line, using nsradmin, or from NetWorker Management Console when viewing a jukebox resource with “diagnostic” details turned on.

Let’s look at nsradmin first:

# nsradmin -s nox
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin> show name:; debug trace level:
nsradmin> print type: NSR jukebox
                        name: LTO1_LIB;
           debug trace level: 0;
nsradmin> update debug trace level: 9
           debug trace level: 9;
Update? y
updated resource id 31.0.255.110.0.0.0.0.60.6.146.73.0.0.0.0.10.0.0.1(5348)
nsradmin> print
                        name: LTO1_LIB;
           debug trace level: 9;

It’s as simple as that!

If you want to see it from within NMC, you’d make the changes in the “Debug Trace Level” field, as per the screen-shot below:

Advanced tab of Autochanger properties

(The debug trace level setting is the bottom-most option on the left-hand side of this tab.)

You’ll find that debug level of 9 will more than likely be more than enough for any regular fault finding exercise, as the amount of information produced by this is quite high. For instance, at the bottom of this post I’ve got output from a single operation, “nsrjb -lvvv -S 1″ to the volume in slot one into a drive.

Remember when you’re done of course to turn the trace level back off by setting the debug trace level to zero.

nox nsrlcpd getenv_ulong(NSR_LCPD_BASEELEMOFFSET=0,0:65535, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_NO_LOAD_EXP=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_NOVUELMASC=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_AUTOEJECT=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_AUTOEXT=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_AUTOEXT2=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_RES_DRVACCESS=0,0:1, 0) returning 0
nox nsrlcpd Event LE_CONTINUE raised for cmdid 16055
nox nsrlcpd Processing: cmdid 16055 & cmdop 12 for event: LE_CONTINUE
nox nsrlcpd Event LE_COMP_OK raised for cmdid 16055
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16055 is_master=yes cmdop:UPDATE.
nox nsrlcpd \ncomplete :no
nox nsrlcpd \n
nox nsrmmgd Loading volume `Clone.001' from slot `1' into device `/dev/nst0'.
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16056 is_master=no cmdop:LOAD.
nox nsrlcpd     MOVE MEDIA INFO
nox nsrlcpd         source : S:000031
nox nsrlcpd         source attrlist : NULL
nox nsrlcpd         dest   : D:000001
nox nsrlcpd Event LE_NEWCMD raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_NEWCMD
Unable to render the following message: freeing unused errinfo with msgid 14947

nox nsrlcpd Event LE_CONTINUE raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_CONTINUE
nox nsrlcpd lg_lstat(): Calling lstat64().
nox nsrlcpd lg_open(): Calling open64().
nox nsrlcpd lcpd_last_status_set: OK->OK (requested OK).
nox nsrlcpd Event LE_CONTINUE raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_CONTINUE
nox nsrlcpd lg_lstat(): Calling lstat64().
nox nsrlcpd lg_open(): Calling open64().
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16057 is_master=no cmdop:STATUS.
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16057 is_master=yes cmdop:STATUS.
nox nsrlcpd pid:7606 lcp_rpc_vers:1, jb_list_len:1
nox nsrlcpd \n
nox nsrlcpd lcpd_last_status_set: OK->OK (requested OK).
Unable to render the following message: freeing unused errinfo with msgid 14947

nox nsrlcpd Event LE_CONTINUE raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_CONTINUE
nox nsrlcpd Event LE_COMP_OK raised for cmdid 16056
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16056 is_master=yes cmdop:LOAD.
nox nsrlcpd \ncomplete :no
nox nsrlcpd offset:0    tag:S:000031    ok:yes full: no barcode:       ? dest:D:000001 bay:?
nox nsrlcpd \n
nox nsrlcpd offset:0      tag:D:000001    ok:yes full:yes barcode:       ? source:S:000031 serial#:? bay:?
nox nsrlcpd \n
nox nsrd /dev/nst0 Verify label operation in progress
nox nsrd /dev/nst0 Mount operation in progress
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16058 is_master=no cmdop:STATUS.
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16058 is_master=yes cmdop:STATUS.
nox nsrlcpd pid:7606 lcp_rpc_vers:1, jb_list_len:1
nox nsrlcpd \n
nox nsrd [Jukebox `LTO1_LIB', operation # 30]. Finished with status: succeeded
nox nsrlcpd Processing: cmdid 4294967295 & cmdop 17 for event: LE_NEWCMD
nox nsrlcpd Event ? raised for cmdid 4294967295
nox nsrlcpd Processing: cmdid 4294967295 & cmdop 17 for event: ?
nox nsrlcpd lg_lstat(): Calling lstat64().
nox nsrlcpd lg_open(): Calling open64().
nox nsrlcpd lcpd_last_status_set: OK->OK (requested OK).
nox nsrlcpd Polling current hw status jukebox:LTO1_LIB status:OK
nox nsrlcpd Event ? raised for cmdid 4294967295
nox nsrlcpd Processing: cmdid 4294967295 & cmdop 17 for event: ?
nox nsrlcpd Event LE_COMP_OK raised for cmdid 4294967295

–
* With the one exception being when it’s used in scripting.

debug trace level option.

Share this using...

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.