Holiday Wrap-Up

 Architecture, Backup theory, General Technology, General thoughts, NetWorker  3 Responses »
Dec 232010
 

The holiday season is upon many of us – whether you celebrate xmas or christmas, or just the new year according to the Julian calendar, we’re approaching that point where things start to ease off for a lot of people and we spend more time with our families and friends.

Before I wrap up for the year, I wanted to spend a few minutes reintroducing some of the most popular topics of the year on the blog – the top ten articles based on directly linked accesses. Going in reverse order, they are:

  • Number 10 – “Why I’d choose NetWorker over NetBackup every time“. I was basically called an idiot by someone in the storage community for writing this, but the fact remains for me that any backup product that fails to support backup dependencies is not one that I would personally choose. Given that a top search that leads people to the blog is of the kind, “netbackup vs networker” or “networker vs netbackup”, clearly people are out there comparing the two products, and I stand by my support of the primacy of backup dependency tracking.
  • Number 9 – “A tale of 4 vendors“. A couple of months ago I attended SNIA’s first Australian storage blogger event, touring EMC, IBM, HDS and NetApp. Initially I’d planned to blog a fairly literal dump of the information I jotted down during the event, but I realised instead I was more drawn to the total solution stories being told by the 4 vendors.
  • Number 8 – “NetWorker 7.5.2 – What’s it got?“. NetWorker 7.5 represented a big upgrade mark for a lot of sites, particularly those that wanted to jump the v7.3 and v7.4 release trees. I still get a lot of searches coming to the blog based on NetWorker 7.5 features and upgrades.
  • Number 7 – “Using NetWorker Client with Opensolaris“. This was written by guest blogger Ronny Egner, and has seen more interest over the last few months as Oracle’s acquisition continues to grind down paid Sun customers. If you’re interested in writing guest blog pieces for the NetWorker Blog in 2011, let me know!
  • Number 6 – “Basics – Fixing ‘NSR peer information’ errors“. I’ve said it before, and I’ll say it again: there is no valid reason why the resolution for this hasn’t been built into NMC!
  • Number 5 – “NetWorker and linuxvtl, Redux“. The open source LinuxVTL project continues to grow and develop. While it’s not suited for production environments, LinuxVTL is certainly a handy VTL to plug into a NetWorker/Linux system for testing purposes. I know – I use it almost every single day.
  • Number 4 and Number 3 – “NetWorker 7.6 SP1“. Interest in NetWorker 7.6 SP1 has been huge, and I had two blog postings about it – a preview posting based on publicly shared information from EMC, and the actual post-release article that covered some key features more in-depth.
  • Number 2 – “Carry a Jukebox with you (if you’re using Linux)“. The first article I wrote about the LinuxVTL project.
  • Number 1 – “micromanual: NetWorker Power User Guide to nsradmin“. The Power User guide to nsradmin has been downloaded well over a thousand times. I’ve been a fan of nsradmin ever since I started using NetWorker and had to administer a few NetWorker servers over extremely slow links (think dial-up speeds). It’s been very gratifying to be able to introduce so many people to such a useful and powerful tool.

Personally this year has been a pretty big one for me. Probably the biggest single event was that my partner and I made the decision to move from central coast NSW to Melbourne, Victoria during the year. We haven’t moved yet; it’s due for June 2011, but it’s going to necessitate a lot of action and work on our part to get there. It’ll be well worth the effort though, and I’ve already reached that odd point where I no longer think of the place I’m living as “home”. The reasons that led us to that decision are covered on my personal blog here. Continuing the personal front, I was extremely pleased to be able to say goodbye to the mobile “netwont” that is Vodafone in Australia. I’ve been using my personal blog to talk about a lot of varied topics running from internet censorship to invasive information requests to more mundane things, such as what makes a good consultant.

Technically I think the coming few years are going to be fascinating. Deduplication has only just started to make a splash; I think it’ll be a while before it becomes as pervasive as say, plain old disk backup, but it will have a continued and growing effect in the enterprise backup market. I predict that another bevy of dopey analysts will insist that tape is dead, just like they have every year for the last 2 decades, and at the end of the year I predict the majority of companies they interface with will still be using tape in some form or another. However, the use of tape will continue to evolve in the marketplace; as nearline disk storage becomes more regular and cheaper for backup solutions, we’ll see tape continue to be pushed out to longer term retention systems and safety nets – i.e., tape is certainly sliding away from being the primary source for recoveries in an enterprise backup environment.

One last thing – I want to thank the readers of this blog. To those people who subscribe to the mailing list, and those who subscribe to the RSS feed, to those who have the site bookmarked and to those who just randomly stumble across the site – I hope in each case you’re finding something useful, and I’m grateful for your readership.

Happy holidays to those of you celebrating or relaxing over the coming weeks, and peaceful times to those working through.

 Posted by Preston de Guise at 12:47  Tagged with: , , , , , , , , , , , , , , , , , , , , , , ,

Basics – Specifying a default Jukebox

 Basics, NetWorker, Scripting  No Responses »
Aug 052010
 

If you’ve got multiple jukeboxes within a NetWorker environment, but primarily work with one of them, you may find ‘nsrjb’ to be a bit of a pain any time you forget to specify the jukebox name. If you’re not familiar with this, here’s how nsrjb reacts in this situation:

[root@tara ~]# nsrjb
1:	VTL1 	[enabled]
2:	VTL2 	[enabled]
No jukebox selected.
Please select a jukebox to use:? [1] _

(Slight aside: never assume the numbered list is the same; NetWorker doesn’t guarantee the order being the same between executions – in fact, I actually only put in an RFE about this a couple of days ago, as I’m hoping it could at least be alphabetically ordered at all times…)

If you want to avoid the jukebox-prompt from nsrjb, one of the easiest ways is to specify the jukebox name as part of the command – e.g.,

[root@tara ~]# nsrjb -j VTL1

That’s fine of course, but if the vast majority of the time you perform operations on a single jukebox, you can specify a default jukebox as an environment variable (NSR_JUKEBOX) and streamline your processes. For example, on Linux, using the bash, this might look as follows:

[root@tara ~]# export NSR_JUKEBOX=VTL1
[root@tara ~]# nsrjb
Jukebox VTL1: (Ready to accept commands)
slot  volume         pool           barcode   volume id        recyclable
1: 800840L4       ClientTesting  800840L4  3814088325       no
2: 800841L4       ClientTesting  800841L4  3797311146       no
3: 800842L4       ClientTesting  800842L4  3847642669       no
4: 800843L4       ClientTesting  800843L4  3780533937       no
5: 800844L4       ClientTesting  800844L4  3763756765       yes
6: 800845L4       ClientTesting  800845L4  3864419885       yes
<snip>

Being an environment variable, this is something you can choose to set locally – say, on a per storage-node basis, when you have multiple storage nodes. It’s relatively common for instance to have a tape library on one or more storage nodes, so for the appropriate logins (or even at a system level) on each storage node it would be possible to set the local jukebox as the default, thereby streamlining usage of the units.

As an example, here’s a lab storage node with the setting in use:

[root@fawn ~]# export NSR_JUKEBOX="rd=fawn:VTL3"
[root@fawn ~]# nsrjb -s tara
Jukebox rd=fawn:VTL3: (Ready to accept commands)
<snip>

For something that can take you less than 30 seconds to set, the environment variable NSR_JUKEBOX can certainly be a big time saver if you have multiple jukeboxes in your environment and (like me) you’re a command line junkie.

 Posted by Preston de Guise at 05:03  Tagged with: , , ,

Aside – Top Stories for February

 Aside, NetWorker  No Responses »
Mar 012010
 

Close enough together that I have to declare them a tie, the top stories for February were:

It’s fair to say that Carry a jukebox with you is remaining a big hit all the time – a bit like the “NSR peer information” story, and so February will be the last month that it gets included in consideration for top articles.

Towards the end of the month, with the release of NetWorker 7.5 SP2, there was quite a lot of interest in the articles “NetWorker 7.5.2 released” and “NetWorker 7.5.2 – What’s it got?“. Obviously if you’ve got Windows 2008 or Windows 7 clients that you need to backup, 7.5 SP2 is almost a no-brainer – you’ll really need to be using it. So far, based on my testing on Linux, 7.5 SP2 is looking fairly good for that platform too. As always, everyone should read the release notes before deciding whether to upgrade their environments.

 Posted by Preston de Guise at 06:43  Tagged with: , , , , , ,

Basics – Peeking inside your jukebox without leaving your desk

 Basics, NetWorker  No Responses »
Sep 102009
 

In order to speed up jukebox operations, NetWorker maintains a cache, or a map, if you will, of the current expected jukebox state based on the operations that have happened since it was last fully queried. This avoids having to do (time) costly SCSI probes before every operation.

(This, for what it’s worth, is why you can’t have another process, or another person, playing with the jukebox as well as NetWorker. For instance, a customer once had their jukebox accessible to all the developers on-site. They found on average the jukebox got into a terrible state several times a day, and thought they had a lemon of a product (either NetWorker or the STK L700) until they found out that having developers open the library door, arbitrarily pull tapes out and put new tapes in was not a good idea.)

Coming back to jukeboxes though, there are times when the cache is out of sync with reality. A few of the more common scenarios where this will happen are:

  • In disaster recovery situations
  • In situations where someone has manually moved around media
  • In situations where NetWorker has lost track of state due to a lengthy timeout on an error

In situations such as these, there’s an invaluable tool called sjirdtag that can come to the rescue. Instead of checking with the NetWorker cached contents of the library, sjirdtag instead delves down into what the library describes as its own content. I.e., it’s like peeking inside the library without having to leave your desk.

In order to use sjirdtag, you need to know the SCSI control port of the library; this is reported in the library properties in NetWorker management console, or you can find it out relatively quickly via inquire:

[root@tara ~]# inquire -l

-l flag found: searching all LUNs, which may take over 10 minutes per adapter
 for some fibre channel adapters.  Please be patient.

scsidev@0.0.0:STK     L700            5500|Autochanger (Jukebox), /dev/sg1
                                           S/N:    XYZZY     
                                           ATNN=STK     L700            XYZZY     
                                           WWNN=5123456003030303
scsidev@0.1.0:QUANTUM SDLT600         5500|Tape, /dev/nst0
                                           S/N:    ZF7584364
                                           ATNN=QUANTUM SDLT600         ZF7584364
                                           WWNN=5123456003030303

In this case, our library (a VTL presenting itself as an STK L700) is on scsidev@0.0.0. So, when we want to check the contents of the library, we run the command sjirdtag 0.0.0 – which looks like the following:

[root@tara ~]# sjirdtag 0.0.0
Tag Data for 0.0.0, Element Type DATA TRANSPORT:
        Elem[001]: tag_val=0 pres_val=1 med_pres=0 med_side=0
Tag Data for 0.0.0, Element Type STORAGE:
        Elem[001]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800843S3                       >
        Elem[002]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800844S3                       >
        Elem[003]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800845S3                       >
        Elem[004]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800846S3                       >
        Elem[005]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800847S3                       >
        Elem[006]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800848S3                       >
        Elem[007]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800849S3                       >
        Elem[008]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800850S3                       >
        Elem[009]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800851S3                       >
        Elem[010]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800852S3                       >
        Elem[011]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800853S3                       >
        Elem[012]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800854S3                       >
        Elem[013]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800855S3                       >
        Elem[014]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800856S3                       >
        Elem[015]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800857S3                       >
        Elem[016]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800858S3                       >
        Elem[017]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800859S3                       >
        Elem[018]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800860S3                       >
        Elem[019]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800861S3                       >
        Elem[020]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<800862S3                       >
        Elem[021]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG990S3                       >
        Elem[022]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG991S3                       >
        Elem[023]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG992S3                       >
        Elem[024]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG993S3                       >
        Elem[025]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG994S3                       >
        Elem[026]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG995S3                       >
        Elem[027]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG996S3                       >
        Elem[028]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG997S3                       >
        Elem[029]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG998S3                       >
        Elem[030]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<BIG999S3                       >
        Elem[031]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<CLN001L1                       >
        Elem[032]: tag_val=1 pres_val=1 med_pres=1 med_side=0
                   VolumeTag=<CLN002L1                       >
Tag Data for 0.0.0, Element Type MEDIA TRANSPORT:
        Elem[001]: tag_val=0 pres_val=1 med_pres=0 med_side=0
Tag Data for 0.0.0, Element Type IMPORT/EXPORT:
        Elem[001]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1
        Elem[002]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1
        Elem[003]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1
        Elem[004]: tag_val=0 pres_val=1 inp_enab=1 exp_enab=1 access=1 full=0 imp_exp=1

For those who are unfamiliar with sjirdtag, let’s break this up into the four sections presented (using the capitalisation in the output – not shouting):

  • DATA TRANSPORT – Refers to the tape drives within the library – i.e., the units responsible for transporting the data.
  • STORAGE – The slots used by the library for storage of cartridges. This does not refer to the slot(s) in the CAP/MAS.
  • MEDIA TRANSPORT – The robot head(s). There’ll be one per robot head.
  • IMPORT/EXPORT – The contents of the slots in the CAP/MAS.

If you’re wondering about those element numbers, they’re essentially the positions or numbers of the units as assigned by the library. In particular, for the drives (DATA TRANSPORT) section, these refer to the drives in order as they are presented by the tape library; this means that if your operating system drive mappings don’t match the library sequence, the output here also won’t match the operating system sequence of devices.

Now for each element other than the CAP/MAS areas, we get the following selection of information:

tag_val=[0|1] pres_val=[0|1] med_pres=[0|1] med_side=[0|1]

Each of these items mean:

  • tag_val – Indicates that there’s SCSI tag data for that element. 1 for yes, 0 for no.
  • med_pres – Jukebox state indicates that there is media present in this location. 1 for yes, 0 for no.
  • pres_val – A bit of an airy-fairy value; if set to 1, then it means that the med_pres value should be fairly believable. If set to 0 but the med_pres value is 1, then while there may be media present, there may also be an error condition. If set to 0, and med_pres is set to 0, then it also means that the med_pres value should be fairly believable.
  • med_side – For jukeboxes/media that supports double-sided media (e.g., older optical disk libraries), this indicates which side of the media is in use; for tape based libraries, this will always be 0.

For any element that has a volume with a barcode, this will be shown on the line underneath the element details with the format:

VolumeTag=<PCL                 >

For our import/export regions, the additional options, inp_enab, exp_enab, access, full and imp_exp are effectively undocumented, but my assumption on these items are:

  • inp_enab – Slot can be used for import.
  • exp_enab – Slot can be used for export.
  • access – Slot is accessible.
  • imp_exp – Slot is an import/export slot.

(The other option, “full”, most definitely indicates whether the slot is occupied or not.)

As can be evidenced by the “airy-fairy” nature of the pres_val tag, there’s no 100% guarantee that this information is physically accurate. However, it is an accurate reflection of the state that the library thinks it’s in, and thus is an accurate reflection of how the library will behave in response to requested operations. Furthermore, if the state shown by sjirdtag differs from the state shown by nsrjb, then it’s a good indication that it’s time to reset/reinventory the library. I.e., time to run:

# nsrjb -HEvvv
# nsrjb -II

(The reset instructs NetWorker to throw away its state information, tell the library to reinitialise itself, and then refreshes the volume state.The inventory command specified is assuming a barcode-supported library with barcoded volumes.)

Things that I routinely use (or get customers to use) sjirdtag for include:

  • Checking to see if there is a tape in a drive that NetWorker thinks is empty.
  • Checking to see if the tape NetWorker thinks is in a drive really is in the drive.
  • Checking to see if operators at a remote library have loaded media into the CAP/MAS.
  • Checking to see if there is a tape stuck in the robot gripper.
  • Finding the bootstrap volume when a disaster recovery (mmrecov) is required.

If you’ve not used sjirdtag before, it’s worthwhile scheduling a time where there’s minimal activity in the library so you can check it out.

 Posted by Preston de Guise at 19:42  Tagged with: , , , ,

Basics – Labelling and Relabelling without Unmounting

 Basics  No Responses »
Aug 202009
 

Here’s a common scenario – you want to label a volume, or relabel a volume, and use it straight away. The default behaviour of NetWorker after labelling or relabelling a volume is to then unmount it, which means having to then manually mount the volume after it has been (unnecessarily) ejected.

Getting around this behaviour is quite easy, and just requires a bit of typing on the command line.

Let’s look first at relabelling, since this is arguably the most common scenario. Say you’ve got a volume in slot 21 of your tape library that you want to relabel and have it remain mounted so you can immediately start using it. For a normal relabel operation you’d consider something like:

# nsrjb -LRYvvv -S 21

Note 1: I always put in the ‘-vvv’ option whenever dealing with a jukebox. These days I practically consider it to be ‘best practices’.

Note 2: In the examples in this article I’m using the -Y switch, which means NetWorker does not prompt for any confirmation on the operation (it assumes Yes in response to any question it may have); this is done only for the purposes of keeping example output simplified, and I don’t recommend you get in the habit of using it.

Instead of using the -L option here, we switch to -l (for load); thus the command becomes:

[root@tara ~]# nsrjb -lRYvvv -S 21
setting verbosity level to `3'
Info: Preparing to load volume `BIG990S3' from slot 21 into device `/dev/nst0'.
Info: Loading volume `BIG990S3' from slot `21' into device `/dev/nst0'.
Info: Load sleep for 5 seconds.
Info: Performing operation `Verify label' on device `/dev/nst0'.
Info: Operation `Verify label' in progress on device `/dev/nst0'
Info: Performing operation `Label' on device `/dev/nst0'.
Info: Operation `Label' in progress on device `/dev/nst0'
Info: Recycling volume `BIG990S3'

That’s it – those of you familiar with highly verbose nsrjb output will recognise that there’s no “Unmount in progress” style message; the volume remains mounted and instantly ready for use once the relabel operation is complete.

Now, moving on to a tape that hasn’t previously been labelled, we’d usually use a command such as:

# nsrjb -LYvvv -b poolName -S x

However, to keep the tape mounted after labelling, we need to include the ‘-m’ option; thus, if we wanted to label the tape in slot 1 into the “Default Clone” pool and keep it mounted after labelling, our command would look like the following:

[root@tara ~]# nsrjb -mLYvvv -b "Default Clone" -S 1
setting verbosity level to `3'
Info: Preparing to load volume `800843S3' from slot 1 into device `/dev/nst0'.
Info: Loading volume `800843S3' from slot `1' into device `/dev/nst0'.
Info: Load sleep for 5 seconds.
Info: Performing operation `Verify label' on device `/dev/nst0'.
Info: Operation `Verify label' in progress on device `/dev/nst0'
Info: Expected volume `800843S3' in slot `1'. The actual volume is `<NULL>'.
Info: Cannot read the current volume label `no tape label found'.
Info: nsrmmgd assumes the volume is unlabeled and will write a new label.
Info: Performing operation `Label' on device `/dev/nst0'.
Info: Operation `Label' in progress on device `/dev/nst0'
Info: Label: `800843S3', pool: `Default Clone', capacity: `<NULL>'.

There you go … and don’t forget Note 2 above! It’s not wise to get into the habit of throwing a -Y into nsrjb commands; the examples only show it to keep the examples simpler.

 Posted by Preston de Guise at 06:15  Tagged with: , , , , , ,

Debugging autochanger operations

 NetWorker, Support  No Responses »
Jun 012009
 

Having a strong support background, and having a preference for working in software rather than hardware, when I use NetWorker’s nsrjb utility with tape libraries I like to get as much feedback as possible on operations.

When performing command line operations with nsrjb, this is relatively straight forward – just add on a few “-v” options to the operation. Indeed, my customers sometimes probably tire of hearing me start every nsrjb command with:

# nsrjb -vvv

It’s become such a force of habit that I literally don’t consider running it without those three verbose flags any more*. If you’re not sure why you would do this, or have never run nsrjb with verbose flags, you should go and do it – you get a lot more information about the progress of the operation, which can actually be very important if you’re debugging or otherwise trying to resolve hanging hardware operations.

Being able to run nsrjb in verbose mode is all well and good, but it doesn’t help you diagnose periodic or random failures if they’re only happening during backup operations. To keep track of progress on these, you need to use a setting within the jukebox resource – the debug trace level option.

This updates the daemon.raw log file with extra details about autochanger operations, in a similar way to the additional details you get when you run nsrjb with additional verbosity flags. It can be set to a number, and like the debug option for a lot of NetWorker commands, defaults to 0 (for no additional information), and can be set to ever larger numbers that generate a lot of extra information. It doesn’t generate the same information as running nsrjb with multiple “-v” flags, but it still helps.

Now, before I go further, I’d suggest that this is probably something you shouldn’t leave enabled all the time. That is, if you’re having an intermittent problem with a tape library, consider turning this on so that you have the option of reviewing log files from overnight backups to see if there’s any relevant/helpful information from the time of failures. However, when you finish your debugging, turn it back off by setting it to zero.

There’s two ways you can set it – either from the command line, using nsradmin, or from NetWorker Management Console when viewing a jukebox resource with “diagnostic” details turned on.

Let’s look at nsradmin first:

# nsradmin -s nox
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin> show name:; debug trace level:
nsradmin> print type: NSR jukebox
                        name: LTO1_LIB;
           debug trace level: 0;
nsradmin> update debug trace level: 9
           debug trace level: 9;
Update? y
updated resource id 31.0.255.110.0.0.0.0.60.6.146.73.0.0.0.0.10.0.0.1(5348)
nsradmin> print
                        name: LTO1_LIB;
           debug trace level: 9;

It’s as simple as that!

If you want to see it from within NMC, you’d make the changes in the “Debug Trace Level” field, as per the screen-shot below:

Advanced tab of Autochanger properties

Advanced tab of Autochanger properties

(The debug trace level setting is the bottom-most option on the left-hand side of this tab.)

You’ll find that debug level of 9 will more than likely be more than enough for any regular fault finding exercise, as the amount of information produced by this is quite high. For instance, at the bottom of this post I’ve got output from a single operation, “nsrjb -lvvv -S 1″ to the volume in slot one into a drive.

Remember when you’re done of course to turn the trace level back off by setting the debug trace level to zero.

nox nsrlcpd getenv_ulong(NSR_LCPD_BASEELEMOFFSET=0,0:65535, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_NO_LOAD_EXP=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_NOVUELMASC=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_AUTOEJECT=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_AUTOEXT=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_AUTOEXT2=0,0:1, 0) returning 0
nox nsrlcpd getenv_ulong(NSR_LCPD_RES_DRVACCESS=0,0:1, 0) returning 0
nox nsrlcpd Event LE_CONTINUE raised for cmdid 16055
nox nsrlcpd Processing: cmdid 16055 & cmdop 12 for event: LE_CONTINUE
nox nsrlcpd Event LE_COMP_OK raised for cmdid 16055
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16055 is_master=yes cmdop:UPDATE.
nox nsrlcpd \ncomplete :no
nox nsrlcpd \n
nox nsrmmgd Loading volume `Clone.001' from slot `1' into device `/dev/nst0'.
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16056 is_master=no cmdop:LOAD.
nox nsrlcpd     MOVE MEDIA INFO
nox nsrlcpd         source : S:000031
nox nsrlcpd         source attrlist : NULL
nox nsrlcpd         dest   : D:000001
nox nsrlcpd Event LE_NEWCMD raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_NEWCMD
Unable to render the following message: freeing unused errinfo with msgid 14947

nox nsrlcpd Event LE_CONTINUE raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_CONTINUE
nox nsrlcpd lg_lstat(): Calling lstat64().
nox nsrlcpd lg_open(): Calling open64().
nox nsrlcpd lcpd_last_status_set: OK->OK (requested OK).
nox nsrlcpd Event LE_CONTINUE raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_CONTINUE
nox nsrlcpd lg_lstat(): Calling lstat64().
nox nsrlcpd lg_open(): Calling open64().
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16057 is_master=no cmdop:STATUS.
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16057 is_master=yes cmdop:STATUS.
nox nsrlcpd pid:7606 lcp_rpc_vers:1, jb_list_len:1
nox nsrlcpd \n
nox nsrlcpd lcpd_last_status_set: OK->OK (requested OK).
Unable to render the following message: freeing unused errinfo with msgid 14947

nox nsrlcpd Event LE_CONTINUE raised for cmdid 16056
nox nsrlcpd Processing: cmdid 16056 & cmdop 4 for event: LE_CONTINUE
nox nsrlcpd Event LE_COMP_OK raised for cmdid 16056
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16056 is_master=yes cmdop:LOAD.
nox nsrlcpd \ncomplete :no
nox nsrlcpd offset:0    tag:S:000031    ok:yes full: no barcode:       ? dest:D:000001 bay:?
nox nsrlcpd \n
nox nsrlcpd offset:0      tag:D:000001    ok:yes full:yes barcode:       ? source:S:000031 serial#:? bay:?
nox nsrlcpd \n
nox nsrd /dev/nst0 Verify label operation in progress
nox nsrd /dev/nst0 Mount operation in progress
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16058 is_master=no cmdop:STATUS.
nox nsrlcpd Responder woke up due to event
nox nsrlcpd LCPD_HDR: server:nox.anywebdb.com vers:1 jbid:LTO1_LIB cmdid:16058 is_master=yes cmdop:STATUS.
nox nsrlcpd pid:7606 lcp_rpc_vers:1, jb_list_len:1
nox nsrlcpd \n
nox nsrd [Jukebox `LTO1_LIB', operation # 30]. Finished with status: succeeded
nox nsrlcpd Processing: cmdid 4294967295 & cmdop 17 for event: LE_NEWCMD
nox nsrlcpd Event ? raised for cmdid 4294967295
nox nsrlcpd Processing: cmdid 4294967295 & cmdop 17 for event: ?
nox nsrlcpd lg_lstat(): Calling lstat64().
nox nsrlcpd lg_open(): Calling open64().
nox nsrlcpd lcpd_last_status_set: OK->OK (requested OK).
nox nsrlcpd Polling current hw status jukebox:LTO1_LIB status:OK
nox nsrlcpd Event ? raised for cmdid 4294967295
nox nsrlcpd Processing: cmdid 4294967295 & cmdop 17 for event: ?
nox nsrlcpd Event LE_COMP_OK raised for cmdid 4294967295

–
* With the one exception being when it’s used in scripting.

debug trace level option.

 Posted by Preston de Guise at 08:23  Tagged with: , , ,

Basics – jbverify

 Basics  No Responses »
Apr 082009
 

Now, some might say that I’m not the smartest card in the deck for what I’m about to note, but sometimes I don’t notice new commands when they appear in NetWorker, particularly if I skimmed through release notes.

I was pleasantly surprised today to find that a new command had slipped in at some point called “jbverify”. I can immediately see it however entering my stable of must-use commands, particularly in a support environment.

To quote the man page, jbverify:

[V]erifies the devices defined in the NetWorker database, making sure that each one of them is configured properly by checking them  for accessibility  and  usability.

This is the sort of diagnostic tool that support people live for, and sites suddenly experiencing strange jukebox issues should think of as a matter of course.

When run on my lab server this afternoon, I got the following badly formatted but still very useful output:

# jbverify
14866:jbverify:
 Jbverify is running on host nox, Linux 2.6.18-128.1.6.el5

14912:jbverify:
 Processing stand-alone devices...

14913:jbverify:
 Processing /d/nsr/02/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/02/_AF_readonly

14913:jbverify:
 Processing /d/nsr/idata/backup/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/idata/backup/_AF_readonly

14913:jbverify:
 Processing /d/nsr/01/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/01/_AF_readonly

14913:jbverify:
 Processing /d/nsr/idata/backup
14915:jbverify:
 Finished processing /d/nsr/idata/backup

14913:jbverify:
 Processing /d/nsr/idata/clone/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/idata/clone/_AF_readonly

14913:jbverify:
 Processing /d/nsr/01
14915:jbverify:
 Finished processing /d/nsr/01

14913:jbverify:
 Processing /d/nsr/03
14915:jbverify:
 Finished processing /d/nsr/03

14913:jbverify:
 Processing /d/nsr/03/_AF_readonly
14915:jbverify:
 Finished processing /d/nsr/03/_AF_readonly

14913:jbverify:
 Processing /d/nsr/02
14915:jbverify:
 Finished processing /d/nsr/02

14913:jbverify:
 Processing /d/nsr/idata/clone
14915:jbverify:
 Finished processing /d/nsr/idata/clone

14917:jbverify:
 Finished processing stand-alone devices.

14918:jbverify:
 Processing jukebox devices...

14920:jbverify:
 Processing jukebox LTO1_LIB:

14733:jbverify:
 Testing drive 1 (/dev/nst0) of JB LTO1_LIB

14927:jbverify:

 Jukebox LTO1_LIB on nox successfully processed.

14929:jbverify:

 Finished processing jukebox devices.

**********************************************************************

         Summary report of jbverify
         ======= ====== == ========

Hostname   Device Handle    Blocksize  Jukebox  Drv No. Status
--------   -------------    ---------  -------  ------- ------
nox        /d/nsr/02/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/idata/backup/_AF_readonly 131072 N/A N/A Pass
nox        /d/nsr/01/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/idata/backup 131072  N/A      N/A     Pass
nox        /d/nsr/idata/clone/_AF_readonly 131072 N/A N/A Pass
nox        /d/nsr/01        131072     N/A      N/A     Pass
nox        /d/nsr/03        131072     N/A      N/A     Pass
nox        /d/nsr/03/_AF_readonly 131072 N/A    N/A     Pass
nox        /d/nsr/02        131072     N/A      N/A     Pass
nox        /d/nsr/idata/clone 131072   N/A      N/A     Pass
nox        /dev/nst0        65536      LTO1_LIB 1       Pass

**********************************************************************

If you’ve come from NetBackup, the nature of this program is somewhat reminsicent of the robtest utility. I don’t claim EMC are special for having introduced this tool, but I do applaud that it’s there (and lament that I didn’t notice it sooner).

(One thing to note: after running jbverify, make sure you reset your jukebox.)

 Posted by Preston de Guise at 19:38  Tagged with: , ,
© 2012 The NetWorker Blog Suffusion theme by Sayontan Sinha