A rather smart gent whom I used to work with at another company, Mark Harvey, has in his own time been working on an open source VTL implementation that can be used for testing/lab purposes. I.e., he’s not aiming for it to be the next competitor to EDLs, FalconStor, etc., but rather, something that people can use when needing to test jukebox functionality without wanting to carry a jukebox around with them.
While Mark has primarily focused on getting his VTL software working with NetBackup, recently he’s made some progress in getting it to work (with a couple of limitations) with NetWorker.
The current limitation is that NetWorker doesn’t quite like the identity of the virtual drives – it sees them all as having the same serial number, and prohibits creating multiple drives with the same serial number. (The VTL presents differing serial numbers, but NetWorker may be working on the WWNN, which is the same on each device…)
Getting the VTL installed and configured
Limit yourself to one drive though, and you’re fine. To get started, you first need to download the VTL code – Mark hosts it at linuxvtl.googlepages.com.
My testing was with the 2009-06-09 tar ball on a CentOS 5.3 virtual machine and NetWorker 7.5.1. I’m not going to repeat the installation instructions – I suggest you build the RPMs, install sg3_util package (required), following the instructions included in Mark’s package.
Before you actually configure a jukebox in NetWorker, you need to strip down the number of devices in the VTL to 1, and the instructions below are geared towards that. Assuming you’ve not yet started the VTL software:
Create /etc/mhvtl/device.conf
Marks’ /etc/init.d/mhvtl startup script will create this file if it doesn’t exist, but we want to manually configure the file to only device. Below is the device.conf file I’ve used:
[root@tara mhvtl]# cat device.conf
VERSION: 2
# VPD page format:
# <page #> <Length> <x> <x+1>... <x+n>
# NOTE: The order of records is IMPORTANT...
# The 'Unit serial number:' should be last (except for VPD data)
# i.e.
# Order is : Vendor ID, Product ID, Product Rev and serial number finally
# Zero, one or more VPD entries.
#
# Each 'record' is sperated by one (or more) blank lines.
# Each 'record' starts at column 1
Library: 0 CHANNEL: 0 TARGET: 0 LUN: 0
Vendor identification: STK
Product identification: L700
Product revision level: 5500
Unit serial number: XYZZY
Drive: 1 CHANNEL: 0 TARGET: 1 LUN: 0
Vendor identification: QUANTUM
Product identification: SDLT600
Product revision level: 5500
Unit serial number: ZF7584364
Max density: 0x46
VPD: b0 04 00 02 01 00
(Note – yes, you can specify the serial number above, but no, if you create a second device with a different serial number it doesn’t yet work.)
Defining the library contents
After creating the device config, you need to configure the library contents – this is done by creating the file /etc/mhvtl/library_contents. Mine looks like the following:
[root@tara mhvtl]# cat library_contents
# Define how many tape drives you want in the vtl..
# The 'XYZZY_...' is the serial number assigned to
# this tape device.
Drive 1: ZF7584364
# Place holder for the robotic arm. Not really used.
Picker 1:
# Media Access Port
# (mailslots, Cartridge Access Port, <insert your favourate name here>)
# Again, define how many MAPs this vtl will contain.
MAP 1:
MAP 2:
MAP 3:
MAP 4:
# And the 'big' on, define your media and in which slot contains media.
# When the rc script is started, all media listed here will be created
# using the default media capacity.
Slot 1: 800843S3
Slot 2: 800844S3
Slot 3: 800845S3
Slot 4: 800846S3
Slot 5: 800847S3
Slot 6: 800848S3
Slot 7: 800849S3
Slot 8: 800850S3
Slot 9: 800851S3
Slot 10: 800852S3
Slot 11: 800853S3
Slot 12: 800854S3
Slot 13: 800855S3
Slot 14: 800856S3
Slot 15: 800857S3
Slot 16: 800858S3
Slot 17: 800859S3
Slot 18: 800860S3
Slot 19: 800861S3
Slot 20: 800862S3
Slot 21:
Slot 22:
Slot 23:
Slot 24:
Slot 25:
Slot 26:
Slot 27:
Slot 28:
Slot 29:
Slot 30:
Slot 31: CLN001L1
Slot 32: CLN002L1
In the above configuration, we’ve got a library with 32 presented slots, with slots 1-20 occupied by writable tapes, and slots 31-32 occupied with cleaning cartridges. Feel free to manipulate the numbers as you wish. (If you’re wondering about the choice of barcode labels, I’m terribly predictable. Every time I start a sequence of barcode labels in examples, I always start with 800843.)
Getting NetWorker and the VTL working together
Once the VTL has been configured, start it using the init script:
[root@tara mhvtl]# /etc/init.d/mhvtl start
vtllibrary process PID is 5315
So long as everything is working, you should see processes along the lines of:
[root@tara mhvtl]# ps -eaf | grep vtl
vtl 5310 1 0 08:56 ? 00:00:00 vtltape -q 1
vtl 5315 1 0 08:56 ? 00:00:00 vtllibrary -q 0
Looking in /opt/vtl, the default location for the VTL data, you should see the following files (suitably adjusted for any changes you make to barcodes/contents):
[root@tara mhvtl]# ls /opt/vtl
800843S3 800845S3 800847S3 800849S3 800851S3 800853S3
800855S3 800857S3 800859S3 800861S3 CLN001L1 800844S3
800846S3 800848S3 800850S3 800852S3 800854S3 800856S3
800858S3 800860S3 800862S3 CLN002L1
If we check the NetWorker inquire output, we get the following*:
[root@tara mhvtl]# inquire -l
-l flag found: searching all LUNs, which may take over 10 minutes per adapter
for some fibre channel adapters. Please be patient.
scsidev@0.0.0:STK L700 5500|Autochanger (Jukebox), /dev/sg1
S/N: XYZZY
ATNN=STK L700 XYZZY
WWNN=5123456003030303
scsidev@0.1.0:QUANTUM SDLT600 5500|Tape, /dev/nst0
S/N: ZF7584364
ATNN=QUANTUM SDLT600 ZF7584364
WWNN=5123456003030303
Assuming you get inquire output like the above, you next need to create your tape library. Below is the output of jbconfig command:
[root@tara mhvtl]# jbconfig
Jbconfig is running on host tara.pmdg.lab (Linux 2.6.18-128.1.16.el5),
and is using tara.pmdg.lab as the NetWorker server.
1) Configure an AlphaStor Library.
2) Configure an Autodetected SCSI Jukebox.
3) Configure an Autodetected NDMP SCSI Jukebox.
4) Configure an SJI Jukebox.
5) Configure an STL Silo.
What kind of Jukebox are you configuring? [1] 2
14484:jbconfig: Scanning SCSI buses; this may take a while ...
Installing 'Standard SCSI Jukebox' jukebox - scsidev@0.0.0.
What name do you want to assign to this jukebox device? MHVTL
15814:jbconfig: Attempting to detect serial numbers on the jukebox and drives ...
15815:jbconfig: Will try to use SCSI information returned by jukebox to configure drives.
Turn NetWorker auto-cleaning on (yes / no) [yes]? yes
The following drive(s) can be auto-configured in this jukebox:
1> sdlt600 @ 0.1.0 ==> /dev/nst0
These are all the drives that this jukebox has reported.
To change the drive model(s) or configure them as shared or NDMP drives,
you need to bypass auto-configure. Bypass auto-configure? (yes / no) [no] no
Jukebox has been added successfully
The following configuration options have been set:
> Jukebox description to the control port and model.
> Autochanger control port to the port at which we found it.
> Networker managed tape autocleaning on.
> Barcode reading to on.
> Volume labels that match the barcodes.
> Slot intended to hold cleaning cartridge to 32. Please insure that a
cleaning cartridge is in that slot
> Number of times we will use a new cleaning cartridge to 5.
> Cleaning interval for the tape drives to 6 months.
You can review and change the characteristics of the autochanger and its
associated devices using the NetWorker Management Console.
Would you like to configure another jukebox? (yes/no) [no]no
Using the VTL with NetWorker
Once you’ve got the jukebox created, start up with a simple command – plain old nsrjb:
[root@tara mhvtl]# nsrjb
Jukebox MHVTL: (Ready to accept commands)
14118:nsrjb: No volumes found in the media database...continuing.
slot volume pool barcode volume id recyclable
1: -* 800843S3 -
2: -* 800844S3 -
3: -* 800845S3 -
4: -* 800846S3 -
5: -* 800847S3 -
6: -* 800848S3 -
7: -* 800849S3 -
8: -* 800850S3 -
9: -* 800851S3 -
10: -* 800852S3 -
11: -* 800853S3 -
12: -* 800854S3 -
13: -* 800855S3 -
14: -* 800856S3 -
15: -* 800857S3 -
16: -* 800858S3 -
17: -* 800859S3 -
18: -* 800860S3 -
19: -* 800861S3 -
20: -* 800862S3 -
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31: -* CLN001L1 -
32: Cleaning Tape (5 uses left) CLN002L1 -
*not registered in the NetWorker media data base
drive 1 (/dev/nst0) slot :
(Note that I ran that about 30 seconds after the jukebox was created, so it had already transitioned into the “Ready to accept commands” state.)
The VTL isn’t built for speed, but it’s still zippy enough for lab testing. Here’s the output from a verbose label command, with timestamps added:
[root@tara mhvtl]# date ; nsrjb -Lvvv -b Default -S 1; date
Sat Jul 11 09:09:53 EST 2009
setting verbosity level to `3'
Info: Preparing to load volume `-' from slot 1 into device `/dev/nst0'.
Info: Loading volume `-' from slot `1' into device `/dev/nst0'.
Info: Load sleep for 5 seconds.
Info: Performing operation `Verify label' on device `/dev/nst0'.
Info: Operation `Verify label' in progress on device `/dev/nst0'
Info: Cannot read the current volume label `Tape label read for volume
? in pool ?, is not recognised by Networker: Input/output error'.
Info: nsrmmgd assumes the volume is unlabeled and will write a new label.
Info: Performing operation `Label without mount' on device `/dev/nst0'.
Info: Operation `Label without mount' in progress on device `/dev/nst0'
Info: Label: `800843S3', pool: `Default', capacity: `<NULL>'.
Info: Performing operation `Eject' on device `/dev/nst0'.
Info: Operation `Eject' in progress on device `/dev/nst0'
Info: Eject sleep for 5 seconds.
Info: Preparing to unload volume `800843S3' from device `/dev/nst0' to slot 1.
Info: Unloading volume `800843S3' from device `/dev/nst0' to slot 1.
Info: Unload sleep for 5 seconds.
Sat Jul 11 09:10:33 EST 2009
Writing a backup, we get the obligitory screen-shot:
Screenshot showing backup to Mark's VTL
It’s still about recovery
Even though this is for lab usage only, we still need to make sure that what we write to virtual tape is what we get back. So after that backup, I ran a recovery, restoring the backup to another location. Performing checksums against the source and the original yielded:
[root@tara /]# md5sum /usr/share/doc/crash-4.0/README
/backup/recover_test/doc/crash-4.0/README
73568e4d9e09ce2847673dd5156cb571 /usr/share/doc/crash-4.0/README
73568e4d9e09ce2847673dd5156cb571 /backup/recover_test/doc/crash-4.0/README
Caveats
In conclusion, I’d like to offer a few caveats:
- In case I’ve not mentioned this enough, this is not a production VTL. Please don’t think I’m advocating it as a replacement to a full VTL.
- The VTL will let you backup as much as you want to any piece of media, so be careful with space management – it’s on your own head to manage media sizes, etc.
- The default placement of the VTL object files (i.e., media) is in /opt/vtl, which is likely to be in the root filesystem on an average Linux host. Thus, if you don’t keep an eye on media capacity, you’re going to overrun your root filesystem (or whatever filesystem the VTL data is stored in).
- You still need either a NetWorker autochanger license or to be running in eval mode to be able to use/configure this.
Again, if I haven’t said this enough – this is for lab testing.
[2009-07-13 Edit]
Proving that sometimes I just don’t read the documentation sufficiently, with a little bit more digging I discovered that Mark has also implemented a mktape command, that creates media with user nominated sizes. By stopping NetWorker, deleting the VTL media, recreating the media with the nominated sizes then restarting the VTL and NetWorker, you can control capacity using this VTL. Most importantly, that means you can simulate tape changes.
[2009-11-15]
See here for an update article covering multiple drive support, now that Mark has this working in a way which is compatible with NetWorker.
–
* Note – acknowledging I’ve adjusted the spacing slightly in the inquire output to ensure it fits on the average browser. That’s the only manipulation that was done though.
