{"id":667,"date":"2009-07-13T06:34:50","date_gmt":"2009-07-12T20:34:50","guid":{"rendered":"http:\/\/nsrd.wordpress.com\/?p=667"},"modified":"2018-12-12T15:57:42","modified_gmt":"2018-12-12T05:57:42","slug":"carry-a-jukebox-with-you-if-youre-using-linux","status":"publish","type":"post","link":"https:\/\/nsrd.info\/blog\/2009\/07\/13\/carry-a-jukebox-with-you-if-youre-using-linux\/","title":{"rendered":"Carry a jukebox with you (if you’re using Linux)"},"content":{"rendered":"

A rather smart gent whom I used to work with at another company, Mark Harvey, has in his own time been working on an open source VTL implementation that can be used for testing\/lab purposes.<\/em> I.e., he’s not aiming for it to be the next competitor to EDLs, FalconStor, etc., but rather, something that people can use when needing to test jukebox functionality without wanting to carry a jukebox around with them.<\/p>\n

While Mark has primarily focused on getting his VTL software working with NetBackup, recently he’s made some progress in getting it to work (with a couple of limitations) with NetWorker.<\/p>\n

The current limitation is that NetWorker doesn’t quite like<\/em> the identity of the virtual drives \u2013 it sees them all as having the same serial number, and prohibits creating multiple drives with the same serial number. (The VTL presents differing serial numbers, but NetWorker may be working on the WWNN, which is the same on each device…)<\/p>\n

Getting the VTL installed and configured<\/h3>\n

Limit yourself to one drive though, and you’re fine. To get started, you first need to download the VTL code \u2013 Mark hosts it at linuxvtl.googlepages.com<\/a>.<\/p>\n

My testing was with the 2009-06-09 tar ball on a CentOS 5.3 virtual machine and NetWorker 7.5.1. I’m not going to repeat the installation instructions \u2013 I suggest you build the RPMs, install sg3_util<\/em> package (required), following the instructions included in Mark’s package.<\/p>\n

Before you actually configure a jukebox in NetWorker, you need to strip down the number of devices in the VTL to 1, and the instructions below are geared towards that. Assuming you’ve not yet started the VTL software:<\/p>\n

Create \/etc\/mhvtl\/device.conf<\/strong><\/p>\n

Marks’ \/etc\/init.d\/mhvtl startup script will create this file if it doesn’t exist, but we want to manually configure the file to only device. Below is the device.conf<\/em> file I’ve used:<\/p>\n

[root@tara mhvtl]# cat device.conf\n\nVERSION: 2\n\n# VPD page format:\n# <page #> <Length> <x> <x+1>... <x+n>\n\n# NOTE: The order of records is IMPORTANT...\n# The 'Unit serial number:' should be last (except for VPD data)\n# i.e.\n# Order is : Vendor ID, Product ID, Product Rev and serial number finally\n# Zero, one or more VPD entries.\n#\n# Each 'record' is sperated by one (or more) blank lines.\n# Each 'record' starts at column 1\n\nLibrary: 0 CHANNEL: 0 TARGET: 0 LUN: 0\n Vendor identification: STK\n Product identification: L700\n Product revision level: 5500\n Unit serial number: XYZZY\n\nDrive: 1 CHANNEL: 0 TARGET: 1 LUN: 0\n Vendor identification: QUANTUM\n Product identification: SDLT600         \n Product revision level: 5500\n Unit serial number: ZF7584364         \n Max density: 0x46\n VPD: b0 04 00 02 01 00<\/pre>\n
(Note \u2013 yes, you can specify the serial number above, but no, if you create a second device with a different serial number it doesn’t yet work.)<\/p>\n
Defining the library contents<\/strong><\/p>\n
After creating the device config, you need to configure the library contents \u2013 this is done by creating the file \/etc\/mhvtl\/library_contents. Mine looks like the following:<\/p>\n
[root@tara mhvtl]# cat library_contents\n# Define how many tape drives you want in the vtl..\n# The 'XYZZY_...' is the serial number assigned to\n# this tape device.\n\nDrive 1: ZF7584364\n\n# Place holder for the robotic arm. Not really used.\nPicker 1:\n\n# Media Access Port\n# (mailslots, Cartridge Access Port, <insert your favourate name here>)\n# Again, define how many MAPs this vtl will contain.\nMAP 1:\nMAP 2:\nMAP 3:\nMAP 4:\n\n# And the 'big' on, define your media and in which slot contains media.\n# When the rc script is started, all media listed here will be created\n# using the default media capacity.\nSlot 1:    800843S3\nSlot 2: 800844S3\nSlot 3: 800845S3\nSlot 4: 800846S3\nSlot 5: 800847S3\nSlot 6: 800848S3\nSlot 7: 800849S3\nSlot 8: 800850S3\nSlot 9: 800851S3\nSlot 10: 800852S3\nSlot 11: 800853S3\nSlot 12: 800854S3\nSlot 13: 800855S3\nSlot 14: 800856S3\nSlot 15: 800857S3\nSlot 16: 800858S3\nSlot 17: 800859S3\nSlot 18: 800860S3\nSlot 19: 800861S3\nSlot 20: 800862S3\nSlot 21:\nSlot 22:\nSlot 23:\nSlot 24:\nSlot 25:\nSlot 26:\nSlot 27:\nSlot 28:\nSlot 29:\nSlot 30:\nSlot 31: CLN001L1\nSlot 32: CLN002L1<\/pre>\n
In the above configuration, we’ve got a library with 32 presented slots, with slots 1-20 occupied by writable tapes, and slots 31-32 occupied with cleaning cartridges. Feel free to manipulate the numbers as you wish. (If you’re wondering about the choice of barcode labels, I’m terribly predictable. Every time I start a sequence of barcode labels in examples, I always start with 800843.)<\/p>\n
Getting NetWorker and the VTL working together<\/h3>\n
Once the VTL has been configured, start it using the init script:<\/p>\n
[root@tara mhvtl]# \/etc\/init.d\/mhvtl start\nvtllibrary process PID is 5315<\/pre>\n
So long as everything is working, you should see processes along the lines of:<\/p>\n
[root@tara mhvtl]# ps -eaf | grep vtl\nvtl       5310     1  0 08:56 ?        00:00:00 vtltape -q 1\nvtl       5315     1  0 08:56 ?        00:00:00 vtllibrary -q 0<\/pre>\n
Looking in \/opt\/vtl, the default location for the VTL data, you should see the following files (suitably adjusted for any changes you make to barcodes\/contents):<\/p>\n
[root@tara mhvtl]# ls \/opt\/vtl\n800843S3  800845S3  800847S3  800849S3  800851S3  800853S3\n800855S3  800857S3  800859S3  800861S3  CLN001L1  800844S3\n800846S3  800848S3  800850S3  800852S3  800854S3  800856S3\n800858S3  800860S3  800862S3  CLN002L1<\/pre>\n
If we check the NetWorker inquire<\/em> output, we get the following*:<\/p>\n
[root@tara mhvtl]# inquire -l\n\n-l flag found: searching all LUNs, which may take over 10 minutes per adapter\n for some fibre channel adapters.  Please be patient.\n\nscsidev@0.0.0:STK     L700     5500|Autochanger (Jukebox), \/dev\/sg1\n                                    S\/N:    XYZZY     \n                                    ATNN=STK     L700            XYZZY     \n                                    WWNN=5123456003030303\nscsidev@0.1.0:QUANTUM SDLT600  5500|Tape, \/dev\/nst0\n                                    S\/N:    ZF7584364\n                                    ATNN=QUANTUM SDLT600         ZF7584364\n                                    WWNN=5123456003030303<\/pre>\n
Assuming you get inquire output like the above, you next need to create your tape library. Below is the output of jbconfig command:<\/p>\n
[root@tara mhvtl]# jbconfig\n\nJbconfig is running on host tara.pmdg.lab (Linux 2.6.18-128.1.16.el5),\n and is using tara.pmdg.lab as the NetWorker server.\n\n 1) Configure an AlphaStor Library.\n 2) Configure an Autodetected SCSI Jukebox.\n 3) Configure an Autodetected NDMP SCSI Jukebox.\n 4) Configure an SJI Jukebox.\n 5) Configure an STL Silo.\n\nWhat kind of Jukebox are you configuring? [1] 2\n14484:jbconfig: Scanning SCSI buses; this may take a while ...\nInstalling 'Standard SCSI Jukebox' jukebox - scsidev@0.0.0.\n\nWhat name do you want to assign to this jukebox device? MHVTL\n15814:jbconfig: Attempting to detect serial numbers on the jukebox and drives ...\n\n15815:jbconfig: Will try to use SCSI information returned by jukebox to configure drives.\n\nTurn NetWorker auto-cleaning on (yes \/ no) [yes]? yes\n\nThe following drive(s) can be auto-configured in this jukebox:\n 1> sdlt600 @ 0.1.0 ==> \/dev\/nst0\nThese are all the drives that this jukebox has reported.\n\nTo change the drive model(s) or configure them as shared or NDMP drives,\n you need to bypass auto-configure. Bypass auto-configure? (yes \/ no) [no] no\n\nJukebox has been added successfully\n\nThe following configuration options have been set:\n\n> Jukebox description to the control port and model.\n> Autochanger control port to the port at which we found it.\n> Networker managed tape autocleaning on.\n> Barcode reading to on.\n> Volume labels that match the barcodes.\n> Slot intended to hold cleaning cartridge to 32.  Please insure that a\n cleaning cartridge is in that slot\n> Number of times we will use a new cleaning cartridge to 5.\n> Cleaning interval for the tape drives to 6 months.\n\nYou can review and change the characteristics of the autochanger and its\n associated devices using the NetWorker Management Console.\n\nWould you like to configure another jukebox? (yes\/no) [no]no<\/pre>\n
Using the VTL with NetWorker<\/h3>\n
Once you’ve got the jukebox created, start up with a simple command – plain old nsrjb:<\/p>\n
[root@tara mhvtl]# nsrjb\n\nJukebox MHVTL: (Ready to accept commands)\n14118:nsrjb: No volumes found in the media database...continuing.\nslot  volume                      pool  barcode   volume id  recyclable\n 1: -*                                  800843S3  -                    \n 2: -*                                  800844S3  -                    \n 3: -*                                  800845S3  -                    \n 4: -*                                  800846S3  -                    \n 5: -*                                  800847S3  -                    \n 6: -*                                  800848S3  -                    \n 7: -*                                  800849S3  -                    \n 8: -*                                  800850S3  -                    \n 9: -*                                  800851S3  -                    \n10: -*                                  800852S3  -                    \n11: -*                                  800853S3  -                    \n12: -*                                  800854S3  -                    \n13: -*                                  800855S3  -                    \n14: -*                                  800856S3  -                    \n15: -*                                  800857S3  -                    \n16: -*                                  800858S3  -                    \n17: -*                                  800859S3  -                    \n18: -*                                  800860S3  -                    \n19: -*                                  800861S3  -                    \n20: -*                                  800862S3  -                    \n21:                                                                        \n22:                                                                        \n23:                                                                        \n24:                                                                        \n25:                                                                        \n26:                                                                        \n27:                                                                        \n28:                                                                        \n29:                                                                        \n30:                                                                        \n31: -*                                  CLN001L1  -                    \n32: Cleaning Tape (5 uses left)         CLN002L1  -                    \n *not registered in the NetWorker media data base\n\ndrive 1 (\/dev\/nst0) slot   :<\/pre>\n
(Note that I ran that about 30 seconds after the jukebox was created, so it had already transitioned into the “Ready to accept commands” state.)<\/p>\n
The VTL isn’t built for speed, but it’s still zippy enough for lab testing. Here’s the output from a verbose label command, with timestamps added:<\/p>\n
[root@tara mhvtl]# date ; nsrjb -Lvvv -b Default -S 1; date\nSat Jul 11 09:09:53 EST 2009\nsetting verbosity level to `3'\nInfo: Preparing to load volume `-' from slot 1 into device `\/dev\/nst0'.\nInfo: Loading volume `-' from slot `1' into device `\/dev\/nst0'.\nInfo: Load sleep for 5 seconds.\nInfo: Performing operation `Verify label' on device `\/dev\/nst0'.\nInfo: Operation `Verify label' in progress on device `\/dev\/nst0'\nInfo: Cannot read the current volume label `Tape label read for volume\n ? in pool ?, is not recognised by Networker: Input\/output error'.\nInfo: nsrmmgd assumes the volume is unlabeled and will write a new label.\nInfo: Performing operation `Label without mount' on device `\/dev\/nst0'.\nInfo: Operation `Label without mount' in progress on device `\/dev\/nst0'\nInfo: Label: `800843S3', pool: `Default', capacity: `<NULL>'.\nInfo: Performing operation `Eject' on device `\/dev\/nst0'.\nInfo: Operation `Eject' in progress on device `\/dev\/nst0'\nInfo: Eject sleep for 5 seconds.\nInfo: Preparing to unload volume `800843S3' from device `\/dev\/nst0' to slot 1.\nInfo: Unloading volume `800843S3' from device `\/dev\/nst0' to slot 1.\nInfo: Unload sleep for 5 seconds.\nSat Jul 11 09:10:33 EST 2009<\/pre>\n
Writing a backup, we get the obligitory screen-shot:<\/p>\n
\"Screenshot
Screenshot showing backup to Mark's VTL<\/figcaption><\/figure>\n
It’s still about recovery<\/h3>\n
Even though this is for lab usage only, we still need to make sure that what we write to virtual tape is what we get back. So after that backup, I ran a recovery, restoring the backup to another location. Performing checksums against the source and the original yielded:<\/p>\n
[root@tara \/]# md5sum \/usr\/share\/doc\/crash-4.0\/README\n\/backup\/recover_test\/doc\/crash-4.0\/README\n73568e4d9e09ce2847673dd5156cb571  \/usr\/share\/doc\/crash-4.0\/README\n73568e4d9e09ce2847673dd5156cb571  \/backup\/recover_test\/doc\/crash-4.0\/README<\/pre>\n
Caveats<\/h3>\n
In conclusion, I’d like to offer a few caveats:<\/p>\n
    \n
  1. In case I’ve not mentioned this enough, this is not a production VTL<\/em>. Please don’t think I’m advocating it as a replacement to a full VTL.<\/li>\n
  2. The VTL will let you backup as much as you want to any piece of media, so be careful with space management \u2013 it’s on your own head to manage media sizes, etc.<\/li>\n
  3. The default placement of the VTL object files (i.e., media) is in \/opt\/vtl, which is likely to be in the root filesystem on an average Linux host. Thus, if you don’t keep an eye on media capacity, you’re going to overrun your root filesystem (or whatever filesystem the VTL data is stored in).<\/li>\n
  4. You still need either a NetWorker autochanger license or to be running in eval mode to be able to use\/configure this.<\/li>\n<\/ol>\n
    Again, if I haven’t said this enough – this is for lab testing.<\/strong><\/p>\n
    [2009-07-13 Edit]<\/strong><\/em><\/p>\n
    Proving that sometimes I just don’t read the documentation sufficiently, with a little bit more digging I discovered that Mark has also implemented a mktape<\/em> command, that creates media with user nominated sizes. By stopping NetWorker, deleting the VTL media, recreating the media with the nominated sizes then restarting the VTL and NetWorker, you can control capacity using this VTL. Most importantly, that means you can simulate tape changes.<\/p>\n
    [2009-11-15]<\/em><\/strong><\/p>\n
    See here for an update article<\/a> covering multiple drive support, now that Mark has this working in a way which is compatible with NetWorker.<\/p>\n
    —
    \n* Note – acknowledging I’ve adjusted the spacing slightly in the inquire output to ensure it fits on the average browser. That’s the only manipulation that was done though.<\/p>\n","protected":false},"excerpt":{"rendered":"
    A rather smart gent whom I used to work with at another company, Mark Harvey, has in his own time…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[4,16],"tags":[517,518,698,994,1101],"class_list":["post-667","post","type-post","status-publish","format-standard","hentry","category-aside","category-networker","tag-lab","tag-lab-testing","tag-open-source","tag-testing","tag-vtl"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pKpIN-aL","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/667","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/comments?post=667"}],"version-history":[{"count":1,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/667\/revisions"}],"predecessor-version":[{"id":7637,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/posts\/667\/revisions\/7637"}],"wp:attachment":[{"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/media?parent=667"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/categories?post=667"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nsrd.info\/blog\/wp-json\/wp\/v2\/tags?post=667"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}