Index

Find us on facebook

License:

GPL v2
Q. Why GPL license.
A. Because I like the idea of sharing.
I don't profess to know everything (not even close). Having something I can share and allow others to share back is a good idea.

Virtual Tape Library consists of several components.

  • LLD - A low level driver implemented as a kernel module - mhvtl
  • Target devices - Daemons vtltape(1) and vtllibrary(1) which implement SCSI target device(s) in user-space
  • Utility commands mktape(1), vtlcmd(1)
  • And startup scripts build_library_config(1), make_vtl_devices(1)

The kernel module is based on the scsi_debug kernel module (http://sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note: As of 0.16, there are no default devices.

The support scripts will add Drives (SSC devices) & library (SMC) depending on the contents of /etc/mhvtl/library_contents and /etc/mhvtl/device.conf.

A char device back-end has been included with the vtl LLD driver This allows data and SCSI commands to be passed from the LLD to user-mode daemons (SCSI targets) which constently poll the driver and process any outstanding SCSI commands.

vtltape

vtltape(1) is the usermode SSC target daemon which writes/reads data to data files in the /opt/mhvtl directory (if a virtual tape has been loaded). The virtual tape files include a Medium Auxiliary Memory (MAM) data structure to store persistent data (number of tape loads, total data written/read, media type etc).

vtllibrary

vtllibrary(1) is the usermode SMC target daemon which reads its configuration from the file /etc/mhvtl/library_contents(5) at startup. The number of storage slots are built dynamically when the daemon starts. Hence changing the number of storage slots and media access slots are a matter of modifying the file contents and restarting the vtllibrary(1) daemon. All 'library' commands are performed on data structures in memory ONLY.

vtlcmd

A utility vtlcmd(1) is used to administrator the daemons vtltape(1) and vtllibrary.
Message queue (key 0x4d61726b) is used to pass messages between vtlcmd(1), vtllibrary(1) and vtltape(1)

When a SCSI 'move medium' from a storage slot to a tape drive is requested, the media location is updated in vtllibrary(1) data structures, and the barcode of the media id is passed via the message queue to the vtltape(1) daemon in question. A file open of the barcode is attempted in the /opt/mhvtl directory and if successful, the vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit Ready requests. An unload SCSI command to the tape drive will close the data file.

Media can be moved out of the VTL via the Media Access Port. Once media is logically moved into the MAP slots, the MAP entries can be cleared using the vxcmd:

# vtlcmd library empty map

The media can be 'moved back again' by re-starting the VTL user-space daemons:

Media can be loaded into the MAP via a :

# vtlcmd library load map <barcode>
or restart the vtl
# /etc/init.d/mhvtl restart

TapeAlert

TapeAlert flags can be set using the vtlcmd(1)

e.g. vtlcmd [index] TapeAlert [flags]

Where index is the message Q offset associated with the drive (or the string 'library').

e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:
Pull out the binary to decimal calculator
1000 0000 0000 0000 0000 (20 bits) => hex => 80000

# vtlcmd 1 TapeAlert 80000

A listing of TapeAlert flags can be found at t10 home page for SSC devices Annex A.

Examples

mhvtl is registered as HBA #3:
# lsscsi -g
[0:0:0:0] disk MAXTOR ATLAS10K4_36SCA DFM0 /dev/sda /dev/sg0
[0:0:6:0] process PE/PV 1x3 SCSI BP 1.1 - /dev/sg1
[2:0:0:0] disk SEAGATE ST336607FC 0003 /dev/sdb /dev/sg2
[2:0:1:0] disk SEAGATE ST336607FC 0003 /dev/sdc /dev/sg3
[2:0:2:0] mediumx ATL 1500 6.0 - /dev/sg4
[2:0:2:1] tape QUANTUM SuperDLT1 2323 /dev/st0 /dev/sg5
[2:0:2:2] tape QUANTUM SuperDLT1 2323 /dev/st1 /dev/sg6
[2:0:2:3] process ATTO 310-LV 1.42 - /dev/sg7
[3:0:0:0] mediumx STK L700 vtl0 - /dev/sg8
[3:0:0:1] tape SONY SDX-900V 5400 /dev/st2 /dev/sg9
[3:0:0:2] tape SONY SDX-900V 5400 /dev/st3 /dev/sg10
[3:0:0:3] tape QUANTUM SDLT600 5400 /dev/st4 /dev/sg11
[3:0:0:4] tape QUANTUM SDLT600 5400 /dev/st5 /dev/sg12
[3:0:0:5] tape QUANTUM SDLT600 5400 /dev/st6 /dev/sg13
[3:0:0:6] tape IBM ULT3580-TD3 5400 /dev/st7 /dev/sg14
[3:0:0:7] tape IBM ULT3580-TD3 5400 /dev/st8 /dev/sg15
[3:0:0:8] tape IBM ULT3580-TD3 5400 /dev/st9 /dev/sg16

Long term goals

Eventually, the functionality will be moved across to the SCSI Target Framework (stgt) http://stgt.berlios.de/

The stgt is a very nicely designed SCSI taret framwork which can support all target types along with multiple SCSI transports in a very structured way.

However, until time permits (and my c coding standards improve), features and improvements on this code base will continue.

Howto get help

Somebody has put up a discussion board (and has lots of great documentation on working with different backup software)
http://mhvtl-community-forums.966029.n3.nabble.com/ - Community Forums

http://www.facebook.com/pages/mhVTL/119540494812584 - Facebook

Howto help out

There are many way of providing assistance

  • Testing with backup software X and reporting success/failure. If possible, test any patches that may result of testing.
  • If you are willing, patches are very welcome.
    Please read the Linux Documentation/CodingStyle
    Note: I only discovered this documentation guide recently, so patches to update old code style to the CodingStyle are also welcome.
    Please run any patches thru the Linux 'mhvtl-<version>/scripts/checkpatch.pl' for coding style sanity check before sending.
  • Pull the source from git repository (github).. Tinker till your hearts content :)

Emulations

Currently, only the tape module has 'personality modules'

For up-to-date view of emulations review vtltape.c and search for "static struct tape_drives_table"

list of tape emulations (as of v1.4-9)

Working emulations include:

  • IBM LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)
  • HP LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)
  • IBM 03592-J1A, 03592-E05 & 03592-E06
  • STK 9840 (A/B/C/D), 9940 (A/B)
  • STK T10000 (A/B/C)
  • Sony AIT (1/2/3 &4)

What about model X....

  • Quantum (DLT/Super-DLT) is NOT supported as their SCSI Programmer's Guide does not document the expected returned data.
  • DDS (4mm DAT) - I've not found any SCSI programmers' Guides for these drives.

Library emulation is coming in the future.

STK L180/L700 emulation works for NetBackup & NetWorker.

TSM - IBM library type needed to use LTO drives.

Note: Please let me know what works for which version of backup software so I can better document it.

Downloads

Confused as to what to get ?

If you are a RedHat based distribution (RedHat / CentOS / Oracle / Scientific Linux etc) than ELrepo may be your best bet. Checkout : http://elrepo.org/tiki/tiki-index.php

Follow the 'Getting Started' section

Import the public key:

Install ELRepo for RHEL-5, SL-5 or CentOS-5 / install ELRepo for RHEL-6, SL-6 or CentOS-6

- $ sudo yum install mhvtl-utils kmod-mhvtl

Otherwise:
 Check what sort of linux distribution.
 Choose 'src.rpm & <platform>.rpm if RPM based and want to use mhVTL in fastest way.
 Choose 'source code tar ball' if using a non-RPM based linux distribution. (e.g. Debian/Ubuntu etc)
 Choose 'Download using git' if your interested in fixing bugs, following development history, as well as using/testing mhVTL

 There are 2 "bits" that make up the 'mhVTL'.
  - User space daemons
  - mhvtl pseudo scsi hba kernel driver.

The kernel module source code is included with any of the 'source' packages (& git) below and needs to be downloaded and built to suit the currently running linux kernel on your linux host.

Download one or more to get going and playing as soon as possible:
package
 Description
mhvtl-2014-04-13.tgz Source code in a tar ball - not required for RPM based distributions
Contains both user space and kernel source code
mhvtl-utils-1.5-0.x86_64.rpm 64bit pre-built binaries of 'user space' portion of mVTL
mhvtl-utils-1.5-0.i586.rpm 32bit pre-built binaries of 'user space' portion of mVTL
mhvtl-utils-1.5-0.src.rpm
Above source code tar ball packaged as a source rpm
git tarball Latest source tree (retrieved via git) as a 'tar ball'

Download using 'git' - This provides a complete history of all changes.

Create a directory where you are going to store the source code.
$ mkdir mhvtl

Initializethe directory as a git repository
$ git init

Pull the source code (it's only a few hundred k)
$ git pull https://github.com/markh794/mhvtl.git

From time to time, check for any updates and synchronize from github.
$ git pull https://github.com/markh794/mhvtl.git

  ===========

Change log

Now that the 1.0 milestone has been reached, any changes to internal functionality will increment the 'x' (dot release) e.g. 1.x
Bug fixes will result in an incremental (double-dot) release. e.g. 1.1-x

Kevan Rehm has re-written the on-disk data structure.
(It's amazing how much effort can be rolled up in one simple sentence).

Each 'tape' is now stored under its own subdirectory /opt/mhvtl/<barcode>

A tape consists of three files.
- Data file => Container of data
- Index file => Contains an array of header structures (pointers into the data file)
- Meta file => Contains the MAM structure, followed by a meta data header, followed by a variable-length array of filemark block numbers.

mhvtl-2014-04-13.tgz / mhvtl-1.5-0:

    - Introduce personality modules for tape libraries
   
- Bug fix with libraries configured with more than 1260 slots
   
- New media type 'NULL' for performance testing
    - Change kernel module bus type from 'pseudo' to 'pseudo9' to prevent address space conflict with scsi_debug

mhvtl-2013-10-20.tgz / mhvtl-1.4-10:

    - Various cleanups/bug fixes as a result of the static code check 'smatch'
    - Fix TapeAlert bit offsets
    - Return bit/byte offset for sense INVALID FIELD IN CDB/PARAMS
    - Fix returned data size when 0 data is requested (kernel module fix)
    - SMC - Ability to keep library media change persistant across restarts
    - SSC - Fix op code 0Fh (read reverse) incorrectly set to 'reserve'
    - SSC - Implement 'read media serial number' op code
    - Fix 'make_vtl_media' script to understand IBM 03592 media type

mhvtl-2013-08-29.tgz / mhvtl-1.4-9:

- MODE SELECT for SSC (tape) overhaul
- Sanitize delay values introduced with previous release
- Code cleanup based on Valgrind - memory leaks and unused file handles
- Set SCSI Revision field for T10000B emulation
- Adjust SCSI Inquiry information for 'Scalar' libraries (add 'BarC' field, pad 'Full Firmware Revision' field with compile time/date information)

mhvtl-2013-06-29.tgz / mhvtl-1.4-8:
- Add additional fields in MAM for multi-partition support capability
- MODE SELECT fixes
- Solaris 11.1 st mode select
- Make virtual media more resilient against disk full conditions
- Add additional error checking between kernel and user-space ioctl
- Add ability to introduce delays in load/unload/position/thread/rewind op codes
- Add 16byte CDB locate command support
- Minor kernel module tweaks - increase timeout & error checking on ioctl()

mhvtl-2013-01-31.tgz / mhvtl-1.4-6:
- Fix crash when position back xx blocks
- Add 'edit_tape' utility. Update virtual media 'metadata'
- Fix TAPE CAPACITY log page - return correct data page and fix buffer overflow

mhvtl-2013-01-12.tgz / mhvtl-1.4-5:
- Fix End-Of-Media (EOM) handling (reported write failure actually successfully wrote the block - resulting in an off-by-one block count)
- Add LTO-6 emulation
- Fix TAPE CAPACITY log page information (Corrected a byte swap error)

mhvtl-2012-09-13.tgz / mhvtl-1.4-4:
- Fix bug (off-by-one) in LOG SELECT
- Implement counter reset via LOG SELECT & PCR bit
- Implement 'performance' counter - Records amount of time (spent in usleep()) since last op code processed.

- Note: Due to build issues, mhvtl-1.4-2 & mhvtl-1.4.3 never made public release.

mhvtl-2012-08-08.tgz / mhvtl-1.4-1:
-
Bug fixes
     Fix dump_tape to understand new library subdirectory format
     Fix vtlcmd to understand new library subdirectory format

mhvtl-2012-08-01.tgz / mhvtl-1.4-0:

- New features
          vtltape: Make the backoff algrithm value configurable
          Default media belonging to a library in a subdir
          Add default "Home directory" entry in device.conf

- Personality module updates
          Allow 'default' module to load all media R/W
          Add IBM half-hight INQUIRY string match
          vtltape: Update log entry regarding media capacity depending on capabiliti
          Initial commit of SDLT600 personality module
          Initial commit of SDLT320 personality module
          Update mktape to work 'better' with SDLT media types

- Cleanups
          Move standard inquiry info into memory structure
          mhvtl.ko: Silence install 'depmod' error
          Standardize on PCODE_OFFSET for inquiry page code mask
          Remove unused variable 'removable'
          Improve logging of mode page creation

- Packaging:
          Remove kmod-mhvtl.spec
          Rename 'mhvtl' package as 'mhvtl-utils'
          Use ELRepo spec file
          Move 'make install' responsibility from RPM to Makefile

mhvtl-2012-06-15.tgz / mhvtl-1.3-0:
- Move on-disk functions into its own shared lib (libvtlcart.so)
- Improve IBM & HP LTO emulation of Inquiry VPD pages
- Add ability to call external commands when moving media
- Toggle APPEND ONLY mode from user-space via 'vtlcmd'

mhvtl-2012-04-04.tgz / mhvtl-1.2-3:
- Fix segfault when using AIT4 emulation and Security Protocol IN/OUT was attempted
- Add more descriptive log messages for LOG & MODE pages

mhvtl-2012-03-22.tgz / mhvtl-1.2-2:
- Changed defaults so fifo is not enabled. If fifo not used, it will
  block writing process (i.e. hang daemons)
- Improved MODE SELECT / SENSE to support 'changeable field bitmaps
- Started implementing ALLOW OVERWRITE support (incomplete)
- Fixed INQUIRY string match for STK T10000A

mhvtl-2012-01-21.tgz / mhvtl-1.2-1:
 - Add selectable compression type lzo or existing zlib
 - Add timestamps to 'state notification' messages

note: These changes mean you need lzo-devel package installed to compile 1.2-x

mhvtl-2012-01-20.tgz / mhvtl-1.2-0:

Never released - memory leak bug identified before posting.

mhvtl-2011-12-24.tgz / mhvtl-1.1-1:
 - Clean up IPC resources when finished with them
 - Simplify fifo reference counting
 - Start including Scientific Linux patches (Still need to include kmod RPM)
 - Fix 'make install' on Ubuntu
 - Various log message cleanups

mhvtl-2011-10-09.tgz / mhvtl-1.1-0:
 -
Add 'real time notification' via a named pipe.
   vtltape & vtllibrary now take an optional '-f <named pipe>'
   Any chance of state within the daemon will be sent to the named pipe ready for use-application
   consumption.
- Log Sense page 0x0c (SEQUENTIAL ACCESS DEVICE PAGE) now updates wit valid data.
- MODE SENSE for LTO devices now update the 'media type' field as typical with real drives.

Just in time for fathers day a jump from 0.18-17 to 1.0...

mhvtl-2011-09-11.tgz / mhvtl-1.0-1:
  - Added mode page 25h (vendor specific) for IBM LTO3/4/5
    This allows the Windows IBM Tape Driver to load correctly (connected via iSCSI)
  - Attempt to correct file permission/ownership when media is created manually
  - A HUP signal to the vtllibrary daemon will cause it to re-read its config
    from the /etc/mhvtl/library_contents.XX file. (So you can change the slot
    config without having to re-start the daemon).

Note: Two current known bugs which are being worked on.
- If the '-f fifo' is not correctly defined or can't be created, the daemons will segfault on exit.
- The first daemon will result in an unlink(fifo); on exit.

mhvtl-2011-09-04.tgz / mhvtl-1.0-0:
Improvements
  - Re-worked MODE SENSE/SELECT data structures into a linked list.
  - Re-worked LOG SENSE/SELECT data structures into a linked list.
  - Added support for sub-page MODE information.
  - Added STK 9x40 drive emulations.
  - Removed dead code from kernel driver.
  - Add working REPORT DENSITY support to the SSC.
      This change required the personality modules to define the list of support
      densities. The media density is defined at 'mktape' time.
      Hence, the optional 'media load' rules in device.conf are now redundant
      and not used.
      BUT: The option to load any media into any drive is also gone.

mhvtl-2011-06-25.tgz / mhvtl-0.18-17:
Improvements
  - Kernel module compile warning since 2.6.33
Fixes
  - Return correct sense from SPACE op code
  - Fix buffer overflow in vtllibrary : product_id[]
  - Test MAP port status before moving media in/out

Note: This version has been successfully tested with:
 - NetBackup 6.x, 7.0 & 7.1 (has always worked)
 - BackupExec 10.5 (Failed to span media in earlier versions of mhvtl)
 - ArcServ r15 SP1 (failed with earlier versions of mhvtl)

mhvtl-2011-05-22.tgz / mhvtl-0.18-16:
Improvements
 - Cleaning media behaves more like real (IBM LTO4) drive.
 - Implement OPEN/CLOSE IMPORT/EXPORT element OP code (Thanks Sebastian)
 - Kernel module support for 2.6.39 (Thanks Sebastian)
Fixes
  - SPACE op code - Space to end-of-data fixed (Thanks Sebastian)
    This fixes an issue triggered using Oracle Backup
  - REQUEST SENSE - Return correct data.
  - SPOUT - Return check_condition on some error paths
  - Cleaning Media - Return 'not ready' instead of 'ready' when loaded
  - Don't write FILEMARKS to WORM or Cleaning media (as appropriate)

mhvtl-2011-05-04.tgz / mhvtl-0.18-15:
Improvements
  - Implemented 'Personality Module' for each drive type
    (many cleanups due to 'PM' change fallout thanks to Sebastian)
  - Increase max barcode length to 16 chars
Fixes
  - Inquiry no longer incorrectly reports support for TrmIOP and CmdQue
  - Fixed block read/writes corruption if 'multiple blocks' specified
  - Fix Device Capabilities mode page - Don't advertise EXCHANGE MEDIUM support

mhvtl-2011-03-17.tgz / mhvtl-0.18-14:
Fixes:
  - kernel module build fixes for Linux kernel 2.6.37 and greater
Improvements
  - Add STK T10000C media/drive support
  - Always log 'fatal' errors
  - Cleanup to mhvtl.spec
  - Catch signals to prevent daemon terminating early.

mhvtl-2011-01-11.tgz / mhvtl-0.18-13:
Fix:
 - READ ELEMENT STATUS when initiator does not allocate enough memory for full list
 - Return ILLEGAL REQUEST for SCSI OP codes which are not supported.
Improvements:
 - Increase SENSE buffer from 38 to 96 bytes

mhvtl-2010-11-07.tgz / mhvtl-0.18-12:
Fix:
 - Positioning error if space 0 blocks/filemarks - Reported & fix thanks to Doug Charnley
 - Only support SECURITY PROTOCOL IN/OUT op codes on IBM LTO4 / IBM 03592E06 and STK T10000 drive types
 - Fix REQUEST SENSE. - Many thanks to Andreas Piesk
 - Reworked vtllibrary SCSI OP code processing.
 - Yet another round of WORM media handling issues - Thanks Albert Pauw
 - Fix kernel module compile error with 2.6.34 kernels

mhvtl-2010-09-23.tgz / mhvtl-0.18-11:
Fix:

 - Potential core dump where SENSE data pointer incorrectly set to NULL
 - Positioning to filemark, followed by write failed to remove cached 'filemark pointer'.
   This caused errors with TSM & HP Dataprotector when writing multiple sequential backups on the same piece of media.
    Many thanks to Nai for testing on TSM
    Many thanks to Philip for testing with HP Dataprotector
 - VPD page 0x83 length incorrect (Thanks Gaetan)
 - VPD NAA data field incorrect (Thanks Gaetan)
Improvements:
 - Remove references to /proc
 - Daemons now create their own device nodes, so no external scripts need to be run. (Note: This required assistance from the kernel module - hence the kernel module from this release needs to be compiled and used - Sorry)
 - Gentoo ebuild file (Thanks Gaetan)
 - RedHat/CentOS kernel module SPEC file (Thanks Dag)
 - Use 'setuidgid' to start daemons if available instead of specified user account.

mhvtl-2010-09-01.tgz / mhvtl-0.18-10:
- Fix TapeAlert
- Added more descriptive logging around SPACE OP code attempting to track down problem with HP Dataprotector
- Added (undocumented) utility 'tapeexerciser' which is a simple utility to test tape read/writes and positioning. WARNING: This will overwrite your media..
- Fix 'is vtl running' logic in rc script. Failed test if 'vtl user' is root (not a recommended config)

mhvtl-2010-07-09.tgz / mhvtl-0.18-9:
- Fix WORM handling - Reported by Albert Pauw
- Fix (finished) LTO5 drive & media handling introduced in 0.18-8

mhvtl-2010-06-23.tgz / mhvtl-0.18-8:
- Several Security Protocol IN updates - thanks Albert Pauw
  * Return certificate data
  * Correct length for 'KEY FORMATS'
  * Correct length for SPIN SUPPORTED PAGES
- Fix kernel compile on RedHat AS4
- Media/drive matching now 'dynamic' and defined in device.conf
- Added man page for device.conf
- Fix media corruption when media is 'formatted'
- Add LTO5 & SDLT-S4 drive/media types
- Handle INQUIRY correctly after media change (return SAM_STAT_GOOD)
- Updated rc script so all devices created on Target & LUN. i.e. Don't use
  channel. Some application software has trouble if only the channel is unique.

mhvtl-2010-05-08.tgz / mhvtl-0.18-7:
Fix: 32bit platforms - Allow media size to be greater than 2G
Fix: core dump on invalid defined NAA strings
Fix: Support SMC devices with embedded space chars in VENDOR ID
Implement STK Vendor unique LOAD DISPLAY SCSI op code. (Logs display info via syslog)

mhvtl-2010-05-02.tgz / mhvtl-0.18-6:
Fix: vtlcmd command line parsing. Fix 'list map' option
Fix: import of media via MAP (off-by-one)
Fix: TapeCapacity LOG SENSE data.
      Now returns bytes/KB/MB as per OEM (Thanks 'kamy')
Support VENDOR ID with embedded spaces. i.e. "HP Ultrium"
Honour MAP status: Fail attempts for robot to move media in CAP if CAP is open.
Relax dependencies on /proc in favour of /sys (note: Work in progress)

mhvtl-2010-04-01.tgz / mhvtl-0.18-5:
Fix: vtlcmd command line parsing. Thanks Herbert Stadler
Fix: Silence local_irq_save(). Normally seen on fedora. Thanks Norm Lunda
Fix: URLs to this homepage.

mhvtl-2010-03-05.tgz / mhvtl-0.18-4:
Fix: ELEMENT STATUS - found and fixed by Norm Lunde (many thanks)
 - This was preventing TSM to work with mhvtl
Add queue depth() callback in kernel module (module date string 20100303)
Test for media mounted before returning status on 'REWIND' op code.
Fix typos in rc script

mhvtl-2010-01-28.tgz / mhvtl-0.18-2...rpm:
The ability to support multiple libraries on the one host !! Thanks once again to Kevan.
Note: There is no conversion of existing configuration file(s) to the new device.conf/library_contents format. Please remove all files under /etc/mhvtl/ before installing mhvtl-2010-01-28 (mhvtl-0.18-2)



What hardware does the mhvtl code base work with:

Tested and developed on Pentium x86 (32bit) and Pentium / AMD x86_64

New: IBM Z-Series. To quote:
"We just fired up the VTL on our z series IBM mainframe. Works like a charm."

What software does it work with:

Symantec NetBackup.

  • 5.1 (Recommend 5.1MP4 minimum)
  • 6.0
  • 6.5.x
  • 7.0.x
  • 7.1.0.x
  • 7.5.0.x
  • Supports SCSI Persistent Reservation
  • Encryption via KMS (NetBackup 6.5.2 and later)

Note: All mhvtl code development is tested using NetBackup

Last set of testing (0.18-8 & -9) with NetBackup 7.0, I had to revert the 'STK/L700' to 'SPECTRA/PYTHON' change made back in Nov 2009. It's just a manual edit of /etc/mhvtl/device.conf

The problem is that NetBackup 7.0 fails to extract the device serial number if SPECTRA/PYTHON is used and hence the automagic device discovery & configuration fails to place the drives within the robot.


Symantec BackupExec.

For the extraordinary writeup : http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37
Includes how to setup the scst (http://mhvtl.nimsa.us/viewtopic.php?f=23&t=32) to present the vtl via iSCSI

I've receive updates from a anonymous contributor identified as 'nia':

Successful Testing of Solaris running mhvtl via iSCSI:  http://mhvtl.nimsa.us/viewtopic.php?f=23&t=47
Successful Testing  of Sun NDMP working with mhvtl via iSCSI  http://mhvtl.nimsa.us/viewtopic.php?f=20&t=53
Successful Testing of Linux ZFS-fuse file-system with "dedup on" for /opt/mhvtl  http://mhvtl.nimsa.us/viewtopic.php?f=13&t=31
Successful Testing of HP Data Protector Linux media agent  http://mhvtl.nimsa.us/viewtopic.php?f=29&t=51
Successful Testing for Backup Exec 2010 on Windows Server 2008 R2. Tested version of mhvtl is 0.18.2 via iSCSI  http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37#p159
Successful Testing for NetBackup 7 Master on Windows 2008 R2 via iSCS  http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37#p159



EMC/Legato NetWorker

Is no longer causing headaches !

Don't use 'STK/L700' Use 'SPECTRA/PYTHON' instead.

Install at least mhvtl-2009-12-16.tgz (mhvtl-0.16-11).
Update: The latest mhvtl-2010-09-23.tgz (mhvtl-0.18-11) is a smoother ride when configuring the library using jbconfig.

Build kernel module from source (Note: /usr/src/packages/ is a SuSE RPM home dir. If you building on RedHat, then this will be /usr/src/redhat/)

# rpm -Uvh mhvtl-0.18-11.src.rpm
# cd /usr/src/packages/BUILD
# tar xvfz ../SOURCES/mhvtl-2009-11-23.tgz
# cd mhvtl-0.18/kernel
# make
# make install

Start the daemons

/etc/init.d/mhvtl start
Configure NetWorker as normal via jbconfig.

An ex colleague who has some very useful/helpful tips with NetWorker (If your interested in NetWorker, his blogs & tips are well worth following)

A detailed setup of CentOS + NetWorker, see:
http://nsrd.info/blog/2010/10/14/new-micromanual-linuxvtl-and-networker/


Bakbone Netvault

(2010-01-09) Thanks to Nick Couchman who has reported NetVault 8.x and mhvtl-2009-12-16 (mhvtl-0.16-11) do indeed work together.

To quote Nick "I have a NetVault 8.x installation here that I'm using, and I just installed LinuxVTL2 and am successfully backing up data from NetVault to LinuxVTL2. I'm not sure if any issues exist for restores and the link, but I'll let you know if I run into anything.  All appears to be working fine at this point!  FWIW I'm running my NetVault server on RHEL5."

TSM

Thanks to Bernardo Clavijo, TSM now works with mhvtl - Need at least 0.18-4 release.


Bacula

A report from Beat Rubischon to advise Bacula works also.

Two changes were needed on Bacula's


bacula-sd.conf:

Device {
Name = Drive-1
Drive Index = 0
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes;
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
AutoChanger = yes
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"

# Needed for mhvtl tape drives
Hardware End of Medium = No
Fast Forward Space File = No
}

Multiple tape drives are supported too.


New: 2010-05-21
Report by Albert Pauw.. Many thanks.

# Needed for mhvtl tape drives
Hardware End of Medium = No
Fast Forward Space File = No

I have used it this way for the last two months, but due to some questions on the mhvtl forum I tried
bacula without these additions. Seems the latest versions of mhvtl work out of the box with bacula,
so these additions are not needed anymore.


Getting Started

A list of recommended base packages for trouble-shooting SCSI devices on Linux
- sg3_utils
- mt
- mtx
- lsscsi

RedHat/CentOS/Scientific Linux/Oracle Linux distributions:

If all you want/need to do is get the VTL up and running as soon as possible, with a minimum of fuss:
Many thanks to the ELRepo project : http://elrepo.org/tiki/tiki-index.php

Import the ELRepo public key:

Decide if your distribution is based around RH 5 or RH 6

If RH 5:
Install the ELRepo RPM for RHEL-5 / SL-5 or CentOS-5

If RH 6:
Install the ELRepo RPM for RHEL-6 / SL-6 or CentOS-6

Install mhVTL packages:

yum install kmod-mhvtl mhvtl-utils

Start mhVTL:
services mhvtl start

All that's left is to configure your backup application..

Non RedHat based RPM Linux distributions:

This requires the gcc development environment to be installed as there is a pseudo SCSI hba driver to compile and install.

To install and get the package running:

1. Install both src.rpm & x86/x86_64 (depending on your CPU/OS version - check using uname -p)

2. Build kernel module from source you just installed:
Note: The directory in the example below is a SuSE RPM home dir (which is /usr/src/packages)
         If you are installing on RedHat:
            /usr/src/redhat or $HOME/rpmbuild/ (depending on release of RedHat)
        Other RPM distributions may have different default RPM home directories.

  cd /usr/src/packages/BUILD
  tar xvfz ../SOURCES/mhvtl-2013-03-22.tgz
  cd mhvtl-1.4/kernel
  make
  make install

3. Start the package using the rc script.
  /etc/init.d/mhvtl start

4. Check it's all running:
  lsscsi -g
  ps -ef|grep vtl (default with 10 daemons - 2 vtllibrary & 8 vtltape)

5. Configure your backup software ;)

NON-RPM based Linux distributions: (or you just want to do it the manual way)

To install and get the package running:

1. Download the 'mhvtl-YYYY-MM-DD.tgz

2. Extract source code
 cd /some/where/safe
 tar xvfz /where/you/downloaded/mhvtl-YYYY-MM-dd.tgz

3. Build user-space daemons
 cd mhvtl-1.4
 make

Note: Since mhvtl-1.3, both lzo and zlib are required to build vtltape
Typical error (missing lzo devel package) looks like:

cc -Wall -Wshadow -g -O2 -D_LARGEFILE64_SOURCE  -I../kernel -DMHVTL_VERSION=\"1.4.0\" -DMHVTL_DEBUG -DUSR=\"vtl\" -DMHVTL_HOME_PATH=\"/opt/mhvtl\" -DMHVTL_CONFIG_PATH=\"/etc/mhvtl\"   -c -o vtltape.o vtltape.c
vtltape.c:92:25: error: lzo/lzoconf.h: No such file or directory
vtltape.c:93:23: error: lzo/lzo1x.h: No such file or directory
vtltape.c: In function 'uncompress_lzo_block':

To fix, install lzo development package and rerun the 'make' command

4. Add a user 'vtl'

 sudo useradd -c "mhvtl user" vtl

5. Install binaries
 sudo make install

6. Build kernel module from source you just installed:
  cd kernel
  make
  make install

7. Start the package using the rc script.
  /etc/init.d/mhvtl start

Note: If you can't find the 'mhvtl' rc script in /etc/init.d, you most likely swapped the order of 4 & 5.
    i.e. You ran 'sudo make install', found it failed due to missing 'vtl' user account and didn't re-run the 'make install' after creating the account.

8. Check it's all running:
  lsscsi -g
  ps -ef|grep vtl (default with 10 daemons - 2 vtllibrary & 8 vtltape)

9. Configure your backup software ;)

Compiling the kernel module

  tar xvfz mhvtl-2013-10-20.tgz (latest at time of edit)
  cd mhvtl-1.4/kernel
  make
  make install

  rmmod mhvtl (remove original kernel module - if loaded)

  modprobe mhvtl opts=0 (load new kernel module)

  cat /sys/modules/mhvtl/version (confirm version is 0.18.15)


Some nice things people are saying about the mhvtl

  • It's really caught on here, I've noticed that some of our developers have stopped using real tape hardware altogether, it's just easier to use the VTL. Plus our customer support has started doing customer training with the VTL instead of real tape hardware. Pretty impressive! A fine piece of software, wish I had found it a few years earlier.
  • The VTL has been very helpful in my testing and for customer demos. Your hard work on this is very much appreciated by myself and others on my team.
  • I am one more user of your magical software vtl here at Symantec. Just want to say that it rocks
  • I finally got round to trying to get this set up and I have to say.. You are a genius ! My laptop now has NBU 6.5.1 on Redhat 4 AS (on VMware) with a VTL robot and 8 drives !! Fantastic ! Thanks for all your work on this ! This will be very useful for me.
  • I have been a Storage and Backup Consultant for many years specialising mainly in TSM but with some NetWorker and NetBackup thrown in for good measure. For years I have been grabbing old used tape libraries for my lab so I can do all sorts of testing but this has always been an expensive proposition. With mhVTL I can now provision so many virtualised backup environments in a very short timescales for test and it is a joy to have such a product available in my arsenal.


I can be reached via mark_harvey@symantec.com or mhvtl2@gmail.com

QR email