Linux Virtual Tape Library - mhVTL
Q. Why GPL license.
A. Because I like the idea of sharing.
I don't profess to know everything (not even close). Having something I can share and allow others to share back is a good idea.
Virtual Tape Library consists of several components.
LLD - A low level driver implemented as a kernel module - mhvtl
Target devices - Daemons vtltape(1) and vtllibrary(1) which implement SCSI target device(s) in user-space
Utility commands mktape(1), vtlcmd(1)
And startup scripts build_library_config(1), make_vtl_devices(1)
The kernel module is based on the scsi_debug kernel module (http://sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note: As of 0.16, there are no default devices.
The support scripts will add Drives (SSC devices) & library (SMC) depending on the contents of /etc/mhvtl/library_contents and /etc/mhvtl/device.conf.
A char device back-end has been included with the vtl LLD driver This allows data and SCSI commands to be passed from the LLD to user-mode daemons (SCSI targets) which constently poll the driver and process any outstanding SCSI commands.
vtltape(1) is the usermode SSC target daemon which writes/reads data to data files in the /opt/mhvtl directory (if a virtual tape has been loaded). The virtual tape files include a Medium Auxiliary Memory (MAM) data structure to store persistent data (number of tape loads, total data written/read, media type etc).
vtllibrary(1) is the usermode SMC target daemon which reads its configuration from the file /etc/mhvtl/library_contents(5) at startup. The number of storage slots are built dynamically when the daemon starts. Hence changing the number of storage slots and media access slots are a matter of modifying the file contents and restarting the vtllibrary(1) daemon. All 'library' commands are performed on data structures in memory ONLY.
A utility vtlcmd(1) is used to administrator the daemons vtltape(1) and vtllibrary.
Message queue (key 0x4d61726b) is used to pass messages between vtlcmd(1), vtllibrary(1) and vtltape(1)
When a SCSI 'move medium' from a storage slot to a tape drive is requested, the media location is updated in vtllibrary(1) data structures, and the barcode of the media id is passed via the message queue to the vtltape(1) daemon in question. A file open of the barcode is attempted in the /opt/mhvtl directory and if successful, the vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit Ready requests. An unload SCSI command to the tape drive will close the data file.
Media can be moved out of the VTL via the Media Access Port. Once media is logically moved into the MAP slots, the MAP entries can be cleared using the vxcmd:
# vtlcmd library empty map
The media can be 'moved back again' by re-starting the VTL user-space daemons:
Media can be loaded into the MAP via a :
or restart the vtl
TapeAlert flags can be set using the vtlcmd(1)
e.g. vtlcmd [index] TapeAlert [flags]
Where index is the message Q offset associated with the drive (or the string 'library').
e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:
Pull out the binary to decimal calculator
1000 0000 0000 0000 0000 (20 bits) => hex => 80000
A listing of TapeAlert flags can be found at t10 home page for SSC devices Annex A.
mhvtl is registered as HBA #3:
Long term goals
Eventually, the functionality will be moved across to the SCSI Target Framework (stgt) http://stgt.berlios.de/
The stgt is a very nicely designed SCSI taret framwork which can support all target types along with multiple SCSI transports in a very structured way.
However, until time permits (and my c coding standards improve), features and improvements on this code base will continue.
Howto get help
Somebody has put up a discussion board (and has lots of great documentation on working with different backup software)
Howto help out
There are many way of providing assistance
Testing with backup software X and reporting success/failure. If possible, test any patches that may result of testing.
If you are willing, patches are very welcome.
Please read the Linux Documentation/CodingStyle
Note: I only discovered this documentation guide recently, so patches to update old code style to the CodingStyle are also welcome.
Please run any patches thru the Linux 'mhvtl-<version>/scripts/checkpatch.pl' for coding style sanity check before sending.
Pull the source from git repository (github).. Tinker till your hearts content :)
Currently, only the tape module has 'personality modules'
For up-to-date view of emulations review vtltape.c and search for "static struct tape_drives_table"
Working emulations include:
What about model X....
Quantum (DLT/Super-DLT) is NOT supported as their SCSI Programmer's Guide does not document the expected returned data.
DDS (4mm DAT) - I've not found any SCSI programmers' Guides for these drives.
Library emulation is coming in the future.
Note: Please let me know what works for which version of backup software so I can better document it.
Download source tar ball : mhvtl-2023-03-10.tgz
Download source RPM: mhvtl-utils-1.7-1.src.rpm
Download x86_64 binary RPM: mhvtl-utils-1.7-1.el8.x86_64.rpm
Confused as to what to get ?
If you are a RedHat based distribution (RedHat / CentOS / Oracle / Scientific Linux, RockyLinux, AlmaLinux etc) than ELrepo may be your best bet. Checkout : http://elrepo.org/tiki/tiki-index.php
- Follow the 'Getting Started' section
- Import the public key:
- Install ELRepo for RHEL-5, SL-5 or CentOS-5 / install ELRepo for RHEL-6, SL-6 or CentOS-6
- $ sudo yum install mhvtl-utils kmod-mhvtl
Check what sort of linux distribution.
Choose 'src.rpm & <platform>.rpm if RPM based and want to use mhVTL in fastest way.
Choose 'source code tar ball' if using a non-RPM based linux distribution. (e.g. Debian/Ubuntu etc)
Choose 'Download using git' if your interested in fixing bugs, following development history, as well as using/testing mhVTL
There are 2 "bits" that make up the 'mhVTL'.
- User space daemons
- mhvtl pseudo scsi hba kernel driver.
The kernel module source code is included with any of the 'source' packages (& git) below and needs to be downloaded and built to suit the currently running linux kernel on your linux host.
Download one or more to get going and playing as soon as possible:
Download using 'git' - This provides a complete history of all changes.
Create a directory where you are going to store the source code.
Clone the source code (it's only a few hundred k)
From time to time, check for any updates and synchronize from github.
Now that the 1.0 milestone has been reached, any changes to internal functionality will increment the 'x' (dot release) e.g. 1.x
Bug fixes will result in an incremental (double-dot) release. e.g. 1.1-x
Kevan Rehm has re-written the on-disk data structure.
(It's amazing how much effort can be rolled up in one simple sentence).
Each 'tape' is now stored under its own subdirectory /opt/mhvtl/<barcode>
A tape consists of three files.
- Data file => Container of data
- Index file => Contains an array of header structures (pointers into the data file)
- Meta file => Contains the MAM structure, followed by a meta data header, followed by a variable-length array of filemark block numbers.
What hardware does the mhvtl code base work with:
Tested and developed on Pentium x86 (32bit) and Pentium / AMD x86_64
New: IBM Z-Series. To quote:
"We just fired up the VTL on our z series IBM mainframe. Works like a charm."
What software does it work with:
5.1 (Recommend 5.1MP4 minimum)
Supports SCSI Persistent Reservation
Encryption via KMS (NetBackup 6.5.2 and later)
Note: All mhvtl code development is tested using NetBackup
Last set of testing (0.18-8 & -9) with NetBackup 7.0, I had to revert the 'STK/L700' to 'SPECTRA/PYTHON' change made back in Nov 2009. It's just a manual edit of /etc/mhvtl/device.conf
The problem is that NetBackup 7.0 fails to extract the device serial number if SPECTRA/PYTHON is used and hence the automagic device discovery & configuration fails to place the drives within the robot.
Note: Recent NetBackup releases (post 8.2) require custom udev rules to be implemented. NetBackup now requires /dev/tape/by-id/ and /dev/tape/by-path/ devices to exist. Default linux distros do not know about mhvtl kenel and hence do not create the devices.
Executing the 'build-persistent-tape-rules.sh' script in the 'scripts' directory will generate a set of rules compatible for NetBackup
Add the output information to a udev rule of your choosing
$ sudo ./scripts/build-persistent-tape-rules.sh > /etc/udev/rules.d/70-persistent-by-path-tape.rules
For the extraordinary writeup : http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37
Includes how to setup the scst (http://mhvtl.nimsa.us/viewtopic.php?f=23&t=32) to present the vtl via iSCSI
I've receive updates from a anonymous contributor identified as 'nia':
Is no longer causing headaches !
Don't use 'STK/L700' Use 'SPECTRA/PYTHON' instead.
Install at least mhvtl-2009-12-16.tgz (mhvtl-0.16-11).
Update: The latest mhvtl-2010-09-23.tgz (mhvtl-0.18-11) is a smoother ride when configuring the library using jbconfig.
Build kernel module from source (Note: /usr/src/packages/ is a SuSE RPM home dir. If you building on RedHat, then this will be /usr/src/redhat/)
Configure NetWorker as normal via jbconfig.
An ex colleague who has some very useful/helpful tips with NetWorker (If your interested in NetWorker, his blogs & tips are well worth following)
Also a follow up article
A detailed setup of CentOS + NetWorker, see:
2015-02 - Thanks to Anup Sreedharan who has written up a 100+ page document which provides step-by-step instructions on setting up mhvtl + mhvtl-GUI + CommVault.
The guide is available via this link Comvault Implementation
Questions / kludos all go to Anup Sreedharan
Linked in: http://in.linkedin.com/pub/anup-sreedharan/19/448/908
(2010-01-09) Thanks to Nick Couchman who has reported NetVault 8.x and mhvtl-2009-12-16 (mhvtl-0.16-11) do indeed work together.
To quote Nick "I have a NetVault 8.x installation here that I'm using, and I just installed LinuxVTL2 and am successfully backing up data from NetVault to LinuxVTL2. I'm not sure if any issues exist for restores and the link, but I'll let you know if I run into anything. All appears to be working fine at this point! FWIW I'm running my NetVault server on RHEL5."
Thanks to Bernardo Clavijo, TSM now works with mhvtl - Need at least 0.18-4 release.
A report from Beat Rubischon to advise Bacula works also.
Two changes were needed on Bacula's
I have used it this way for the last two months, but due to some questions on the mhvtl forum I tried
bacula without these additions. Seems the latest versions of mhvtl work out of the box with bacula,
so these additions are not needed anymore.
A list of recommended base packages for trouble-shooting SCSI devices on Linux
RedHat/CentOS/Scientific Linux/Oracle Linux distributions:
If all you want/need to do is get the VTL up and running as soon as possible, with a minimum of fuss:
Many thanks to the ELRepo project : http://elrepo.org/tiki/tiki-index.php
Import the ELRepo public key:
rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org
Decide if your distribution is based around RH 5 or RH 6
If RH 5:
Install the ELRepo RPM for RHEL-5 / SL-5 or CentOS-5
If RH 6:
Install the ELRepo RPM for RHEL-6 / SL-6 or CentOS-6
Install mhVTL packages:
yum install kmod-mhvtl mhvtl-utils
If using systemctl : systemctl start mhvtl.target
All that's left is to configure your backup application..
Non RedHat based RPM Linux distributions:
This requires the gcc development environment to be installed as there is a pseudo SCSI hba driver to compile and install.
To install and get the package running:
1. Install both src.rpm & x86/x86_64 (depending on your CPU/OS version - check using uname -p)
2. Build kernel module from source you just installed:
Note: The directory in the example below is a SuSE RPM home dir (which is /usr/src/packages)
If you are installing on RedHat:
/usr/src/redhat or $HOME/rpmbuild/ (depending on release of RedHat)
Other RPM distributions may have different default RPM home directories.
3. Start the package using the rc script.
3. Start the package using the systemctl.
4. Check it's all running:
5. Configure your backup software ;)
NON-RPM based Linux distributions: (or you just want to do it the manual way)
To install and get the package running:
1. Download the 'mhvtl-YYYY-MM-DD.tgz
2. Extract source code
3. Build user-space daemons
Note: Since mhvtl-1.3, both lzo and zlib are required to build vtltape
Typical error (missing lzo devel package) looks like:
To fix, install lzo development package and rerun the 'make' command
4. Add a user 'vtl' (not required as of 1.6-2 or later)
5. Install binaries
sudo make install
6.Pre 1.7-1 (2023-03-10) Build kernel module from source you just installed:
6. From 1.7-1 (2023-03-10) Use a more user-friendly script
7. Pre 'systemd' Start the package using the rc script.
Post 'systemd' Start the package using systemd
Note: If you can't find the 'mhvtl' rc script in /etc/init.d, you most likely swapped the order of 4 & 5.
i.e. You ran 'sudo make install', found it failed due to missing 'vtl' user account and didn't re-run the 'make install' after creating the account.
8. Check it's all running:
9. Configure your backup software ;)
Compiling the kernel module
rmmod mhvtl (remove original kernel module - if loaded)
Load new module:
Confirm module version
Some nice things people are saying about the mhvtl
It's really caught on here, I've noticed that some of our developers have stopped using real tape hardware altogether, it's just easier to use the VTL. Plus our customer support has started doing customer training with the VTL instead of real tape hardware. Pretty impressive! A fine piece of software, wish I had found it a few years earlier.
The VTL has been very helpful in my testing and for customer demos. Your hard work on this is very much appreciated by myself and others on my team.
I am one more user of your magical software vtl here at Symantec. Just want to say that it rocks
I finally got round to trying to get this set up and I have to say.. You are a genius ! My laptop now has NBU 6.5.1 on Redhat 4 AS (on VMware) with a VTL robot and 8 drives !! Fantastic ! Thanks for all your work on this ! This will be very useful for me.
I have been a Storage and Backup Consultant for many years specialising mainly in TSM but with some NetWorker and NetBackup thrown in for good measure. For years I have been grabbing old used tape libraries for my lab so I can do all sorts of testing but this has always been an expensive proposition. With mhVTL I can now provision so many virtualised backup environments in a very short timescales for test and it is a joy to have such a product available in my arsenal.
I can be reached via firstname.lastname@example.org or email@example.com