Linux Virtual Tape Library - mhVTL

Index

• Description

• vtltape

• vtllibrary

• vtlcmd

• What sort of emulation

• Download - (current 1.7)

• Getting Started

• vtl Kludos

License:

GPL v2

Q. Why GPL license.

A. Because I like the idea of sharing.

I don't profess to know everything (not even close). Having something I can share and allow others to share back is a good idea.

Virtual Tape Library consists of several components.

• LLD - A low level driver implemented as a kernel module - mhvtl

• Target devices - Daemons vtltape(1) and vtllibrary(1) which implement SCSI target device(s) in user-space

• Utility commands mktape(1), vtlcmd(1)

• And startup scripts build_library_config(1), make_vtl_devices(1)

The kernel module is based on the scsi_debug kernel module (http://sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note: As of 0.16, there are no default devices.

The support scripts will add Drives (SSC devices) & library (SMC) depending on the contents of /etc/mhvtl/library_contents and /etc/mhvtl/device.conf.

A char device back-end has been included with the vtl LLD driver This allows data and SCSI commands to be passed from the LLD to user-mode daemons (SCSI targets) which constently poll the driver and process any outstanding SCSI commands.

vtltape

vtltape(1) is the usermode SSC target daemon which writes/reads data to data files in the /opt/mhvtl directory (if a virtual tape has been loaded). The virtual tape files include a Medium Auxiliary Memory (MAM) data structure to store persistent data (number of tape loads, total data written/read, media type etc).

vtllibrary

vtllibrary(1) is the usermode SMC target daemon which reads its configuration from the file /etc/mhvtl/library_contents(5) at startup. The number of storage slots are built dynamically when the daemon starts. Hence changing the number of storage slots and media access slots are a matter of modifying the file contents and restarting the vtllibrary(1) daemon. All 'library' commands are performed on data structures in memory ONLY.

vtlcmd

A utility vtlcmd(1) is used to administrator the daemons vtltape(1) and vtllibrary.

Message queue (key 0x4d61726b) is used to pass messages between vtlcmd(1), vtllibrary(1) and vtltape(1)

When a SCSI 'move medium' from a storage slot to a tape drive is requested, the media location is updated in vtllibrary(1) data structures, and the barcode of the media id is passed via the message queue to the vtltape(1) daemon in question. A file open of the barcode is attempted in the /opt/mhvtl directory and if successful, the vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit Ready requests. An unload SCSI command to the tape drive will close the data file.

Media can be moved out of the VTL via the Media Access Port. Once media is logically moved into the MAP slots, the MAP entries can be cleared using the vxcmd:

# vtlcmd library empty map

The media can be 'moved back again' by re-starting the VTL user-space daemons:

Media can be loaded into the MAP via a :

# vtlcmd library load map <barcode>

or restart the vtl

# /etc/init.d/mhvtl restart

TapeAlert

TapeAlert flags can be set using the vtlcmd(1)

e.g. vtlcmd [index] TapeAlert [flags]

Where index is the message Q offset associated with the drive (or the string 'library').

e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:

Pull out the binary to decimal calculator

1000 0000 0000 0000 0000 (20 bits) => hex => 80000

# vtlcmd 1 TapeAlert 80000

A listing of TapeAlert flags can be found at t10 home page for SSC devices Annex A.

Examples

mhvtl is registered as HBA #3:

# lsscsi -g

<snip>

[5:0:0:0]    mediumx STK      L700             0106  /dev/sch0  /dev/sg8 

[5:0:1:0]    tape    IBM      ULT3580-TD5      0106  /dev/st4   /dev/sg10

[5:0:2:0]    tape    IBM      ULT3580-TD5      0106  /dev/st1   /dev/sg6 

[5:0:3:0]    tape    IBM      ULT3580-TD4      0106  /dev/st0   /dev/sg5 

[5:0:4:0]    tape    IBM      ULT3580-TD4      0106  /dev/st6   /dev/sg12

[5:0:8:0]    mediumx STK      L80              0106  /dev/sch1  /dev/sg14

[5:0:9:0]    tape    STK      T10000B          0106  /dev/st2   /dev/sg7 

[5:0:10:0]   tape    STK      T10000B          0106  /dev/st5   /dev/sg11

[5:0:11:0]   tape    STK      T10000B          0106  /dev/st7   /dev/sg13

[5:0:12:0]   tape    STK      T10000B          0106  /dev/st3   /dev/sg9 

Long term goals

Eventually, the functionality will be moved across to the SCSI Target Framework (stgt) http://stgt.berlios.de/

The stgt is a very nicely designed SCSI taret framwork which can support all target types along with multiple SCSI transports in a very structured way.

However, until time permits (and my c coding standards improve), features and improvements on this code base will continue.

Howto get help

Somebody has put up a discussion board (and has lots of great documentation on working with different backup software)

https://github.com/markh794/mhvtl.git - github

Howto help out

There are many way of providing assistance

• Testing with backup software X and reporting success/failure. If possible, test any patches that may result of testing.

• If you are willing, patches are very welcome.

  • Please read the Linux Documentation/CodingStyle

  • Note: I only discovered this documentation guide recently, so patches to update old code style to the CodingStyle are also welcome.

  • Please run any patches thru the Linux 'mhvtl-<version>/scripts/checkpatch.pl' for coding style sanity check before sending.

• Pull the source from git repository (github).. Tinker till your hearts content :)

Emulations

Currently, only the tape module has 'personality modules'

For up-to-date view of emulations review vtltape.c and search for "static struct tape_drives_table"

list of tape emulations (as of v1.4-9)

Working emulations include:

• IBM LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)

• HP LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)

• IBM 03592-J1A, 03592-E05 & 03592-E06

• STK 9840 (A/B/C/D), 9940 (A/B)

• STK T10000 (A/B/C)

• Sony AIT (1/2/3 &4)

What about model X....

• Quantum (DLT/Super-DLT) is NOT supported as their SCSI Programmer's Guide does not document the expected returned data.

• DDS (4mm DAT) - I've not found any SCSI programmers' Guides for these drives.

Library emulation is coming in the future.

STK L180/L700 emulation works for NetBackup & NetWorker.

TSM - IBM library type needed to use LTO drives.

Note: Please let me know what works for which version of backup software so I can better document it.

Downloads

Download source tar ball : mhvtl-2023-03-10.tgz 

Download source RPM: mhvtl-utils-1.7-1.src.rpm 

Download x86_64 binary RPM: mhvtl-utils-1.7-1.el8.x86_64.rpm 

Confused as to what to get ?

If you are a RedHat based distribution (RedHat / CentOS / Oracle / Scientific Linux, RockyLinux, AlmaLinux etc) than ELrepo may be your best bet. Checkout : http://elrepo.org/tiki/tiki-index.php

- Follow the 'Getting Started' section

- Import the public key:

- Install ELRepo for RHEL-5, SL-5 or CentOS-5 / install ELRepo for RHEL-6, SL-6 or CentOS-6

- $ sudo yum install mhvtl-utils kmod-mhvtl

Otherwise:

 Check what sort of linux distribution.

 Choose 'src.rpm & <platform>.rpm if RPM based and want to use mhVTL in fastest way.

 Choose 'source code tar ball' if using a non-RPM based linux distribution. (e.g. Debian/Ubuntu etc)

 Choose 'Download using git' if your interested in fixing bugs, following development history, as well as using/testing mhVTL

 There are 2 "bits" that make up the 'mhVTL'.

  - User space daemons

  - mhvtl pseudo scsi hba kernel driver.

The kernel module source code is included with any of the 'source' packages (& git) below and needs to be downloaded and built to suit the currently running linux kernel on your linux host.

Download one or more to get going and playing as soon as possible:

Download using 'git' - This provides a complete history of all changes.

Create a directory where you are going to store the source code.

$ mkdir mhvtl

Clone the source code (it's only a few hundred k) 

$ git clone https://github.com/markh794/mhvtl.git

From time to time, check for any updates and synchronize from github.

$ git pull https://github.com/markh794/mhvtl.git

  ===========

Change log

Now that the 1.0 milestone has been reached, any changes to internal functionality will increment the 'x' (dot release) e.g. 1.x

Bug fixes will result in an incremental (double-dot) release. e.g. 1.1-x

Kevan Rehm has re-written the on-disk data structure.

(It's amazing how much effort can be rolled up in one simple sentence).

Each 'tape' is now stored under its own subdirectory /opt/mhvtl/<barcode>

A tape consists of three files.

- Data file => Container of data

- Index file => Contains an array of header structures (pointers into the data file)

- Meta file => Contains the MAM structure, followed by a meta data header, followed by a variable-length array of filemark block numbers.

* Fri Mar 10 16:30:00 AEST 2023 Mark Harvey <markh794@gmail.com>

- review of all utilities and use '-m <barcode>' the standard option

- Add support for 'INQUIRY_DATA_HAS_CHANGED'

- Added 'Extended Inquiry Data VPD page for Ultrium

- Add SELinux ACL so systemctl scripts will work if SELinux is enforcing

- mhvtl.ko: Various improvements to suit recent kernel changes (thanks Lee Duncan and Tamas Revesz)

- mhvtl.ko: Change initiator ID from 15 to -1 (resolves conflict if target also configured with ID 15)

- mhvtl.ko: Remove printk in favour of pr_* calls

- pre-package mhvtl.ko source into /usr/lib/firmware/mhvtl/ and add wrapper script 'mhvtl_kernel_mod_build' to extract, compile and install

- Various 64bit/32bit casts fixed.

- Updated Vagrant to use AlmaLinux, RockyLinux, CentOS, OpenSuse, Ubuntu

* Thu Mar 10 10:30:00 AEST 2022 Mark Harvey <markh794@gmail.com>

- Updated default drives in library 10 from LTO4/LTO5 to LTO6 & LTO8

- Added initial support for Data Integrity Validation (via Logical Block Protection) - IBM LTO6,LTO7 & LTO8 with this release.

- mhvtl.ko: Various updates to match the latest kernels

- Additional checks for Vagrant build on CentOS, Ubuntu and OpenSUSE

- Improve tape load/unload handshake to make it more robust

- Improve handling of missing slot numbers in library_contents.xx

- Makefile - Move variables to a common location (config.mk)

- SEND DIAGNOSTICS: Implement basic sanity check for SSC devices

- VERIFY(6) op code support

- Update install instructions with sg3_utils vs sg3-utils package names

- LTO-6 media load to match LTO standards (LTO8 cannot read LTO6 media)

- New utility 'preload_tape' bypassing need to write via /dev/st*

- Add missing 'ENCRYPTION capable' bit for LTO6, 7 & 8 media

- systemd device config: Best effort, log errors via klog

- Obligatory spelling fixes

- Bump version to 1.7 : Beginning of LBP, VERIFY(6) and SEND DIAGNOSTICS support

* Thu Oct 07 10:40:00 AEST 2021 Mark Harvey <markh794@gmail.com>

- rename list.h to avoid generic name conflict

- rename scsi.h to avoid generic name conflict

- rename log.h to avoid generic name conflict

- Exclude __builtin_cpu_supports() on non x86_64 CPU types

- Fix spelling of retrieving

- mhvtl.ko: Define SG_SEGMENT_SZ only if not defined

- Increase default kmem_cache bounce buffer size to 64k

- mhvtl.ko: kernel module oops on PPC

- mhvtl.ko: Log errors if they occur

- mhvtl.ko: Correct compiler warning about printf var sizes

- mhvtl.ko: Simplify file_inode()

- mhvtl.ko: Use _safe version of list_for_each_entry

- mhvtl.ko: Limit number of outstanding queued commands

- mhvtl.ko: Log minor number with other ioctl info

- mhvtl.ko: clean up noisy logging at level 2

- mhvtl.ko: Initialise outstanding op struct before adding to list

- mktape: Set default version string based from MHVTL_VERSION

- Update LTO-8 media denisty codes

- Security Protocol OUT: Fix null pointer check

- Fix routine to extract barcode from string

- Fix typos in mktape man page

- dump_tape: Fix segfault due to local variable conflict

- Use bounce buffer if tape block size is larger than request buffer

- Remove unused scsi_host_template field

- mhvtl.ko: Compile on RedHat 8.x kernel

- mhvtl.ko: reinstate HAVE_UNLOCKED_IOCTL

- mhvtl.ko: Remove reference to DRIVER_SENSE

- Update function names with 'mhvtl_' prefix

- Update kernel /sys/ location to suit new pseudo name space

- OOM: /proc/<pid>/oom_adj is deprecated. Using oom_score_adj.

- systemd: Update Makefile to include systemd install path

- mhvtl.spec: Update to build a package on CentOS8

* Tue Mar 10 09:00:00 AEST 2020 Mark Harvey <markh794@gmail.com>

- Fix kernel module oops/panic due to race between placing OP ready for user space before all variable were initialised

- Add ccan version of crc32c routine which includes CPU optimisations for sse4.2 capabile CPUs

* Sun Oct 06 12:20:00 AEST 2019 Mark Harvey <markh794@gmail.com>

- Fix kernel copy_to_user()/copy_from_user() to use white-listed buffer

- Fix kernel compile for 5.0+ kernels

* Wed Apr 10 10:30:30 AEST 2019 Mark Marvey <markh794@gmail.com>

- Fix long standing bug on media unload

- Added CRC32 checks for each block written/read

- allow kernel module to compile on 5.0 kernels

- Add example on how to configure LIO pscsi (passthru) driver with mhVTL

- Fix parsing args in dump_tape

* Thu Mar 28 14:30:30 AEST 2019 Mark Marvey <markh794@gmail.com>

- Create virtual media on rpm install

- Fixed a couple typos in spec file (one from several years ago)

- Don’t overwrite /etc/mhvtl/* but install as ‘rpmnew’

- Replace SUSE specific rpm macros with hand crafted ’systemctl’ commands.

- systems-generators dir needed munging for CentOS7

- Include ‘ChangeLog’ as part of the tar ball - the time/date build on man pages is dependant

* Thu Nov 29 15:42:24 PST 2018 Lee Duncan <lduncan@suse.com>

- Many changes to support systemd:

  - Removed /etc/init.d/mhvtl script

  - Config files are now generated at install time, not the first

    time the service is run

  - Created service files and templates, and a generator

  - Updated INSTALL file with sytemd info

  - removed build_library_config and mhvtl init script

  - Updated man pages and doc/index.html

  - Updated make files

  - created generate_device_conf and generate_library_contents, called at

    'make install' time, to take the place of what the init file used to do

  - updated SPEC file for systemd

  - updated example scripts

  - fixed asprintf calls to check return values (to make compiler happy)

  - added some arguments and better arg parsing to vtltape and vtllibray

- moved some .gitignore entries around, to be more local

- Bumped version to 1.5-5

- Fixed config file error in library_contents.* where VERSION was missing

mhvtl-2016-03-10.tgz / mhvtl-1.5-4:

    - Fix HP library emulation 'serial number' handling (DVCID in AVOL field)

    - Add LTO 7 emulation (lightly tested)

    - Embed minilzo source and use 'safe' lzo1x decompression routines (prevent intermittent segfault)

    - Prevent internal block number overflow.

    - Add 16byte SPACE op code.

mhvtl-2014-09-07.tgz / mhvtl-1.5-2:

    - Fix to library emulation 'serial number' handling (increased max serial number to 15)

    - Fix Inquiry VPD updates in library emulation

    - Add Scalar emulation

mhvtl-2014-04-13.tgz / mhvtl-1.5-0:

    - Introduce personality modules for tape libraries

    - Bug fix with libraries configured with more than 1260 slots

    - New media type 'NULL' for performance testing

    - Change kernel module bus type from 'pseudo' to 'pseudo9' to prevent address space conflict with scsi_debug

mhvtl-2013-10-20.tgz / mhvtl-1.4-10:

    - Various cleanups/bug fixes as a result of the static code check 'smatch'

    - Fix TapeAlert bit offsets

    - Return bit/byte offset for sense INVALID FIELD IN CDB/PARAMS

    - Fix returned data size when 0 data is requested (kernel module fix)

    - SMC - Ability to keep library media change persistant across restarts

    - SSC - Fix op code 0Fh (read reverse) incorrectly set to 'reserve'

    - SSC - Implement 'read media serial number' op code

    - Fix 'make_vtl_media' script to understand IBM 03592 media type

mhvtl-2013-08-29.tgz / mhvtl-1.4-9:

- MODE SELECT for SSC (tape) overhaul

- Sanitize delay values introduced with previous release

- Code cleanup based on Valgrind - memory leaks and unused file handles

- Set SCSI Revision field for T10000B emulation

- Adjust SCSI Inquiry information for 'Scalar' libraries (add 'BarC' field, pad 'Full Firmware Revision' field with compile time/date information)

mhvtl-2013-06-29.tgz / mhvtl-1.4-8:

- Add additional fields in MAM for multi-partition support capability

- MODE SELECT fixes

- Solaris 11.1 st mode select

- Make virtual media more resilient against disk full conditions

- Add additional error checking between kernel and user-space ioctl

- Add ability to introduce delays in load/unload/position/thread/rewind op codes

- Add 16byte CDB locate command support

- Minor kernel module tweaks - increase timeout & error checking on ioctl()

mhvtl-2013-01-31.tgz / mhvtl-1.4-6:

- Fix crash when position back xx blocks

- Add 'edit_tape' utility. Update virtual media 'metadata'

- Fix TAPE CAPACITY log page - return correct data page and fix buffer overflow

mhvtl-2013-01-12.tgz / mhvtl-1.4-5:

- Fix End-Of-Media (EOM) handling (reported write failure actually successfully wrote the block - resulting in an off-by-one block count)

- Add LTO-6 emulation

- Fix TAPE CAPACITY log page information (Corrected a byte swap error)

mhvtl-2012-09-13.tgz / mhvtl-1.4-4:

- Fix bug (off-by-one) in LOG SELECT

- Implement counter reset via LOG SELECT & PCR bit

- Implement 'performance' counter - Records amount of time (spent in usleep()) since last op code processed.

- Note: Due to build issues, mhvtl-1.4-2 & mhvtl-1.4.3 never made public release.

mhvtl-2012-08-08.tgz / mhvtl-1.4-1:

- Bug fixes

     Fix dump_tape to understand new library subdirectory format

     Fix vtlcmd to understand new library subdirectory format

mhvtl-2012-08-01.tgz / mhvtl-1.4-0:

- New features

          vtltape: Make the backoff algrithm value configurable

          Default media belonging to a library in a subdir

          Add default "Home directory" entry in device.conf

- Personality module updates

          Allow 'default' module to load all media R/W

          Add IBM half-hight INQUIRY string match

          vtltape: Update log entry regarding media capacity depending on capabiliti

          Initial commit of SDLT600 personality module

          Initial commit of SDLT320 personality module

          Update mktape to work 'better' with SDLT media types

- Cleanups

          Move standard inquiry info into memory structure

          mhvtl.ko: Silence install 'depmod' error

          Standardize on PCODE_OFFSET for inquiry page code mask

          Remove unused variable 'removable'

          Improve logging of mode page creation

- Packaging:

          Remove kmod-mhvtl.spec

          Rename 'mhvtl' package as 'mhvtl-utils'

          Use ELRepo spec file

          Move 'make install' responsibility from RPM to Makefile

mhvtl-2012-06-15.tgz / mhvtl-1.3-0:

- Move on-disk functions into its own shared lib (libvtlcart.so)

- Improve IBM & HP LTO emulation of Inquiry VPD pages

- Add ability to call external commands when moving media

- Toggle APPEND ONLY mode from user-space via 'vtlcmd'

mhvtl-2012-04-04.tgz / mhvtl-1.2-3:

- Fix segfault when using AIT4 emulation and Security Protocol IN/OUT was attempted

- Add more descriptive log messages for LOG & MODE pages

mhvtl-2012-03-22.tgz / mhvtl-1.2-2:

- Changed defaults so fifo is not enabled. If fifo not used, it will

  block writing process (i.e. hang daemons)

- Improved MODE SELECT / SENSE to support 'changeable field bitmaps

- Started implementing ALLOW OVERWRITE support (incomplete)

- Fixed INQUIRY string match for STK T10000A

mhvtl-2012-01-21.tgz / mhvtl-1.2-1:

 - Add selectable compression type lzo or existing zlib

 - Add timestamps to 'state notification' messages

note: These changes mean you need lzo-devel package installed to compile 1.2-x

mhvtl-2012-01-20.tgz / mhvtl-1.2-0:

Never released - memory leak bug identified before posting.

mhvtl-2011-12-24.tgz / mhvtl-1.1-1:

 - Clean up IPC resources when finished with them

 - Simplify fifo reference counting

 - Start including Scientific Linux patches (Still need to include kmod RPM)

 - Fix 'make install' on Ubuntu

 - Various log message cleanups

mhvtl-2011-10-09.tgz / mhvtl-1.1-0:

 - Add 'real time notification' via a named pipe.

   vtltape & vtllibrary now take an optional '-f <named pipe>'

   Any chance of state within the daemon will be sent to the named pipe ready for use-application

   consumption.

- Log Sense page 0x0c (SEQUENTIAL ACCESS DEVICE PAGE) now updates wit valid data.

- MODE SENSE for LTO devices now update the 'media type' field as typical with real drives.

Just in time for fathers day a jump from 0.18-17 to 1.0...

mhvtl-2011-09-11.tgz / mhvtl-1.0-1:

  - Added mode page 25h (vendor specific) for IBM LTO3/4/5

    This allows the Windows IBM Tape Driver to load correctly (connected via iSCSI)

  - Attempt to correct file permission/ownership when media is created manually

  - A HUP signal to the vtllibrary daemon will cause it to re-read its config

    from the /etc/mhvtl/library_contents.XX file. (So you can change the slot

    config without having to re-start the daemon).

Note: Two current known bugs which are being worked on.

- If the '-f fifo' is not correctly defined or can't be created, the daemons will segfault on exit.

- The first daemon will result in an unlink(fifo); on exit.

mhvtl-2011-09-04.tgz / mhvtl-1.0-0:

Improvements

  - Re-worked MODE SENSE/SELECT data structures into a linked list.

  - Re-worked LOG SENSE/SELECT data structures into a linked list.

  - Added support for sub-page MODE information.

  - Added STK 9x40 drive emulations.

  - Removed dead code from kernel driver.

  - Add working REPORT DENSITY support to the SSC.

      This change required the personality modules to define the list of support

      densities. The media density is defined at 'mktape' time.

      Hence, the optional 'media load' rules in device.conf are now redundant

      and not used.

      BUT: The option to load any media into any drive is also gone.

mhvtl-2011-06-25.tgz / mhvtl-0.18-17:

Improvements

  - Kernel module compile warning since 2.6.33

Fixes

  - Return correct sense from SPACE op code

  - Fix buffer overflow in vtllibrary : product_id[]

  - Test MAP port status before moving media in/out

Note: This version has been successfully tested with:

 - NetBackup 6.x, 7.0 & 7.1 (has always worked)

 - BackupExec 10.5 (Failed to span media in earlier versions of mhvtl)

 - ArcServ r15 SP1 (failed with earlier versions of mhvtl)

mhvtl-2011-05-22.tgz / mhvtl-0.18-16:

Improvements

 - Cleaning media behaves more like real (IBM LTO4) drive.

 - Implement OPEN/CLOSE IMPORT/EXPORT element OP code (Thanks Sebastian)

 - Kernel module support for 2.6.39 (Thanks Sebastian)

Fixes

  - SPACE op code - Space to end-of-data fixed (Thanks Sebastian)

    This fixes an issue triggered using Oracle Backup

  - REQUEST SENSE - Return correct data.

  - SPOUT - Return check_condition on some error paths

  - Cleaning Media - Return 'not ready' instead of 'ready' when loaded

  - Don't write FILEMARKS to WORM or Cleaning media (as appropriate)

mhvtl-2011-05-04.tgz / mhvtl-0.18-15:

Improvements

  - Implemented 'Personality Module' for each drive type

    (many cleanups due to 'PM' change fallout thanks to Sebastian)

  - Increase max barcode length to 16 chars

Fixes

  - Inquiry no longer incorrectly reports support for TrmIOP and CmdQue

  - Fixed block read/writes corruption if 'multiple blocks' specified

  - Fix Device Capabilities mode page - Don't advertise EXCHANGE MEDIUM support

mhvtl-2011-03-17.tgz / mhvtl-0.18-14:

Fixes:

  - kernel module build fixes for Linux kernel 2.6.37 and greater

Improvements

  - Add STK T10000C media/drive support

  - Always log 'fatal' errors

  - Cleanup to mhvtl.spec

  - Catch signals to prevent daemon terminating early.

mhvtl-2011-01-11.tgz / mhvtl-0.18-13:

Fix:

 - READ ELEMENT STATUS when initiator does not allocate enough memory for full list

 - Return ILLEGAL REQUEST for SCSI OP codes which are not supported.

Improvements:

 - Increase SENSE buffer from 38 to 96 bytes

mhvtl-2010-11-07.tgz / mhvtl-0.18-12:

Fix:

 - Positioning error if space 0 blocks/filemarks - Reported & fix thanks to Doug Charnley

 - Only support SECURITY PROTOCOL IN/OUT op codes on IBM LTO4 / IBM 03592E06 and STK T10000 drive types

 - Fix REQUEST SENSE. - Many thanks to Andreas Piesk

 - Reworked vtllibrary SCSI OP code processing.

 - Yet another round of WORM media handling issues - Thanks Albert Pauw

 - Fix kernel module compile error with 2.6.34 kernels

mhvtl-2010-09-23.tgz / mhvtl-0.18-11:

Fix:

 - Potential core dump where SENSE data pointer incorrectly set to NULL

 - Positioning to filemark, followed by write failed to remove cached 'filemark pointer'.

   This caused errors with TSM & HP Dataprotector when writing multiple sequential backups on the same piece of media.

    Many thanks to Nai for testing on TSM

    Many thanks to Philip for testing with HP Dataprotector

 - VPD page 0x83 length incorrect (Thanks Gaetan)

 - VPD NAA data field incorrect (Thanks Gaetan)

Improvements:

 - Remove references to /proc

 - Daemons now create their own device nodes, so no external scripts need to be run. (Note: This required assistance from the kernel module - hence the kernel module from this release needs to be compiled and used - Sorry)

 - Gentoo ebuild file (Thanks Gaetan)

 - RedHat/CentOS kernel module SPEC file (Thanks Dag)

 - Use 'setuidgid' to start daemons if available instead of specified user account.

mhvtl-2010-09-01.tgz / mhvtl-0.18-10:

- Fix TapeAlert

- Added more descriptive logging around SPACE OP code attempting to track down problem with HP Dataprotector

- Added (undocumented) utility 'tapeexerciser' which is a simple utility to test tape read/writes and positioning. WARNING: This will overwrite your media..

- Fix 'is vtl running' logic in rc script. Failed test if 'vtl user' is root (not a recommended config)

mhvtl-2010-07-09.tgz / mhvtl-0.18-9:

- Fix WORM handling - Reported by Albert Pauw

- Fix (finished) LTO5 drive & media handling introduced in 0.18-8

mhvtl-2010-06-23.tgz / mhvtl-0.18-8:

- Several Security Protocol IN updates - thanks Albert Pauw

  * Return certificate data

  * Correct length for 'KEY FORMATS'

  * Correct length for SPIN SUPPORTED PAGES

- Fix kernel compile on RedHat AS4

- Media/drive matching now 'dynamic' and defined in device.conf

- Added man page for device.conf

- Fix media corruption when media is 'formatted'

- Add LTO5 & SDLT-S4 drive/media types

- Handle INQUIRY correctly after media change (return SAM_STAT_GOOD)

- Updated rc script so all devices created on Target & LUN. i.e. Don't use

  channel. Some application software has trouble if only the channel is unique.

mhvtl-2010-05-08.tgz / mhvtl-0.18-7:

Fix: 32bit platforms - Allow media size to be greater than 2G

Fix: core dump on invalid defined NAA strings

Fix: Support SMC devices with embedded space chars in VENDOR ID

Implement STK Vendor unique LOAD DISPLAY SCSI op code. (Logs display info via syslog)

mhvtl-2010-05-02.tgz / mhvtl-0.18-6:

Fix: vtlcmd command line parsing. Fix 'list map' option

Fix: import of media via MAP (off-by-one)

Fix: TapeCapacity LOG SENSE data.

      Now returns bytes/KB/MB as per OEM (Thanks 'kamy')

Support VENDOR ID with embedded spaces. i.e. "HP Ultrium"

Honour MAP status: Fail attempts for robot to move media in CAP if CAP is open.

Relax dependencies on /proc in favour of /sys (note: Work in progress)

mhvtl-2010-04-01.tgz / mhvtl-0.18-5:

Fix: vtlcmd command line parsing. Thanks Herbert Stadler

Fix: Silence local_irq_save(). Normally seen on fedora. Thanks Norm Lunda

Fix: URLs to this homepage.

mhvtl-2010-03-05.tgz / mhvtl-0.18-4:

Fix: ELEMENT STATUS - found and fixed by Norm Lunde (many thanks)

 - This was preventing TSM to work with mhvtl

Add queue depth() callback in kernel module (module date string 20100303)

Test for media mounted before returning status on 'REWIND' op code.

Fix typos in rc script

mhvtl-2010-01-28.tgz / mhvtl-0.18-2...rpm:

The ability to support multiple libraries on the one host !! Thanks once again to Kevan.

Note: There is no conversion of existing configuration file(s) to the new device.conf/library_contents format. Please remove all files under /etc/mhvtl/ before installing mhvtl-2010-01-28 (mhvtl-0.18-2)

What hardware does the mhvtl code base work with:

Tested and developed on Pentium x86 (32bit) and Pentium / AMD x86_64

New: IBM Z-Series. To quote:

"We just fired up the VTL on our z series IBM mainframe. Works like a charm."

What software does it work with:

Symantec NetBackup.

• 5.1 (Recommend 5.1MP4 minimum)

• 6.0

• 6.5.x

• 7.0.x

• 7.1.0.x

• 7.5.0.x

• 8.x

• Supports SCSI Persistent Reservation

• Encryption via KMS (NetBackup 6.5.2 and later)

Note: All mhvtl code development is tested using NetBackup

Last set of testing (0.18-8 & -9) with NetBackup 7.0, I had to revert the 'STK/L700' to 'SPECTRA/PYTHON' change made back in Nov 2009. It's just a manual edit of /etc/mhvtl/device.conf

The problem is that NetBackup 7.0 fails to extract the device serial number if SPECTRA/PYTHON is used and hence the automagic device discovery & configuration fails to place the drives within the robot.

Note: Recent NetBackup releases (post 8.2) require custom udev rules to be implemented. NetBackup now requires /dev/tape/by-id/ and /dev/tape/by-path/ devices to exist. Default linux distros do not know about mhvtl kenel and hence do not create the devices.

Executing the 'build-persistent-tape-rules.sh' script in the 'scripts' directory will generate a set of rules compatible for NetBackup

e.g.

$ ./scripts/build-persistent-tape-rules.sh 

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab000100", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab000200", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab000300", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab000400", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab000900", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab001000", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab001100", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

ACTION=="add|change", KERNEL=="nst*", ENV{SUBSYSTEM}=="scsi_tape", ENV{ID_SERIAL_SHORT}=="50223344ab001200", SYMLINK+="tape/by-path/%E{ID_VENDOR}-%E{ID_MODEL}-%E{ID_SCSI_SERIAL}-nst", MODE="0666"

Add the output information to a udev rule of your choosing

e.g.

$ sudo ./scripts/build-persistent-tape-rules.sh > /etc/udev/rules.d/70-persistent-by-path-tape.rules 

Symantec BackupExec.

For the extraordinary writeup : http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37

Includes how to setup the scst (http://mhvtl.nimsa.us/viewtopic.php?f=23&t=32) to present the vtl via iSCSI

I've receive updates from a anonymous contributor identified as 'nia':

EMC/Legato NetWorker

Is no longer causing headaches !

Don't use 'STK/L700' Use 'SPECTRA/PYTHON' instead.

Install at least mhvtl-2009-12-16.tgz (mhvtl-0.16-11).

Update: The latest mhvtl-2010-09-23.tgz (mhvtl-0.18-11) is a smoother ride when configuring the library using jbconfig.

Build kernel module from source (Note: /usr/src/packages/ is a SuSE RPM home dir. If you building on RedHat, then this will be /usr/src/redhat/)

# rpm -Uvh mhvtl-0.18-11.src.rpm

# cd /usr/src/packages/BUILD

# tar xvfz ../SOURCES/mhvtl-2009-11-23.tgz

# cd mhvtl-0.18/kernel

# make

# make install

Start the daemons

/etc/init.d/mhvtl start

Configure NetWorker as normal via jbconfig.

An ex colleague who has some very useful/helpful tips with NetWorker (If your interested in NetWorker, his blogs & tips are well worth following)

http://nsrd.info/blog/2009/07/13/carry-a-jukebox-with-you-if-youre-using-linux/

Also a follow up article

http://nsrd.info/blog/2009/11/14/networker-and-linuxvtl-redux/

A detailed setup of CentOS + NetWorker, see:

http://nsrd.info/blog/2010/10/14/new-micromanual-linuxvtl-and-networker/

CommVault

2015-02 - Thanks to Anup Sreedharan who has written up a 100+ page document which provides step-by-step instructions on setting up mhvtl + mhvtl-GUI + CommVault.

The guide is available via this link Comvault Implementation

Questions / kludos all go to Anup Sreedharan

Email: anupsreedharan@gmail.com

Linked in: http://in.linkedin.com/pub/anup-sreedharan/19/448/908

Bakbone Netvault

(2010-01-09) Thanks to Nick Couchman who has reported NetVault 8.x and mhvtl-2009-12-16 (mhvtl-0.16-11) do indeed work together.

To quote Nick "I have a NetVault 8.x installation here that I'm using, and I just installed LinuxVTL2 and am successfully backing up data from NetVault to LinuxVTL2. I'm not sure if any issues exist for restores and the link, but I'll let you know if I run into anything.  All appears to be working fine at this point!  FWIW I'm running my NetVault server on RHEL5."

TSM

Thanks to Bernardo Clavijo, TSM now works with mhvtl - Need at least 0.18-4 release.

Bacula

A report from Beat Rubischon to advise Bacula works also.

Two changes were needed on Bacula's

bacula-sd.conf:

Device {

 Name = Drive-1

 Drive Index = 0

 Media Type = LTO-4

 Archive Device = /dev/nst0

 AutomaticMount = yes;

 AlwaysOpen = yes;

 RemovableMedia = yes;

 RandomAccess = no;

 AutoChanger = yes

 Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"

 # Needed for mhvtl tape drives

 Hardware End of Medium = No

 Fast Forward Space File = No

}

Multiple tape drives are supported too.

New: 2010-05-21

Report by Albert Pauw.. Many thanks.

# Needed for mhvtl tape drives

Hardware End of Medium = No

Fast Forward Space File = No

I have used it this way for the last two months, but due to some questions on the mhvtl forum I tried

bacula without these additions. Seems the latest versions of mhvtl work out of the box with bacula,

so these additions are not needed anymore.

Getting Started

A list of recommended base packages for trouble-shooting SCSI devices on Linux

- sg3_utils

- mt-st

- mtx

- lsscsi

RedHat/CentOS/Scientific Linux/Oracle Linux distributions:

If all you want/need to do is get the VTL up and running as soon as possible, with a minimum of fuss:

Many thanks to the ELRepo project : http://elrepo.org/tiki/tiki-index.php

Import the ELRepo public key:

rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org

Decide if your distribution is based around RH 5 or RH 6

If RH 5:

Install the ELRepo RPM for RHEL-5 / SL-5 or CentOS-5

rpm -Uvh http://elrepo.org/elrepo-release-5-4.el5.elrepo.noarch.rpm

If RH 6:

Install the ELRepo RPM for RHEL-6 / SL-6 or CentOS-6

rpm -Uvh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpm

Install mhVTL packages:

yum install kmod-mhvtl mhvtl-utils

Start mhVTL:

services mhvtl start

If using systemctl : systemctl start mhvtl.target

All that's left is to configure your backup application..

Non RedHat based RPM Linux distributions:

This requires the gcc development environment to be installed as there is a pseudo SCSI hba driver to compile and install.

To install and get the package running:

1. Install both src.rpm & x86/x86_64 (depending on your CPU/OS version - check using uname -p)

2. Build kernel module from source you just installed:

Note: The directory in the example below is a SuSE RPM home dir (which is /usr/src/packages)

         If you are installing on RedHat:

            /usr/src/redhat or $HOME/rpmbuild/ (depending on release of RedHat)

        Other RPM distributions may have different default RPM home directories.

  cd /usr/src/packages/BUILD

  tar xvfz ../SOURCES/mhvtl-2013-03-22.tgz

  cd mhvtl-1.4/kernel

  make

  make install

3. Start the package using the rc script.

  /etc/init.d/mhvtl start

3. Start the package using the systemctl.

# systemctl start mhvtl.target

4. Check it's all running:

  lsscsi -g

  ps -ef|grep vtl (default with 10 daemons - 2 vtllibrary & 8 vtltape)

5. Configure your backup software ;)

NON-RPM based Linux distributions: (or you just want to do it the manual way)

To install and get the package running:

1. Download the 'mhvtl-YYYY-MM-DD.tgz

2. Extract source code

 cd /some/where/safe

 tar xvfz /where/you/downloaded/mhvtl-YYYY-MM-dd.tgz

3. Build user-space daemons

 cd mhvtl-1.6

 make

Note: Since mhvtl-1.3, both lzo and zlib are required to build vtltape

Typical error (missing lzo devel package) looks like:

cc -Wall -Wshadow -g -O2 -D_LARGEFILE64_SOURCE  -I../kernel -DMHVTL_VERSION=\"1.4.0\" -DMHVTL_DEBUG -DUSR=\"vtl\" -DMHVTL_HOME_PATH=\"/opt/mhvtl\" -DMHVTL_CONFIG_PATH=\"/etc/mhvtl\"   -c -o vtltape.o vtltape.c

vtltape.c:92:25: error: lzo/lzoconf.h: No such file or directory

vtltape.c:93:23: error: lzo/lzo1x.h: No such file or directory

vtltape.c: In function 'uncompress_lzo_block':

To fix, install lzo development package and rerun the 'make' command

4. Add a user 'vtl' (not required as of 1.6-2 or later)

 sudo useradd -c "mhvtl user" vtl

5. Install binaries

 sudo make install

6.Pre 1.7-1 (2023-03-10) Build kernel module from source you just installed:

  cd kernel

  make

  make install

6. From 1.7-1 (2023-03-10) Use a more user-friendly script

/usr/bin/mhvtl_kernel_mod_build

7. Pre 'systemd' Start the package using the rc script.

  /etc/init.d/mhvtl start

Post 'systemd' Start the package using systemd

# systemctl start mhvtl.target

Note: If you can't find the 'mhvtl' rc script in /etc/init.d, you most likely swapped the order of 4 & 5.

    i.e. You ran 'sudo make install', found it failed due to missing 'vtl' user account and didn't re-run the 'make install' after creating the account.

8. Check it's all running:

  lsscsi -g

  ps -ef|grep vtl (default with 10 daemons - 2 vtllibrary & 8 vtltape)

9. Configure your backup software ;)

Compiling the kernel module

  tar xvfz mhvtl-2021-10-07.tgz (latest at time of edit)

  cd mhvtl-1.6/kernel

  make

  make install

  rmmod mhvtl (remove original kernel module - if loaded)

Load new module:

  modprobe mhvtl opts=0

Confirm module version

  cat /sys/modules/mhvtl/version (confirm version is 0.18.30)

Some nice things people are saying about the mhvtl

• It's really caught on here, I've noticed that some of our developers have stopped using real tape hardware altogether, it's just easier to use the VTL. Plus our customer support has started doing customer training with the VTL instead of real tape hardware. Pretty impressive! A fine piece of software, wish I had found it a few years earlier.

• The VTL has been very helpful in my testing and for customer demos. Your hard work on this is very much appreciated by myself and others on my team.

• I am one more user of your magical software vtl here at Symantec. Just want to say that it rocks

• I finally got round to trying to get this set up and I have to say.. You are a genius ! My laptop now has NBU 6.5.1 on Redhat 4 AS (on VMware) with a VTL robot and 8 drives !! Fantastic ! Thanks for all your work on this ! This will be very useful for me.

• I have been a Storage and Backup Consultant for many years specialising mainly in TSM but with some NetWorker and NetBackup thrown in for good measure. For years I have been grabbing old used tape libraries for my lab so I can do all sorts of testing but this has always been an expensive proposition. With mhVTL I can now provision so many virtualised backup environments in a very short timescales for test and it is a joy to have such a product available in my arsenal.

I can be reached via mark.harvey@nutanix.com or mhvtl2@gmail.com