Using APTrust‎ > ‎

Bagging

THIS CONTENT IS MOVED TO "THE WIKI" here now. Please update that location going forward. 

Bag Names

Root directories for bags will be named using a combination of institutional id as determined by the institutional profile inside of APTrust and the unique identifier of the item to be preserved. Dots in the bag root name should be used as delimiters between name parts as designated above and any dots or other special characters normally found in either institution ID or item unique ID should be truncated or converted to dashes or underscores. 

Multipart bag names must end with ‘b###.of###’ where ### is the number of that bag in the bag count.  Bag count sequences begin at 001. 

For example, if the University of Virginia has institutional code ‘virginia.edu’ and is creating a bag for an item with the unique ID ‘uva-lib:1229365’ then the bag root directory should be named ‘virginia.edu.uva-lib_1229365’

If this was a 200 multipart bag then the first bag root directory could be named ‘virginia.edu.uva-lib_1229365.b001.of200’, the second ‘virginia.edu.uva-lib_1229365.b002.of200’, and the last bag being ‘virginia.edu.uva-lib_1229365.b200.of200’.  When tarred these will of course carry the .tar extension for for example ‘virginia.edu.uva-lib_1229365.b016.of200.tar’

We enforce bag naming conventions because when we untar bags in a staging area to validate their contents, we don't want bags untarring to the same directory and overwriting each other.

 NAME         OK? Comment
photos.tar No Institution name is missing.
ncsu.photos.tar Yes* Should untar to a directory called ncsu.photos
ncsu.edu.photos.tar Yes* Should untar to a directory called ncsu.edu.photos
ncsu.edu.photos.b1.tar No Should be b01.of10, assuming there are 10 parts to this bag.
ncsu.edu.photos.b01.of10.tar Yes Note the dot before "of10", and note that ".tar" comes at the end.


* Because early versions of this document were unclear, some institutions uploaded bags with names like "institution.bag.tar" while others used "institution.edu.bag.tar," and the system accepted both naming schemes. The system will continue to accept both naming schemes, but for the sake of consistency, and to simplify your internal processes, you should stick with one or the other.

Bag Size Limits

As of January 30, 2017, the production repository will accept bags up to 5TB in size.

The demo repository accepts bags up to 100MB in size.

File and Directory Names

File and Folder names must follow POSIX conventions:
  • Contain upper or lower case letters, numbers, dots, underscores, percent signs, or dashes. (A–Z a–z 0–9 . _% -)
  • May contain virtually any printable character, except newlines, carriage returns, tabs, vertical tabs and ASCII bells. (As of 1/30/2017)
  • Are considered case sensitive.
  • MUST not begin with a dash. (-)
  • MUST not contain whitespaces
  • May contain whitespaces. (As of 1/30/2017)
  • Restricted to 255 characters in length including extension.
  • MUST be at least 1 character in length.
Other Considerations to be aware of:
  • Generic Files in APTrust are referenced by their uri, which is the original filepath relative to the bag. This will support atomistic updating of items in the future.
  • File and folder names should be unique across multi-part bags to make sure all items are processed and not treated as a file update.
  • Though APTrust does not currently version files, you can easily create item versions in APTrust by writing files to a bag using the datetime stamp in the filename.
  • File and Folder names are treated as case sensitive for processing purposes.


Bag Structure

Bags must have the following structure. Items in bold are required. Others are optional. Additional notes appear below. Note the new rules on manifests!

<institution_id.item_uid[.b###.of###]>/

|    aptrust-info.txt

|    bag-info.txt

|    bagit.txt

|    manifest-md5.txt and/or manifest-sha256.txt

|    tagmanifest-md5.txt

|    tagmanifest-sha256.txt

|    [custom tag files]

\----data/

    |    [payload files]

\----[custom_tag_dir]/

    |    [custom tag files]

\----[custom_tag_dir]/

    |    [custom tag files]


Manifests

Before March 29, 2016, APTrust accepted and verified only the manifest-md5.txt file. We will now accept either manifest-md5.txt or manifest-sha256.txt. If you supply both, we will validate both.

Tag Manifests

We will validate all tag manifests, though they are optional. We will accept tag files not listed in the tag manifests, though obviously we cannot validate their checksums.

Custom Tag Files

As of March 29, 2016, we preserve all tag files, except bagit.txt, which will be recreated when you restore a bag. Custom tag files may be in any format, including binary. We will not try to parse them, but we will validate their checksums if they are listed in the tag manifests.


Required Tag Files

bagit.txt

This is requited by the BagIt specification, and should contain the following:


BagIt-Version:  0.97

Tag-File-Character-Encoding:  UTF-8

bag-info.txt

Valid APTrust bags MUST contain a bag-info.txt file with the following fields, which may be blank:

  • Source-Organization: This should be the human readable name of the APTrust partner organization. For example, "University of Virginia." You may be more specific, if you wish, specifying a specific college or library within the university, such as "Georgetown University Law Library." However, when APTrust restores bags, the source organization in the bag-info.txt file will be set to the name of the partner institution. 
  • Bagging-Date: using ISO 8601 UTC format.
  • Bag-Count: as per specification
  • Internal-Sender-Description: [Optional] Human readable description of the contents of the bag. This will appear as the bag description in our web UI.
  • Internal-Sender-Identifier: [Optional] Internal or alternate identifier used at the senders location. This will appear in the web UI, and you can search for bags using this identifier.
This file MAY contain additional fields.

aptrust-info.txt

This file MUST be present and MUST contain the following tag fields.
  • Title: Human readable title for searching and listing in APTrust. This cannot be empty.  
  • Access: One of three enumerated access conditions. [“Consortia”, “Restricted”, “Institution”]. These access restrictions describe who can see the object metadata, including the object's name and description, a list of its generic files and events. APTrust does not currently provide access to the objects themselves, except when the owning institution restores a bag it owns. In other words, no matter which access setting you choose, no other institution can access your intellectual object. The general public cannot see any information in the APTrust system.
    • Restricted: Metadata about this object is accessible to the institutional administrator (at the depositing institution) and to the APTrust admin. No one else can even see that this object exists in the repository.
    • Institution: All users at the depositing institution can see metadata about this object.
    • Consortia: All APTrust members can see this object's metadata. 

Bag Serialization

Bags serialize for use by APTrust must use TAR as their serialization format, MUST not use compression and MUST follow the file and folder naming restrictions as well as end with the .tar extension.

Bag Size

Initially bags sent to APTrust should be limited to 250 GB for the final tarred bag. Space available for temporary file processing puts a practical limit on total bag sizes in APTurst. We expect this limit to grow over time but the initial performance data will help determine the final limits for the service.

Quick Checklist

Valid bags meet all of the following criteria:
  • The bag was submitted as a tar file, without compression
  • Bag name follows the pattern <institution.edu>.bag_name[.b###.of###].tar. 
  • Bag untars to a directory whose name matches the name of the tar file, minus the .tar extension.
  • Bag contains an md5 or sha256 manifest (or both)
  • Bag contains the data directory
  • Bag contains bagit.txt, as described above
  • Bag contains bag-info.txt as described above
  • Bag contains aptrust-info.txt as described above
  • All data files are in the manifest, and all checksums matched
  • All tag files mentioned in the tag manifest are present, and checksums match (you may omit tag files from the tag manifests)

Bagging Interest Group


The APTrust Bagging Interest Group discusses bagging best practices, problems, and concerns, and posts occasionally to the APTrust Group Space.



Comments