Squash File System and Singularity

View available datasets on the Datasets page.

Working with Datasets

Writable ext3 overlay images have conda environments installed inside, Singularity can work with squashFS for fixed datasets, such as the coco datasets.

If you have many tiny files as fixed datasets, please make squashFS files to work with Singularity. Here is an example

1. Make a temporary folder in /state/partition1, it is a folder in local hard drive on each computer node

2. Unzip files there, for example

3. Change access permissions in case we'll share files with others

4. Convert to a single squashFS file on host

http://www.iitk.ac.in/LDP/HOWTO/SquashFS-HOWTO/mksqoverview.html

5. Copy this file to /scratch

6. To test, files are in /imagenet-example inside Singularity container 

7. To delete the tempoary folder on host