Singularity Container

Official Web Site: https://sylabs.io/singularity/

Documentation: https://sylabs.io/guides/3.4/user-guide/

Singularity container is for running Linux OS and the security model is mainly targeted for users running applications in high-performance computing environment. The version we examined is 3.4.1, which is the latest version during the testing time, October 2019. Requirement for testing is only a laptop running Linux OS. The installation is tested on a Ubuntu 18.04 host. Singular security model is to allow untrusted users to run untrusted containers. Singularity does not need sudo privileges to run a container, but in order to build a container with writeable images you do need sudo privileges. This means an existing container from Docker will run as it is, but if you want to build a container yourself then you need sudo privilege.

Installation Ubuntu:

System Dependencies

1. sudo apt-get update && sudo apt-get install -y \ build-essential \ libssl-dev \ uuid-dev \ libgpgme11-dev \ squashfs-tools \ libseccomp-dev \ pkg-config

Installing Go, since Singularity is written in Go (https://golang.org/dl/). Download the appropriate version and tar the file into /usr/local directory.

2. export VERSION=1.12 OS=linux ARCH=amd64 && \ # Replace the values as needed wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \ # Downloads the required Go package sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \ # Extracts the archive rm go$VERSION.$OS-$ARCH.tar.gz # Deletes the ``tar`` file

Update the path in the $HOME/.bashrc file

3. echo 'export PATH=/usr/local/go/bin:$PATH' >> ~/.bashrc && \ source ~/.bashrc

Download Singularity as a tar file, untar, compile and install. By default, it gets installed to /usr/local/ directory.

4. export VERSION=3.4.2 && # adjust this as necessary \ wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \ tar -xzf singularity-${VERSION}.tar.gz && \ cd singularity

Compile the code and install, by default everything goes to /usr/local. If you want to change it use the option ./mconfig --prefix=/opt/singularity (for example)

5. ./mconfig && \ make -C builddir && \ sudo make -C builddir install

Upgrading to newer version (find /usr/local -name "*singularity")

6. sudo rm -rf \ /usr/local/libexec/singularity \ /usr/local/var/singularity \ /usr/local/etc/singularity \ /usr/local/bin/singularity \ /usr/local/bin/run-singularity \ /usr/local/etc/bash_completion.d/singularity

If you are upgrading repeat the steps 2 to 5 and you are ready to go assuming you have /usr/local/bin in your PATH

Singularity Files with suid bit and Security

sudo find /usr/local -perm -4000
-rwsr-xr-x 1 root root 15046736 Oct 28 09:33 /usr/local/libexec/singularity/bin/starter-suid
Refer: https://github.com/sylabs/singularity/releases/tag/v3.2.0 for security fix related to starter-suid.
cat /usr/local/etc/singularity/singularity.conf | grep -v ^# | grep -v ^$ to see all default options.
Run 'singularity --debug run library://sylabsed/examples/lolcow ' and watch the output to see what singularity is doing during the container deploying process.

Installation CentOS:

1. Install Dependencies

sudo yum update -y && \sudo yum groupinstall -y 'Development Tools' && \ sudo yum install -y \openssl-devel \libuuid-devel \libseccomp-devel \wget \squashfs-tools

2. Install Go

export VERSION=1.12 OS=linux ARCH=amd64 && \ wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \ sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \ rm go$VERSION.$OS-$ARCH.tar.gz

3. Add Go to the PATH

echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \ echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \ source ~/.bashrc

4. Download Singularity latest release:

export VERSION=3.4.2 && # adjust this as necessary \ wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \ tar -xzf singularity-${VERSION}.tar.gz && \ cd singularity

5. Install any missing packages

sudo yum -y install cryptsetup

6. Compile Singularity

./mconfig && make -C ./builddir && sudo make -C ./builddir install

Verifying

singularity help <command> [<subcommand>]singularity help buildsingularity help instance startsingularity help verify

Locations to download images from

Library (https://cloud.sylabs.io/library)
Docker hub (https://hub.docker.com/)
Build yourself (will need sudo privilege)

Example Commands

Download prebuilt images

Search a Container Library for users and containers matching the search query.
1. singularity search centos
pull and build are primary commands to download images from container library or docker. The main difference between pull and build is that, with pull you will get different containers at different time. If you use the build command you must specify a container name. build is also used in creating files from scratch using definition files or to create images from other formats.
2. singularity pull library://library/default/centos
Find out who signed it.
3. singularity verify centos_latest.sif
Use the build command to download a prebuilt image from external source.
4. singularity build ubuntu.sif library://ubuntu5. singularity build lolcow.sif docker://godlovedc/lolcow

Interact with Containers in shell mode

The command below will open a shell where you can execute commands like 'whoami', 'id', 'pwd'. That is also available in the container. But if you execute command 'python', it will tell you command not found because we don't have python installed in that container. 1. singularity shell lolcow_latest.sif2. singularity shell library://sylabsed/examples/lolcow # This is ephmeral meaning once you exit the container will disappear.

Execute commands from the container with exec

1. singularity exec lolcow_latest.sif cowsay "Hello World!"2. singularity exec library://sylabsed/examples/lolcow cowsay "Hello World!"3. singularity exec docker://godlovedc/lolcow cowsay "Hello World!"4. echo "Hello from inside the container" > $HOME/hostfile.txt5. singularity exec lolcow_latest.sif cat $HOME/hostfile.txt
By default Singularity bind mounts /home/$USER, /tmp, and $PWD into your container at runtime. Additionally, one can use --bind command to mount other directories over to the container
6. echo "Drink milk (and never eat hamburgers)." > /data/cow_advice.txt7. singularity exec --bind /data:/mnt lolcow_latest.sif cat /mnt/cow_advice.txt8. cat /data/cow_advice.txt | singularity exec lolcow_latest.sif cowsay
(Ref: https://sylabs.io/guides/3.4/user-guide/bind_paths_and_mounts.html)
9. ls -lt /data/-rw-rw-r-- 1 root root 39 Oct 23 07:55 cow_advice.txt10. singularity exec --bind /data:/mnt lolcow_latest.sif rm /mnt/cow_advice.txtrm: remove write-protected regular file '/mnt/cow_advice.txt'? yrm: cannot remove '/mnt/cow_advice.txt': Permission denied
(Ref: https://sylabs.io/guides/3.4/user-guide/cli/singularity_exec.html)(Ref: https://sylabs.io/guides/3.4/user-guide/cli/singularity_shell.html)(Ref: https://sylabs.io/guides/3.4/user-guide/build_a_container.html)

Exercise 1

1. singularity build --sandbox /tmp/debian docker://debian:latest2. singularity exec --writable /tmp/debian apt-get update3. singularity exec --writable /tmp/debian apt-get upgrade4. sudo singularity exec --writable /tmp/debian apt-get install python5. sudo singularity build /tmp/debian2.sif /tmp/debian6. singularity exec /tmp/debian2.sif cat /etc/debian_version7. singularity exec /tmp/debian2.sif python ./hello_world.py8. cat hello_world.py | singularity exec /tmp/debian2.sif python9. singularity instance start /tmp/debian2.sif debian10. singularity instance list11. singularity stop debian12. singularity instance list13. singularity shell /tmp/debian2.sif Singularity debian2.sif:~/container-images> pwd/home/ppk/container-imagesSingularity debian2.sif:~/container-images> exitexit
14. singularity shell -C /tmp/debian2.sifSingularity debian2.sif:~> pwd/home/ppkSingularity debian2.sif:~> ls -lttotal 0Singularity debian2.sif:~> exitexit

Running Container runscripts using run command

When the singularity containers are built the builder can add default run commands using runscripts as shown below in the def file.

%runscript echo "Hello World!" echo "Arguments received: $*" exec echo "$@"
1. singularity run lolcow_latest.sif2. ./lolcow_latest.sif

Building Images from Scratch

Put the container in a directory in your home directory. This is done with --sandbox option.

1. singularity build --sandbox ubuntu/ library://ubuntu2. singularity shell --writable ubuntu

Now you are inside the container

3. Singularity ubuntu:~/container-images> apt-get update && apt-get upgrade

You can see that apt-get upgrade command failing. Now let us do the same commands with sudo option after removing the previous directory.

4. rm -rf ubuntu5. sudo singularity build --sandbox ubuntu/ library://ubuntu6. sudo singularity shell --writable ubuntu
7. Singularity ubuntu:~> apt-get update && apt-get upgrade

Now you can see that apt-get upgrade command works. If you do an ls -ltR / then you can see some files are owned by root and some file are owned by the user. Now you can build a new sif file using the build command for the files in the ubuntu directory.

8. sudo singularity build updated-ubuntu.sif ubuntu9. sudo singularity shell --writable updated-ubuntu.sifSingularity updated-ubuntu.sif:~>

If you try apt-get update here the command will fail. This is to demonstrate the difference between a privileged creating the container as opposed to regular user creating the container.

Singularity Build using Definition file:

(Ref: https://sylabs.io/guides/3.4/user-guide/definition_files.html)

Save the text below as my_ubuntu.def in a file

-----------------------------------------------------------------------------

Bootstrap: libraryFrom: ubuntu:18.04Stage: build
%setup touch /file1 touch ${SINGULARITY_ROOTFS}/file2
%files /file1 /file1 /opt
%environment export LISTEN_PORT=12345 export LC_ALL=C
%post apt-get update && apt-get install -y netcat NOW=`date` echo "export NOW=\"${NOW}\"" >> $SINGULARITY_ENVIRONMENT
%runscript echo "Container was created $NOW" echo "Arguments received: $*" exec echo "$@"
%startscript nc -lp $LISTEN_PORT
%test grep -q NAME=\"Ubuntu\" /etc/os-release if [ $? -eq 0 ]; then echo "Container base is Ubuntu as expected." else echo "Container base is not Ubuntu." fi
%labels Author d@sylabs.io Version v0.0.1
%help This is a demo container used to illustrate a def file that uses all supported sections.

-----------------------------------------------------------------------------------------------------

1. sudo /usr/local/bin/singularity build --notest my_ubuntu.sif my_ubuntu.def2. singularity test my_ubuntu.sif3. singularity exec my_ubuntu.sif env | grep -E 'LISTEN_PORT|LC_ALL'4. singularity instance start my_ubuntu.sif instance15. lsof | grep LISTEN6. ./my_ubuntu.sif7. ./my_ubuntu.sif this that and the other8. singularity inspect my_ubuntu.sif9. singularity run-help my_ubuntu.sif

Page updated

Google Sites

Report abuse