Playing with docker container file system

Introduction

Laymen explanation

If you have created a docker container, you might remember that you can mount external storage using -v option. Moreover, docker container allows you to read/write data as it uses host storage directly as default.

Ideally, very little data is written to a container’s writable layer, and you use Docker volumes to write data. However, some workloads require you to be able to write to the container’s writable layer. This is where storage drivers come in. Example of this workload will be storing binary, libraries, logs etc. So, should you care about the file system used by docker for this? You should. Note that it introduces an additional layer above the host file system. A wrong selection can result in performance overhead due to this extra layer. This article helps in this regard.

Technical explanation

Docker supports several different storage drivers, using a pluggable architecture. The storage driver controls how images and containers are stored and managed on your Docker host.

Supported storage drivers

https://success.docker.com/article/Compatibility_Matrix lists file system support based on the OS.

Docker default file system

When possible, overlay2 is the recommended storage driver. When installing Docker for the first time, overlay2 is used by default. Previously, aufs was used by default when available, but this is no longer the case. If you want to use aufs on new installations going forward, you need to explicitly configure it,

Why docker prefers overlay2

• • The copy-on-write (CoW) strategy: When you start a container, a thin writable container layer is added on top of the other layers. Any changes the container makes to the filesystem are stored here. Any files the container does not change do not get copied to this writable layer. This means that the writable layer is as small as possible. The OverlayFS copy_up operation is faster than the same operation with AUFS, because AUFS supports more layers than OverlayFS and it is possible to incur far larger latencies if searching through many AUFS layers.

  • Page Caching: OverlayFS supports page cache sharing. Multiple containers accessing the same file share a single page cache entry for that file.

When not to use writable layer (overlay2)

The container’s writable layer may grow quite large in write-heavy workloads. For example, aufs, and overlay2 all operate at the file level rather than the block level. This uses memory more efficiently, But, If we execute write permission to /var and subdirectories, then overlay2 will be forced to maintain a writable copy. It will result in extra storage use. If this operation is used in boot-up time, it will cause boot delay as well.

In this case, using volume will help. Volumes provide the best and most predictable performance for write-heavy workloads. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write.

Method to examine writable layer of container

Check disk-space used by writable layer of container

In below example, 0 bytes are used for writable layer (since there is no write done in this container)

Method to see disk space used for writable layer of container

deepak@deepak-VirtualBox:~/personal/docker$ sudo docker ps -s

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE

d747b9904f2b ubuntu:latest "/bin/bash" About a minute ago Up About a minute xenodochial_visvesvaraya 0B (virtual 123MB)

Below examine shows increase in size after write operation.

Text Box

deepak@deepak-VirtualBox:~/personal/docker$ sudo docker exec -it d74 bash

root@d747b9904f2b:/# touch test.txt

root@d747b9904f2b:/# exit

exit

deepak@deepak-VirtualBox:~/personal/docker$ sudo docker ps -s

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE

d747b9904f2b ubuntu:latest "/bin/bash" 14 minutes ago Up 14 minutes xenodochial_visvesvaraya 20B (virtual 123MB)

Check writable layer location in host machine

Below example shows output for overlay2/aufs file system based layers

Check overlay2 based writable layers in the host machine

deepak@deepak-VirtualBox:~/personal/docker$ sudo ls /var/lib/docker/overlay2

11849beee8b48c9d47b4545c685703b7d4984214823b08b8d1351320fc5163ea cc9c8ecdaab493c216d14350fab516fbd5b40ff9186a67f2ea0cc06ba01a1d18

14b1d0258b7a9d9ccb7e750e7d8fe2866203d155d17d9ad6e11c4f1738e75241 df36e42cdc2aa6cc55b9a2644170f9f66b6829f0e7bd75de61d670e9c66d5337

308a6fdc5bd50ee0ef7b6c4f934740840777895e7f1716faf0f107e003b14512 e3feea41db6daf0148947c0e343b68e79c1ec4d170b324613628eb4dc6d85b53

47f096eb7bb5ec42b55208de51520bd730e28a778441c66a9f276e67ec024ede e619705c7f647226643366195f3a81599b8510794d34c1e5e949acbe32ac0384

4a8d31a8615eefa5103be5a9432a7b233fc766dbb2b912f363012a0576fc37aa e807e4da8d2fec1d7afec075701c25a5420edf6dc1591aedde3c73c40ac9c9d7

4aac3765ab44b44d765db95b3208782c2f42557910bd2d015a1bb8d94cc0b482 ea530e797df82dbe1d27f0a3bc42013d76398e5fbd0a14fbd70a5bead8abb979

6b7895b7b6d0145da8723eb97aef6ddfa3730b7b2778739304073608a0eba61f ea530e797df82dbe1d27f0a3bc42013d76398e5fbd0a14fbd70a5bead8abb979-init

7c3912fae6855e3a10226d0561e67bdad7f6b1dc8883edbe30b1875a8b3acce0 f35ab082edcd4c2ad057c4d9f4e9677a6c58d4f5cb5bf75d68f2b44c7805c220

8422f0e77863721fef95033d44c8dbf245c9f349c4d7bf48017d0b4d3d6be8fd f86b4a46af3bc946c16652be333710392ea2dba2672a5f5950b164e646b7bdcd

a14c4dc32f91e09deb92abb76b0c6d34f5ee3e8130169e3fea1c2b96a08a39a0 fa7b35ae0a112e32943137176cf4f2960a40f8b6deef8a25618a79ac52a7fa4e

a5785ffa922da3aad6cd211e9c68dd11aee7b243fddd879ea5e7bcd19fbb730a l

check aufs file system based layer

[root@ubuntu ~]# ls /var/lib/docker/aufs/layers/

00201912527fdc2f036640075afe471aeb02b377f2b74b46cb254a8087749599 7ad28a065904faa7a99ed7bc9244d851fde514068ed4954215687b6967045975

03e0ba84e9e3da2f981250229d82c3f0573c4ea457e25b8d6718bcec4bd249ce 7cf4867796724cc60b5867676f06f93169e6bc8a3c62483f134dc74153fc6545

04fdc1959e263a33cd1464347df7fa6638b520c67381066c3b023399bc5f0f0b 7d307e44db8cd549418be195c0c37537164e1b0a266220747f3eb5897a62e7e2

05f47ed82b869af7df2459ed681727948c132ebdc980c58aa31f740dc4d14532 7e5589bc936c86bcad29cd9643160966bd267bc0a13ccf733b0d292176ae4428

0623fe6ce38a5db7bba678e3331b91e5529b4f126b89713c0c07ec2f65026187 7eed184d7a9925c78d6cfc03b9c29c3f51c9e6d887fad2b6d0ce3758c3b3d880

069de569010e0af86ba3467456262ae70d5c086de97dd9444dc8b7114bd754f7 7fde10adb89ec599d25e03b3769feb9e0b90df2316455ceadde4cd01fec76849

06fc571fe254f96fc9dcdba1f665e44504bfccaa9bb3fb5d942b7c447c0015be 80bff2d08e058fb9a650a40dd1ccd75e954ccfac8dae8e17b1bf170fba76253f

07dba0c115a4eab9ac9708d2de74f5b844a794a0ec7bd434fc208769eab520b0 816ed013d6425e45ba85c0fde5a2f5e73f374295c00a6cd89532c41f7f727d2d

07dba0c115a4eab9ac9708d2de74f5b844a794a0ec7bd434fc208769eab520b0-init 81c000b3a9f325a4a4e379a79e1e5fc7421fc1748568413a4fefa45e5d56fd54

086c91710e047d85952d583250ee2a07660f75e3ef410fe56255f67112fd7626 82033a5d38df536c35c2cdc4e7ef68e4afcb798cf1369af0cf2bd3b8cdc41e62

088ba95ab4b98d5c25e61b7a2991d23befe8bdd7a37a5cac1b7e6d3eb0fc0d05 846fc74dca7a60615d6b69164b5154e886236d7f15539f5a245e7c67333b595a

0b8d2e2c9342f29d8345bb22954eb73669d6fb8ee39eb4031e00f8c512296add 859ff07746c53741ed7addc06e2c43200c3b8628630a1690dba61dfb4a021e85

0ca516de1e33aed5a68185c6d4d8f14f00b839ae39f7aa489d17eccbb75b6759 85f68fc9c56a28990bd947edf99d2d4f360ac9d3ae7924567e519a056556362d

0cad8200a5d7459b3eb64bedebacf39537724ff88d27e46031674c9959505ac0 86f1bad5d805748f9b04b1716c2274d3e54532f42a6029941ef132a91f1d0243

0d38ca51b55c2bc20212e61fc7284a12f60180d01e2a4c54d447307d81b1a053 8783be3d3403f8c145a9e4b89623ee1a173fc87acf0b295354e9ad811f6e0cca

.....

Handling data volume (-v option)

A data volume is a directory or file in the Docker host’s filesystem that is mounted directly into a container. Data volumes are not controlled by the storage driver. Reads and writes to data volumes bypass the storage driver and operate at native host speeds.

When a container is deleted, any data written to the container that is stored in a data volume is not deleted.

Controlling writable layer

use /etc/docker/daemon.json to configure it

Controlling globally configuration of file system

deepak@deepak-VirtualBox:~/personal/docker$ sudo cat /etc/docker/daemon.json

{

"storage-driver": "aufs"

}

Reference

https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/#sharing-promotes-smaller-images

https://www.google.com/search?client=ubuntu&channel=fs&q=advantage+of+overlay+in+docker&ie=utf-8&oe=utf-8