Deep dive into mmap - memory mapped I/0

Introduction

Laymen explanation

If you have multiple processes which needs data and these data is stored as file, then reading and writing data by these program can result in multiple times disk access.Can we reduce this? We know that accessing memory is much faster than disk access. So, can we keep a copy of this file in memory and then access directly via memory? This is done by mmap.

Technical explanation

On modern operating systems, it is possible to mmap (pronounced “em-map”) a file to a region of memory. When this is done, the file can be accessed just like an array in the program.

This is more efficient than read or write, as only the regions of the file that a program actually accesses are loaded. Accesses to not-yet-loaded parts of the mmapped region are handled in the same way as swapped out pages.

Since mmapped pages can be stored back to their file when physical memory is low, it is possible to mmap files orders of magnitude larger than both the physical memory and swap space. The only limit is address space. The theoretical limit is 4GB on a 32-bit machine - however, the actual limit will be smaller since some areas will be reserved for other purposes. If the LFS interface is used the file size on 32-bit systems is not limited to 2GB (offsets are signed which reduces the addressable area of 4GB by half); the full 64-bit are available.

Memory mapping only works on entire pages of memory. Thus, addresses for mapping must be page-aligned, and length values will be rounded up. To determine the size of a page the machine uses one should use

There are two important modes as below

• MAP_PRIVATE— Writes to the memory range should not be written back to the attached file, but to a private copy of the file. No other process sees these writes. This mode may not be used with MAP_SHARED.

• MAP_SHARED— Writes are immediately reflected in the underlying file rather than buffering writes. Use this mode when using mapped memory for IPC. This mode may not be used with MAP_PRIVATE.

Caution

Sometime big memory map size can result in memory allocation error. Refer How memory map can fill up process memory without any memory allocation for more insight.

read() steps:

1. System call copies data from disk to page cache (may or may not page fault, data may already be in page cache causing this to be skipped)

2. Data copied from page cache to process memory (may or may not page fault)

Memory map steps:

1. System call to create virtual mappings - very expensive

2. Process accesses memory for the first time, causing a page fault - expensive (and may need to be repeated if paged out)

3. Process actually reads the memory

Use-cases for mmap

• mmap is great if you have multiple processes accessing data in a read only fashion from the same file

  • It reduces number of system calls

  • It reduces the memory need as only one memory copy is shared among these processes

    • It reduces Rea/write time for a process

• mmap is also useful for inter process communication. You can mmap a file as read / write in the processes that need to communicate and then use synchronization primitives in the mmap'dregion (this is what the MAP_HASSEMAPHORE flag is for).

• mmap has the advantage when you have random access on big files.

• DBMS like SQLite has the option of accessing disk content directly using memory-mapped I/O

Coder advantages of mmap

• you can access it with memory operations (memcpy, pointer arithmetic), without bothering about system call. For this, mmap will copy disk data into memory using contiguous block of addresses.

When not to use mmap

• • Accessing very large file using 32 bit pointer is not right. This is because mmap has to find a contiguous block of addresses in your process's address space that is large enough to fit the entire range of the file being mapped. So, it may be out of available address pointer range.

• mmap can't be used on things like pipes and ttys.

  • There is initial cost of setting up the mapping. For short lived processes, mmap should be used only if this cost is superseded by use of the data for enough frequency(access pattern).

  • If you're streaming a 30 GB video file on a machine with 4 GB of RAM, and you never go back and reread any of the data, memory-mapping the file is probably the worst way to read it

  • The standard I/O approach is costly due to system call overhead and memory copying. The memory-mapped approach has its cost in minor page faults—when a block of data is loaded in page cache, but is not yet mapped into the process's virtual memory space. This may result in slowness for small file where data is accessed rarely. So, needs to be instrumented for this case.

What happens if underlying file changes after mmap() call

• For MAP_PRIVATE option in mmap, It is unspecified whether changes made to the file after the mmap() call are visible in the mapped region.

  • For MAP_SHARED option in mmap, it is reflected to the memory mapped region

Experiment with underlying file change after mmap() call

Compile this code

copy a file content to another file using mmap

#include <sys/types.h>#include <sys/stat.h>#include <sys/mman.h> /* mmap() is defined in this header */#include <fcntl.h>void err_quit(char *msg){ printf(msg); return 0;}int main (int argc, char *argv[]){ int fdin, fdout; char *src, *dst; struct stat statbuf; int mode = 0x0777; if (argc != 3) err_quit ("usage: a.out <fromfile> <tofile>"); /* open the input file */ if ((fdin = open (argv[1], O_RDONLY)) < 0) {printf("can't open %s for reading", argv[1]); return 0; } /* open/create the output file */ if ((fdout = open (argv[2], O_RDWR | O_CREAT | O_TRUNC, mode )) < 0)//edited here {printf ("can't create %s for writing", argv[2]); return 0; } /* find size of input file */ if (fstat (fdin,&statbuf) < 0) {printf ("fstat error"); return 0; } /* go to the location corresponding to the last byte */ if (lseek (fdout, statbuf.st_size - 1, SEEK_SET) == -1) {printf ("lseek error"); return 0; } /* write a dummy byte at the last location */ if (write (fdout, "", 1) != 1) {printf ("write error"); return 0; } /* mmap the input file */ if ((src = mmap (0, statbuf.st_size, PROT_READ, MAP_SHARED, fdin, 0)) == (caddr_t) -1) {printf ("mmap error for input"); return 0; } /* mmap the output file */ if ((dst = mmap (0, statbuf.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fdout, 0)) == (caddr_t) -1) {printf ("mmap error for output"); return 0; }

printf("sleeping for 30 seconds. Please modify input file\n");

sleep(30);

printf("Hoping that you have modified input file\n");

/* this copies the input file to the output file */ memcpy (dst, src, statbuf.st_size); return 0;} /* main */

This code will sleep for 30 seconds allowing you to edit the input file content. You will see that it has read edited content of file and written to the output file.

Edited input file is read by mmap

In Shell-1

deepak@containerd121:~/deepak/mmap$ ./a.out data.in data.out

sleeping for 30 seconds. Please modify input file

Hoping that you have modified input file

deepak@containerd121:~/deepak/mmap$

In Shell-2 at the same time (during 30 second wait)

deepak@containerd121:~/deepak/mmap$ echo "my name is deepak" > data.in

deepak@containerd121:~/deepak/mmap$ echo "my name is raj" > data.in

deepak@containerd121:~/deepak/mmap$ cat data.in

my name is raj

deepak@containerd121:~/deepak/mmap$ echo "my name is no" > data.in

deepak@containerd121:~/deepak/mmap$ cat data.in

my name is no

deepak@containerd121:~/deepak/mmap$ cat data.out

<<After finish of 30 seconds wait>>

deepak@containerd121:~/deepak/mmap$ cat data.out

my name is no

deepak@containerd121:~/deepak/mmap$

Reference

http://man7.org/linux/man-pages/man2/mmap.2.html

http://www.gnu.org/software/libc/manual/html_node/Memory_002dmapped-I_002fO.html

http://www.makelinux.net/alp/037

https://stackoverflow.com/questions/258091/when-should-i-use-mmap-for-file-access

https://www.sqlite.org/mmap.html

https://unix.stackexchange.com/questions/474926/how-does-memory-mapping-a-file-have-significant-performance-increases-over-the-s/475014