How memory map can fill up process memory without any memory allocation

Introduction

Laymen explanation

Consider that you want to read file faster. Since hard-disk access is slow,so you coded to keep this file in memory and so, access will be simply as fast as any variable access. Its great. However, since it is mapped to main memory, the program will access it using pointer address. In 32 bit machine, pointer will be of 4 bytes and so, it will have 4GB(2^32) pointer addresses. Assume that you have done memory map of 2 files each of size 1.7GB. This file is shared across multiple programs. So, actually only one copy will exist in main memory. However, 3.4GB (2*1.7GB) space will be used by each program. So, most of address spaces are used. Note that program has not allocated any memory. However, it is almost full.

Technical explanation

Each process in a multi-tasking OS(Linux, windows runs in its own memory sandbox. This sandbox is the virtual address space, which in 32-bit mode is always a 4GB block of memory addresses.

One of the principal advantages of virtual memory is that each process has its own virtual address space, which is mapped to physical memory by the operating system. Moreover, A portion of the virtual address space must be reserved to the kernel to handle interrupts or system calls at any time(Ref:Below diagram).

This does not mean the kernel uses that much physical memory, only that it has that portion of address space available to map whatever physical memory it wishes. The mapping for the user-mode portion of the address space changes whenever a process switch happens(Ref: Below diagram).

Blue regions represent virtual addresses that are mapped to physical memory, whereas white regions are unmapped. In the example above, Firefox has used far more of its virtual address space. Below diagram shows the standard segment layout in a Linux process.

So, assume that memory mapped file is pretty large and think about the question :)

Memory footprint of a linux process

/proc/<pid>/status shows VmSize which tells total current size of address space used.

=================

deepk-ive->@deepk-newive$ cat /proc/27889/status

Name: ng-

State: S (sleeping)

Tgid: 27889

Pid: 27889

PPid: 27795

TracerPid: 0

Uid: 28544 28544 28544 28544

Gid: 756 756 756 756

Utrace: 0

FDSize: 256

Groups: 756 950 2617 20021

VmPeak: 3752428 kB

VmSize: 3488240 kB

VmLck: 0 kB

VmHWM: 21588 kB

VmRSS: 20800 kB

VmData: 6776 kB

VmStk: 96 kB

VmExe: 1732 kB

VmLib: 53188 kB

VmPTE: 164 kB

VmSwap: 0 kB

Threads: 1

SigQ: 0/24092

SigPnd: 0000000000000000

ShdPnd: 0000000000000000

SigBlk: 0000000000000000

SigIgn: 0000000000001000

SigCgt: 0000000188004800

CapInh: 0000000000000000

CapPrm: 0000000000000000

CapEff: 0000000000000000

CapBnd: ffffffffffffffff

Cpus_allowed: 3

Cpus_allowed_list: 0-1

Mems_allowed: 1

Mems_allowed_list: 0

voluntary_ctxt_switches: 100743

nonvoluntary_ctxt_switches: 79

=====================

/proc/27889/smaps shows in detail memory map info, for example RSS (Resident Set Size).

Reference

https://www.kernel.org/doc/gorman/html/understand/understand007.html

http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory/

Related