Friday, July 07, 2017

Oracle Linux – understanding Linux process memory mapping with pmap

Every operating system uses memory for running processes. In most cases you do not worry about how memory is used, how it is mapped and what the internal processes are that are governing the memory. However, in cases where you start to run into issues with performance and memory looks to be one of the root causes your system is not performing the way you expect it the time has come to take a deep dive into the Linux internals.

One of the tools which can be used to gain more understanding of how memory is used in somewhat more detail is pmap. You can use pmap to gain more insight in the memory usage of one or multiple processes based upon a specified PID.

Redis memory usage example:
In this example we will dive a bit into the memory consumption of a Redis instance running on Oracle Linux. To find out the current PID used by Redis we execute a ps command to find it as shown below.

[root@localhost proc]# ps -ef| grep redis-server | head -1
redis     1100     1  0 Jul04 ?        00:01:36 /usr/sbin/redis-server /etc/redis.conf
[root@localhost proc]#

now that we know that the redis-server is running with PID 1100 we can start to dive into the memory consumption using pmap. The pmap command takes the PID as input to display the desired information. An example of a details view pmap command is shown below, the –x flag is used to ensure pmap shows the details view instead of the more compact version.

[root@localhost proc]# pmap -x 1100
1100:   /usr/sbin/redis-server /etc/redis.conf
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000     312     312       0 r-x--  redis-server
000000000064d000       8       8       8 rw---  redis-server
000000000064f000      84      84      84 rw---    [ anon ]
00000000021ff000     132       4       4 rw---    [ anon ]
00007f55223fe000       4       0       0 -----    [ anon ]
00007f55223ff000    8192       8       8 rw---    [ anon ]
00007f5522bff000       4       0       0 -----    [ anon ]
00007f5522c00000    8192    2048    2048 rw---    [ anon ]
00007f5523400000    4096    2048    2048 rw---    [ anon ]
00007f5523c00000    4096    2048    2048 rw---    [ anon ]
00007f55242c7000    1576    1112       0 r-x--  libc-2.12.so (deleted)
00007f5524451000    2048       0       0 -----  libc-2.12.so (deleted)
00007f5524651000      16      16      16 r----  libc-2.12.so (deleted)
00007f5524655000       8       8       8 rw---  libc-2.12.so (deleted)
00007f5524657000      16      12      12 rw---    [ anon ]
00007f552465b000      92      84       0 r-x--  libpthread-2.12.so (deleted)
00007f5524672000    2048       0       0 -----  libpthread-2.12.so (deleted)
00007f5524872000       4       4       4 r----  libpthread-2.12.so (deleted)
00007f5524873000       4       4       4 rw---  libpthread-2.12.so (deleted)
00007f5524874000      16       4       4 rw---    [ anon ]
00007f5524878000       8       8       0 r-x--  libdl-2.12.so (deleted)
00007f552487a000    2048       0       0 -----  libdl-2.12.so (deleted)
00007f5524a7a000       4       4       4 r----  libdl-2.12.so (deleted)
00007f5524a7b000       4       4       4 rw---  libdl-2.12.so (deleted)
00007f5524a7c000     524      60       0 r-x--  libm-2.12.so (deleted)
00007f5524aff000    2044       0       0 -----  libm-2.12.so (deleted)
00007f5524cfe000       4       4       4 r----  libm-2.12.so (deleted)
00007f5524cff000       4       4       4 rw---  libm-2.12.so (deleted)
00007f5524d00000     128     128       0 r-x--  ld-2.12.so (deleted)
00007f5524f14000      16      16      16 rw---    [ anon ]
00007f5524f1e000       4       4       4 rw---    [ anon ]
00007f5524f1f000       8       8       8 r----  ld-2.12.so (deleted)
00007f5524f21000       4       4       4 rw---  ld-2.12.so (deleted)
00007f5524f22000       4       4       4 rw---    [ anon ]
00007ffc44e66000     132      20      20 rw---    [ stack ]
00007ffc44fa4000       8       0       0 r----    [ anon ]
00007ffc44fa6000       8       4       0 r-x--    [ anon ]
ffffffffff600000       4       0       0 r-x--    [ anon ]
----------------  ------  ------  ------
total kB           35904    8076    6368
[root@localhost proc]#

The above output can be used to gain more insight into the way Redis is using memory. As we used the –x flag in the previous example we did have a detailed view. In case you want to see the offset and the device ID you can use the –d flag as show in the example below:

[root@localhost proc]# pmap -d 1100
1100:   /usr/sbin/redis-server /etc/redis.conf
Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000     312 r-x-- 0000000000000000 0fb:00001 redis-server
000000000064d000       8 rw--- 000000000004d000 0fb:00001 redis-server
000000000064f000      84 rw--- 0000000000000000 000:00000   [ anon ]
00000000021ff000     132 rw--- 0000000000000000 000:00000   [ anon ]
00007f55223fe000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f55223ff000    8192 rw--- 0000000000000000 000:00000   [ anon ]
00007f5522bff000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f5522c00000    8192 rw--- 0000000000000000 000:00000   [ anon ]
00007f5523400000    4096 rw--- 0000000000000000 000:00000   [ anon ]
00007f5523c00000    4096 rw--- 0000000000000000 000:00000   [ anon ]
00007f55242c7000    1576 r-x-- 0000000000000000 0fb:00001 libc-2.12.so (deleted)
00007f5524451000    2048 ----- 000000000018a000 0fb:00001 libc-2.12.so (deleted)
00007f5524651000      16 r---- 000000000018a000 0fb:00001 libc-2.12.so (deleted)
00007f5524655000       8 rw--- 000000000018e000 0fb:00001 libc-2.12.so (deleted)
00007f5524657000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f552465b000      92 r-x-- 0000000000000000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524672000    2048 ----- 0000000000017000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524872000       4 r---- 0000000000017000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524873000       4 rw--- 0000000000018000 0fb:00001 libpthread-2.12.so (deleted)
00007f5524874000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f5524878000       8 r-x-- 0000000000000000 0fb:00001 libdl-2.12.so (deleted)
00007f552487a000    2048 ----- 0000000000002000 0fb:00001 libdl-2.12.so (deleted)
00007f5524a7a000       4 r---- 0000000000002000 0fb:00001 libdl-2.12.so (deleted)
00007f5524a7b000       4 rw--- 0000000000003000 0fb:00001 libdl-2.12.so (deleted)
00007f5524a7c000     524 r-x-- 0000000000000000 0fb:00001 libm-2.12.so (deleted)
00007f5524aff000    2044 ----- 0000000000083000 0fb:00001 libm-2.12.so (deleted)
00007f5524cfe000       4 r---- 0000000000082000 0fb:00001 libm-2.12.so (deleted)
00007f5524cff000       4 rw--- 0000000000083000 0fb:00001 libm-2.12.so (deleted)
00007f5524d00000     128 r-x-- 0000000000000000 0fb:00001 ld-2.12.so (deleted)
00007f5524f14000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f5524f1e000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f5524f1f000       8 r---- 000000000001f000 0fb:00001 ld-2.12.so (deleted)
00007f5524f21000       4 rw--- 0000000000021000 0fb:00001 ld-2.12.so (deleted)
00007f5524f22000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007ffc44e66000     132 rw--- 0000000000000000 000:00000   [ stack ]
00007ffc44fa4000       8 r---- 0000000000000000 000:00000   [ anon ]
00007ffc44fa6000       8 r-x-- 0000000000000000 000:00000   [ anon ]
ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]
mapped: 35904K    writeable/private: 25012K    shared: 0K
[root@localhost proc]#

Additional flags that can be used are:
-q  quiet; less header/footer info
-V  show the version number
-A  limit results to the given range

As from version 3.3.4 also the flag –XX is introduced to provide even more information however this is not detailed in this post.

Reading pmap output:
Now that we understand how to get output from pmap the more important question is how to read and interpret this data so we can give it meaning.  The values returned by pmap are listed below, which values are returned is depending on the flags provided during execution of the pmap command.

Address
The address is to provide the start address of map.

Kbytes
The size of the page in kbytes. Depending on the flag given you will get a sum in the output of the pmap command.

RSS
RSS or resident set size which can be used to calculate how much of the memory is actually using main memory. The difference between the total amount of memory and the RSS values is what is not in memory. The difference of the occupied memory exists in the swap space or file system, either because some parts of the occupied memory were paged out, or because some parts of the executable were never loaded.

Dirty
dirty pages (both shared and private). This can be usefull when finding out which parts of the complete memory stack are responsible for the total set of dirty pages in your memory. For shared file-backed mappings, dirty pages can be written back to the underlying file if the kernel feels it has to make some room in RAM or that there are too many dirty pages.

Mode
Each map can have multiple modes set, reading the mode of a map will give you more insight and hints of what the use and role is of the map. The mode representation provided by pmap differs slightly from the actual kernel flags. Using the below list as reference provides a good insight in the meaning of a mode flag.

  • r: if set, the map is readable
  • w: if set, the map is writable
  • x: if set, the map contains executable code
  • s: if set, the map is shared. While the kerel maintains two flags for stating the shared or private mode of a map the pmap representation only shows s for shared while the kernel itself uses s for shared and p for private.
  • R: if set, the map has no swap space reserved (MAP_NORESERVE flag of mmap)

Offset
Offset into the file, this can be used in combination with the address to find the exact location.

Device
Device provides you the major:minor notation of the device used for storing the map.

Mapping
Each map has a mapping (type) associated with it. The following mappings can be observed when looking at pmap output;

  • Anon - non-file backed memory, Anonymous memory: Memory not relating to any named object or file within the file system is reported as [ anon ].
  • Stack – memory allocation reserved for the process stack
  • “file backed” – file backed maps will provide the name of the file that is mapped. You can observe the “(deleted)” statement associated with a file backed page. In effect the term is misleading as the file is not deleted as the term indicates, it is unlinked and the map can be deleted. The file will (in most cases) still be present. Also, you will notice that executable file (file backed maps with a mode x flag) will have multiple entries in the nmap output. Reason for this is that executable files are build up out of different segments who are mapped in different ways.


No comments: