-

This is the first post about investigation memory issues. The OOM-Killer can be hard to detect if we don’t look where we should and don’t interpret the data properly.

Where does Out of Memory killer comes from?

The linux kernel allows program to reserve more memory than it actually can offer. This behaviour can be disabled, however this is the default. It comes from the fact programs usually reserve more memory than they will actually use. One scenario can be a JVM reserving 1G of memory right away (Xmx=1g Xms=1g) running a program never actually using more than 512M. If the kernel didn’t allow memory allocation over the physical constraints, you would be able to run less program than you currently are.

What happen when every process request their memory?

The kernel has to sacrifice a process :( When this happen, you will find something like the following log in /var/log/messages or /var/log/kern.log.

May  1 21:13:05 dev-VirtualBox kernel: [ 4563.079551] memory-hog-2048 invoked oom-killer: gfp_mask=0x2084d0, order=0, oom_score_adj=0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079563] memory-hog-2048 cpuset=/ mems_allowed=0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079821] Mem-Info:
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079827] Node 0 DMA per-cpu:
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079834] CPU    0: hi:    0, btch:   1 usd:   0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079839] CPU    1: hi:    0, btch:   1 usd:   0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079845] CPU    2: hi:    0, btch:   1 usd:   0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079850] CPU    3: hi:    0, btch:   1 usd:   0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079855] Node 0 DMA32 per-cpu:
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079861] CPU    0: hi:  186, btch:  31 usd:  26
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079866] CPU    1: hi:  186, btch:  31 usd:  30
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079872] CPU    2: hi:  186, btch:  31 usd:  34
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079877] CPU    3: hi:  186, btch:  31 usd:  30
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079882] Node 0 Normal per-cpu:
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079888] CPU    0: hi:  186, btch:  31 usd: 172
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079893] CPU    1: hi:  186, btch:  31 usd:  59
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079898] CPU    2: hi:  186, btch:  31 usd: 119
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079903] CPU    3: hi:  186, btch:  31 usd:  64
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079913] active_anon:1832420 inactive_anon:146186 isolated_anon:0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079913]  active_file:44 inactive_file:120 isolated_file:0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079913]  unevictable:4 dirty:0 writeback:0 unstable:0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079913]  free:25271 slab_reclaimable:9320 slab_unreclaimable:5765
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079913]  mapped:1454 shmem:2124 pagetables:9331 bounce:0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079913]  free_cma:0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079922] Node 0 DMA free:15900kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079934] lowmem_reserve[]: 0 3488 7966 7966
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079942] Node 0 DMA32 free:47276kB min:29532kB low:36912kB high:44296kB active_anon:3225472kB inactive_anon:256744kB active_file:20kB inactive_file:172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3653568kB managed:3574580kB mlocked:0kB dirty:0kB writeback:0kB mapped:184kB shmem:2220kB slab_reclaimable:14660kB slab_unreclaimable:8120kB kernel_stack:816kB pagetables:15740kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:336 all_unreclaimable? yes
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079955] lowmem_reserve[]: 0 0 4478 4478
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079962] Node 0 Normal free:37908kB min:37920kB low:47400kB high:56880kB active_anon:4104208kB inactive_anon:328000kB active_file:156kB inactive_file:308kB unevictable:16kB isolated(anon):0kB isolated(file):0kB present:4718592kB managed:4586280kB mlocked:16kB dirty:0kB writeback:0kB mapped:5632kB shmem:6276kB slab_reclaimable:22620kB slab_unreclaimable:14932kB kernel_stack:1832kB pagetables:21584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1827 all_unreclaimable? yes
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079973] lowmem_reserve[]: 0 0 0 0
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.079981] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080005] Node 0 DMA32: 98*4kB (E) 78*8kB (E) 264*16kB (UEM) 144*32kB (UEM) 97*64kB (UE) 41*128kB (UE) 16*256kB (UEM) 14*512kB (EM) 14*1024kB (UEM) 0*2048kB 0*4096kB = 46904kB
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080028] Node 0 Normal: 127*4kB (EM) 97*8kB (EM) 299*16kB (E) 149*32kB (UE) 103*64kB (UEM) 48*128kB (E) 8*256kB (EM) 6*512kB (UEM) 9*1024kB (UEM) 0*2048kB 0*4096kB = 37908kB
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080052] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080057] 2321 total pagecache pages
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080063] 0 pages in swap cache
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080069] Swap cache stats: add 1498255, delete 1498255, find 623850/695336
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080073] Free swap  = 0kB
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080078] Total swap = 0kB
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080083] 2097038 pages RAM
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080088] 0 pages HighMem/MovableOnly
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080093] 33078 pages reserved
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080098] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080112] [  283]     0   283     4869       78      14        0             0 upstart-udev-br
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080120] [  289]     0   289    12954      234      27        0         -1000 systemd-udevd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080128] [  467]     0   467     3815       68      12        0             0 upstart-socket-
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080135] [  588]   102   588    10010      297      23        0             0 dbus-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080142] [  610]     0   610    82559      278      65        0             0 ModemManager
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080149] [  656]     0   656     3850       72      13        0             0 upstart-file-br
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080156] [  670]   101   670    63961      224      28        0             0 rsyslogd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080163] [  691]     0   691     4823       65      14        0             0 bluetoothd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080170] [  700]     0   700    10863       93      26        0             0 systemd-logind
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080178] [  725]   111   725     8089       76      21        0             0 avahi-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080185] [  735]   111   735     8056       62      20        0             0 avahi-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080192] [  760]     0   760     5005       41      13        0             0 getty
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080198] [  764]     0   764     5005       41      13        0             0 getty
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080226] [  771]     0   771     5005       42      13        0             0 getty
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080233] [  772]     0   772     5005       39      12        0             0 getty
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080240] [  776]     0   776     5005       41      13        0             0 getty
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080247] [  850]     0   850    18838      230      41        0             0 cups-browsed
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080254] [  859]     0   859     4797       63      14        0             0 irqbalance
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080261] [  867]     0   867     1092       45       9        0             0 acpid
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080268] [  873]     0   873     5914       62      17        0             0 cron
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080275] [  932]     0   932    89147      411      72        0             0 NetworkManager
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080282] [  977]   106   977     9286       83      22        0             0 kerneloops
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080289] [  988]     0   988    53793      686      42        0             0 gdm
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080296] [  999]     0   999    73977      564      45        0             0 polkitd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080303] [ 1018]     0  1018    74655      225      48        0             0 gdm-simple-slav
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080310] [ 1051]     0  1051     2558      573       8        0             0 dhclient
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080318] [ 1059]   109  1059    91360      372      78        0             0 whoopsie
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080325] [ 1072] 65534  1072     8807       64      22        0             0 dnsmasq
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080332] [ 1133]     0  1133    96904    11205     174        0             0 Xorg
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080339] [ 1182]     0  1182    75554      778      51        0             0 accounts-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080346] [ 1213]     0  1213    56259      102      20        0             0 VBoxService
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080353] [ 1405]     0  1405    73022     2875     109        0             0 apache2
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080360] [ 1419]     0  1419   101577      819      68        0             0 gdm-session-wor
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080367] [ 1453]  1000  1453     9984      179      24        0             0 init
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080374] [ 1554]     0  1554     5005       42      13        0             0 getty
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080381] [ 1592]  1000  1592     6110       65      17        0             0 dbus-launch
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080387] [ 1600]  1000  1600     9779       81      23        0             0 dbus-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080394] [ 1617]  1000  1617    28411      121      27       15             0 VBoxClient
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080401] [ 1624]  1000  1624    28938      139      26        0             0 VBoxClient
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080408] [ 1628]  1000  1628    11509      110      23        0             0 VBoxClient
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080415] [ 1633]  1000  1633    28088      137      24        0             0 VBoxClient
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080422] [ 1649]  1000  1649     2654       79       8        0             0 ssh-agent
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080429] [ 1655]  1000  1655    10004      307      23        0             0 dbus-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080436] [ 1662]  1000  1662     5576       52      15        0             0 upstart-event-b
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080443] [ 1680]  1000  1680     5630       90      12        0             0 upstart-dbus-br
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080450] [ 1681]  1000  1681     5578       52      13        0             0 upstart-dbus-br
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080457] [ 1685]  1000  1685     7697       75      16        0             0 upstart-file-br
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080464] [ 1692]  1000  1692    94282      552      52        0             0 ibus-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080471] [ 1706]  1000  1706   212350     2254     177        0             0 gnome-settings-
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080478] [ 1709]  1000  1709    91899      707      47        0             0 at-spi-bus-laun
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080485] [ 1710]  1000  1710   198563     1068     122        0             0 gnome-session
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080492] [ 1715]  1000  1715     9812      116      24        0             0 dbus-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080499] [ 1728]  1000  1728    49161      162      32        0             0 gvfsd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080506] [ 1733]  1000  1733    73723      183      43        0             0 gvfsd-fuse
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080513] [ 1735]  1000  1735    73943      205      46        0             0 ibus-dconf
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080520] [ 1736]  1000  1736   133401     1187     107        0             0 ibus-ui-gtk3
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080527] [ 1738]  1000  1738    79529      471      88        0             0 ibus-x11
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080534] [ 1755]  1000  1755    31228      154      30        0             0 at-spi2-registr
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080541] [ 1776]     0  1776    59837      230      46        0             0 upowerd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080548] [ 1786]  1000  1786    99095      267      52        0             0 gnome-keyring-d
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080555] [ 1797]  1000  1797    54980      268      43        0             0 ibus-engine-sim
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080562] [ 1813]  1000  1813    93447      576      90        0             0 pulseaudio
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080569] [ 1815]   107  1815    42229       58      19        0             0 rtkit-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080576] [ 1838]  1000  1838   535003   100313     522        0             0 gnome-shell
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080583] [ 1843]   113  1843    77760      386      55        0             0 colord
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080590] [ 1845]  1000  1845    44575      147      22        0             0 dconf-service
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080596] [ 1850]  1000  1850    97788      340      86        0             0 gsd-printer
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080603] [ 1967]  1000  1967   129444      635      82        0             0 gnome-shell-cal
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080610] [ 1973]  1000  1973   276760     1285     154        0             0 evolution-sourc
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080617] [ 1975]  1000  1975   101098      375      65        0             0 mission-control
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080624] [ 1978]  1000  1978    77893      958      54        0             0 gvfs-udisks2-vo
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080631] [ 1983]  1000  1983   160981     1618     143        0             0 goa-daemon
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080638] [ 1987]     0  1987   112920      960      56        0             0 udisksd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080645] [ 1998]  1000  1998    53112      190      38        0             0 gvfs-gphoto2-vo
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080652] [ 2003]  1000  2003    46914      146      28        0             0 gvfs-goa-volume
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080659] [ 2023]  1000  2023    50071      667      35        0             0 gvfs-mtp-volume
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080666] [ 2027]  1000  2027    71490      181      40        0             0 gvfs-afc-volume
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080673] [ 2044]  1000  2044   120574    17714      93        0             0 tracker-store
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080679] [ 2046]  1000  2046   266727    17213     107        0             0 tracker-miner-f
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080686] [ 2095]  1000  2095   206485     9093     162        0             0 evolution-calen
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080693] [ 2208]  1000  2208    67595      170      34        0             0 gvfsd-burn
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080700] [ 2249]  1000  2249   140235     1498      97        0             0 zeitgeist-datah
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080726] [ 2254]  1000  2254    90908      805      47        0             0 zeitgeist-daemo
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080733] [ 2262]  1000  2262    64096      593      56        0             0 zeitgeist-fts
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080740] [ 2280]  1000  2280     2854       24      11        0             0 cat
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080747] [ 2282]  1000  2282   191359     2475     204        0             0 evolution-alarm
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080754] [ 2320]  1000  2320   156245     1641     121        0             0 update-notifier
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080761] [ 2385]  1000  2385   181535     3033     147        0             0 gnome-terminal
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080768] [ 2391]  1000  2391    14278      133      32        0             0 gconfd-2
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080775] [ 2396]  1000  2396     3706       40      13        0             0 gnome-pty-helpe
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080782] [ 2397]  1000  2397     7761     1573      20        0             0 bash
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080788] [ 2494]  1000  2494    96298      763      56        0             0 deja-dup-monito
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080796] [ 3166]  1000  3166     7726     1534      20        0             0 bash
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080837] [ 3517]     0  3517    19213      270      41        0             0 cupsd
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080844] [ 3693]  1000  3693   179795    13040     185        0             0 update-manager
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080851] [ 3912]  1000  3912     7706      374      20        0             0 htop
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080858] [ 4490]  1000  4490    31121      139      31        0             0 gvfsd-metadata
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080865] [ 6399]  1000  6399     7772     1585      20        0             0 bash
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080872] [ 6777]  1000  6777   525339   424087     836        0             0 memory-hog-2048
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080879] [ 6778]  1000  6778   525339   467427     921        0             0 memory-hog-2048
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080886] [ 6779]  1000  6779   525339   392670     775        0             0 memory-hog-2048
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080893] [ 6780]  1000  6780   525339   479845     945        0             0 memory-hog-2048
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080899] Out of memory: Kill process 6780 (memory-hog-2048) score 235 or sacrifice child
May  1 21:13:06 dev-VirtualBox kernel: [ 4563.080908] Killed process 6780 (memory-hog-2048) total-vm:2101356kB, anon-rss:1919364kB, file-rss:16kB

The logs informs us the state of the box when the kernel sacrificed a child process. We also have the killed process pid and name at the end.

How to read this log

There a lot of information in this log and I will focus on the part that are the most interesting. There is a table with the list of running pid when the memory starvation occured. For each PID we have the amount of memory being physically used (rss column) and the oom score adjustment oom_score_adj. The rss column is in number of page and each page is 4kb. Hence if you multiply the figures in this column by 4 you will have the memory used by the process in kb. The PID 6780 was using 479845*4/1024 = 1874 Mb of physical memory. The kernel uses an algorithm to determine which process it will kill. However, you can influence the algorithm.

How to make my proces unkillable by oom killer

The kernel will compute a score for each process taking into account several low level information. The score will be between 0 and 1000 and the highest score is the process that will be killed. However this score is adjusted adding the oom_score_adj. Hence, if you set oom_score_adj to -1000 your process will never be chosen by the kernel to be killed. In order to do that, you need to update the file /proc/$PID/oom_score_adj.

comments powered by Disqus