CodePark/ questions / Memory, stack & heap
START PRACTICINGPRACTICE
Backend question · OS, networking & Linux › Memory, stack & heap

What is the Linux OOM killer, and how can you tell that it killed your process?

The OOM killer (Out Of Memory killer) is the part of the Linux kernel that kills a process when memory runs out.

Linux overcommits memory. This means it lets processes allocate more memory than the machine really has. It bets that they will not use all of it. Sometimes RAM and swap are truly full, and the kernel can't free enough cache. Then the OOM killer chooses a process and kills it with SIGKILL. It prefers the processes that use the most memory. You can adjust this choice per process with oom_score_adj.

The signal is SIGKILL, so the process can't clean up or write a log. Signs that the OOM killer killed it:

  • The process died from signal 9. A shell or supervisor reports this as exit code 137 (128 + 9).
  • The kernel log has a line like Out of memory: Killed process 1234 (node). Read it with dmesg or journalctl -k.

In containers, the same thing happens per cgroup when the container goes over its memory limit. A cgroup is the Linux feature that sets resource limits for a group of processes.

How to fix it:

  • Find out why memory grew.
  • Set limits that match real usage.
  • Keep runtime limits, like V8's heap size, below the container limit.
  • Linux overcommits memory, and the OOM killer runs when the kernel can't free enough memory.
  • It picks a victim mostly by memory use, and you can tune this with oom_score_adj.
  • The victim gets SIGKILL, so it can't clean up, and it exits with code 137.
  • The evidence is in the kernel log: dmesg or journalctl -k.
  • In containers, going over the cgroup memory limit triggers the same thing.