CodePark/ questions / Debugging a server in production
START PRACTICINGPRACTICE
Backend question · OS, networking & Linux › Debugging a server in production

In top, the CPU line shows us, sy, wa and st. What does each mean, and what would a high value of each suggest?

Each value is a share of CPU time. Each one points to a different kind of problem.

us (user) is time spent running application code. High us means processes are CPU-bound. Find the hot process in the list and profile it. For Node, use node --cpu-prof or --prof.

sy (system) is time the kernel spends working for processes. High sy points to heavy syscall traffic. A syscall is a request from a program to the kernel. Examples: many tiny writes, or many connections opening and closing (connection churn).

wa (iowait) is time the CPU was idle while I/O was still pending. High wa means the CPU is waiting on storage. Check iostat, and find which process is doing the I/O.

st (steal) is time a virtual machine wanted to run, but the hypervisor gave the physical CPU to another VM. The hypervisor is the software that runs the VMs on one physical machine. High st on a cloud VM means a noisy neighbour (another busy VM on the same host) or a throttled instance. No code change will fix this.

Press 1 in top to see each core separately. An average can hide one core stuck at 100%. This is common with single-threaded Node.

  • us is application code time; profile the hot process.
  • sy is kernel time; it means too many syscalls or context switches.
  • wa is time waiting on I/O; check the disks.
  • st is time stolen by the hypervisor; it is a VM-level problem.
  • Press 1 in top to see the usage of each core.