CodePark/ questions / Debugging a server in production
START PRACTICINGPRACTICE
Backend question · OS, networking & Linux › Debugging a server in production

uptime shows load average: 12.3, 8.1, 4.0 on a 4-core server. What does that mean, and is the CPU necessarily the problem?

The machine has about three times more work than its 4 cores can handle right now, and the load is rising. But the CPU is not necessarily the cause.

The three numbers are averages of system load over the last 1, 5 and 15 minutes. They are exponentially damped averages: recent moments count more than older ones.

What load counts:

  • Tasks that are running on a CPU.
  • Tasks that are waiting for a CPU.
  • On Linux only: tasks in uninterruptible sleep (state D). These usually wait on disk or on network file system I/O.

Compare load to the number of cores. On 4 cores, a load of 4 means the CPUs are fully busy with no queue. A load of 12.3 means about three times more demand than capacity right now. The 15-minute value is 4.0 and the 1-minute value is 12.3, so load is climbing.

On Linux, load is not only CPU. Check top to find the real cause:

  • If us and sy are high, the machine is CPU-bound. Find the hot process and profile it.
  • If the CPU is mostly idle but wa is high, or many processes are in state D, the bottleneck is storage or a network mount. vmstat or iostat will show it.
  • The three numbers are 1, 5 and 15-minute load averages.
  • On Linux, load counts runnable tasks plus tasks in uninterruptible sleep (state D).
  • Compare the load against the number of cores.
  • Load is rising when the 1-minute value is higher than the 15-minute value.
  • High load with an idle CPU points to an I/O problem.