CodePark/ questions / Debugging a server in production
START PRACTICINGPRACTICE
Backend question · OS, networking & Linux › Debugging a server in production

df says the root filesystem is full, but du accounts for far less. Where did the space go, and how do you get it back?

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   50G     0 100% /
$ sudo du -shx /
21G     /

The space is usually held by a deleted file that is still open. Restart the process that holds it, and the space comes back.

Why the two tools disagree:

  • df asks the filesystem how many blocks are in use.
  • du walks the directory tree and adds up the files it can find.

So they disagree when something uses space but has no visible path.

How it happens: someone ran rm on a huge log file. But the process writing it still has it open. The kernel keeps the data until the last file descriptor (open handle) closes.

How to fix it:

  • Find these files with sudo lsof +L1. It lists open files with zero links. They are marked (deleted).
  • Restart the process, or make it reopen its log files.
  • Or truncate (empty) the file in place through /proc/<pid>/fd/<n>.
  • In future, truncate active logs instead of deleting them, or use logrotate's copytruncate option.

Other causes to check:

  • Files hidden under a mount point. Something was written to the folder before another disk was mounted on top of it.
  • Inodes. Run df -i. If the disk runs out of inodes (the records that describe each file), you get "No space left on device" even with free blocks.
  • df counts the blocks in use, while du adds up the files it can see.
  • A deleted file that is still open keeps its space until it is closed.
  • Find these files with lsof +L1 and look for (deleted).
  • Restart the process or reopen its logs, or truncate the file through /proc/<pid>/fd.
  • Also check for files hidden under mount points, and check inodes with df -i.