CodePark/ questions / TCP connections in depth
START PRACTICINGPRACTICE
Backend question · OS, networking & Linux › TCP connections in depth

What is the TIME_WAIT state, which side of a connection enters it, and when do thousands of TIME_WAIT sockets become a problem?

TIME_WAIT is a state a socket stays in after its connection closes. The side that closes first enters it. That side sends the first FIN (the "I'm done" message) and is called the active closer. It stays in TIME_WAIT for twice the maximum segment lifetime (2 x MSL). Linux uses a fixed 60 seconds.

TIME_WAIT exists for two reasons:

  • If the final ACK is lost, the other side resends its FIN. The socket must still exist to acknowledge it. Otherwise it would reply with a reset.
  • It lets delayed segments from this connection expire before a new connection reuses the same four-tuple (source IP, source port, destination IP, destination port).

TIME_WAIT is normal. It becomes a problem when a client opens many short connections to the same server. For example, a Node service that creates a new HTTP connection for every request. The client can run out of ephemeral ports (the temporary local ports for outgoing connections) and fail with EADDRNOTAVAIL.

The real fix is connection reuse with keep-alive or a pool. net.ipv4.tcp_tw_reuse helps outgoing connections. The old tcp_tw_recycle setting broke clients behind NAT, and Linux removed it.

  • The active closer enters TIME_WAIT. It is the side that sends the first FIN.
  • TIME_WAIT lasts 2 x MSL, which is 60 seconds on Linux.
  • It lets the final ACK be resent and lets stale segments expire.
  • Many short outgoing connections can use up all the ephemeral ports.
  • Fix it with keep-alive and pooling, not with tcp_tw_recycle.