What is the difference between TCP keepalive and HTTP keep-alive? And how can they cause sporadic 502s from a load balancer in front of Node?

They work at different layers. HTTP keep-alive reuses one connection for many requests. TCP keepalive checks that an idle connection is still alive.

HTTP keep-alive is also called persistent connections. The client sends many HTTP requests over one TCP connection, instead of opening a new connection for each request. This saves the cost of a new handshake each time. It is the default in HTTP/1.1. Servers close idle connections after a timeout. In Node, this timeout is server.keepAliveTimeout, and its default is 5 seconds.

TCP keepalive is a feature of the transport layer (TCP itself, not HTTP). When a connection has been idle for some time, the kernel sends small probes. A probe is a tiny packet that asks "are you still there?". If the other side never answers, the kernel drops the connection. On Linux, the default idle time before the first probe is 2 hours. So applications often make it shorter, for example with socket.setKeepAlive(true, 30000).

Why the 502s happen: sometimes the load balancer's idle timeout is longer than Node's keepAliveTimeout. Then Node can close an idle connection at the same moment the load balancer sends a new request on it. The request fails, and the load balancer returns a 502.

Fix: set Node's keepAliveTimeout higher than the load balancer's idle timeout. Set headersTimeout higher too.