Design rate limiting for a public API that runs on 20 instances behind a load balancer. Where does the limiter live, where is its state, and what happens when the state store fails?

Put the main limiter at the edge. This means the API gateway, or a middleware that every instance runs before any business work. Then a rejected request costs almost nothing.

What to key the limit by:

• the API key or the user
• the IP address for anonymous traffic, as a fallback
• extra per-endpoint rules for expensive routes

Services can add their own limits too.

With 20 instances, counters in each instance's memory are wrong. The load balancer spreads one client's requests across many instances, so no single instance sees the full count. Keep the counters in a shared low-latency store such as Redis, and update them atomically (in one step that cannot be interrupted). Usually you use a token bucket or a sliding window counter. A cheaper approximation is to divide the limit by the number of instances and count locally. Use it when precision matters less.

When you reject a request, return 429 Too Many Requests. Add a Retry-After header and headers that show the remaining quota.

If the store is down, you must choose:

• fail-open: allow the traffic, maybe with a local fallback limit
• fail-closed: reject the traffic

Most APIs fail open, so that the limiter does not cause an outage itself.