One customer's 50,000 req/min spreads across 20 instances. Only shared state sees the whole stream.

01Symptom

Your public API serves 50,000 paying customers, each allotted 100 requests/minute. There is no rate limiting at all. A bug in one customer's integration starts firing 50,000 requests/minute at /search — 500x its quota — and latency degrades for everyone else. The API runs as 20 stateless instances behind a round-robin load balancer with no sticky sessions, so any instance can get any customer's request. A Redis cluster is already available, used for caching elsewhere.

02Constraints

• 50,000 customers, each limited to 100 req/min
• 20 API instances, no sticky sessions — round-robin, so a customer's stream is spread across all 20
• The limit must hold globally per customer. An instance-local counter lets one customer reach 100 × 20 = 2,000 req/min
• Well under 5ms added to the request path at p99
• Sane behaviour when Redis is slow or briefly unavailable — rate limiting must never be what takes the API down
• Rolling reset: no bursting 100 requests either side of a minute boundary

03Evidence

• →The runaway customer's ~833 req/s arrive spread evenly over all 20 instances — roughly 2,500/min each, about 25x the whole 100/min quota, and each instance sees a slice that looks locally plausible
• →p99 latency for other customers rises in the same window as the flood, which points at a shared downstream resource rather than one customer's code path
• →A read-then-increment limiter passes every single-instance test and still lets the quota be exceeded once two instances overlap — the failure only appears under cross-instance concurrency
• →The limiter's own Redis call is ~0.3-1ms p99 when Redis is healthy, but the same call has a p99 in the seconds when Redis is degraded, so the request path inherits Redis's tail whether it wants to or not
• →At full quota across all 50,000 customers the limiter issues ~83,000 script calls/sec, a large share of what one single-threaded Redis node can execute

→The question

Design the limiter end to end: where the counter lives, what makes the check-and-increment indivisible, what happens when Redis is down, and what changes when limits are per-endpoint.

04Your prediction