An API that survives a spike
A public API runs comfortably at 2,000 requests a second. Three seconds in, a product launch sends four times that for three seconds.
The current servers can handle a little under 3,000 a second. Provisioning for the peak would mean paying for four times the machines all month for a few seconds of traffic, and the budget doesn't allow it.
When more work arrives than a server can do, it doesn't do less work: it queues it. Every request waits behind the ones before it, so latency climbs for everyone, requests start timing out, and the queue takes seconds to drain after the spike ends.
The alternative is to turn some requests away at once with HTTP 429 Too Many Requests, so the ones you accept stay fast. A well-behaved client backs off and retries. Rejecting on purpose is allowed here; timing out is not.
Requirements
- Serve 2,000 requests a second normally, and as many as the servers can handle during the spike.
- Requests that are accepted must stay fast, even during the spike.
- Turning requests away with a 429 is acceptable; failing or timing out is not.
- Stay within the budget: no provisioning for four times the normal load.
Traffic
2K requests a second, 80% reads
20K distinct keys, a few of them very hot
A spike to 4× for 3 s from 3 s
Targets
- At least 1.9K successful requests a second
- p99 latency at most 100 ms
- At most 1% of requests fail or time out (turning requests away with a 429 is allowed)
- Costs at most $700 a month