A replicated key-value store
A key-value store behind an API: get(key) and put(key, value), 5,000 requests a second, four in five of them reads.
The data is already split over two shards, each with a primary and a read replica. Replicas copy the primary's writes asynchronously: a write is acknowledged once the primary has it, and the replica catches up a little later. Until it does, a read from the replica returns the old value.
Machines fail. The check runs the traffic twice: once normally, and once with shard 0's primary crashing three seconds in. Writes to that shard have nowhere to go unless something takes the primary's place.
The replicas are in a distant data centre and lag a full second behind. Bring them closer (lower the lag, though no closer than 100 ms: they stay in another building for safety) and make the store survive losing a primary.
Requirements
- get(key) and put(key, value): 5,000 requests a second, 80% reads.
- Survive a primary crashing: keep serving reads and writes.
- Few reads may return an out-of-date value.
- Fast for everyone.
Traffic
5K requests a second, 80% reads
100K distinct keys, some hotter than others
Targets
- At least 5K successful requests a second
- p99 latency at most 50 ms
- At most 1% of requests fail, time out or are rejected
- At most 2% of reads are stale
- Costs at most $2,000 a month
- Shard 0’s primary crashes at 3 s: still at most 2% errors