System design: the core trade-offs

A service that answers a thousand requests a second on two servers and one database is easy. Ten times the traffic is not, and every way of handling it trades something away. Add servers, and the load moves to the next part in line. Add a cache, and some reads come back out of date. Add database replicas, and they lag behind. Add a queue, and writes are acknowledged before they have happened. System design is choosing which of those costs your system can live with.

This lesson builds those choices on a simulator. Every lab below runs real traffic through a design, request by request: the cache really evicts, the queue really fills up, and a crashed machine really drops what it was doing. Change one thing at a time, run it, and watch what moves.

bottleneck

The part of a system that is busiest, close to 100% of its capacity. Requests queue in front of it, so it sets the latency and the most the whole system can do.

utilisation

How much of the time a part is busy. Near 100%, waiting times explode, as the queueing lesson showed.

cache-aside

The application reads from a cache first and, on a miss, from the database, then puts what it read into the cache. Writes must update or delete the cached copy, or it goes stale.

replication lag

How far behind the primary a read replica is. A read from a replica can miss writes made in the last moments.

shard

One partition of the data. Each shard holds a slice of the keys on its own machines, so writes spread across shards.

backlog

Work waiting in a queue. If it keeps growing, the consumers can't keep up with what arrives.

failover

Promoting a replica to take over when the primary dies, so writes can carry on.

Each lab is a canvas: select a component to change it, add new ones from the row of buttons, and press Run. The labs use the same seed, so the numbers in the captions are the ones you will see when you change exactly what they say.

Two app servers get 3,000 requests a second. Each request does a little work on an app server, then holds that server's worker while it waits for the database. Run it first: the app servers are 100% busy, and most requests time out.

Adding app servers helps until they stop being the busiest part. From 6 instances on, the database is the bottleneck, and more app servers just add more requests waiting for it. The findings under the results name the bottleneck after every run: read them before changing anything.

The same 3,000 requests a second, now over 10,000 keys with a few hot ones, and 4 app servers. Without a cache, this is the design that ran at a p99 of 846 ms above. Now reads go to a cache first, holding up to 2,000 keys. Writes still go straight to the database.

Before you run it, decide: how much of the database's work will the cache take away, and will every read still return the latest value?

The cache took most of the reads off the database, which is the point of a cache: a hot key read a hundred times costs one database query. But writes that skip the cache leave the old copy in it, and every read of a popular key that has since changed returns the old value. Sending writes through the cache fixes that. Invalidating deletes the copy, so the next read misses and refetches it; write-through updates the copy in place and keeps the hit.

Read replicas: load against freshness

3,500 requests a second, 95% of them reads, on one database. Read replicas copy the primary's writes and serve reads, so the primary only takes writes. They copy asynchronously, a little behind: here 1 second behind.

Replicas scale reads, not writes, and they trade freshness for it. A replica that is a second behind answers reads with a second-old view of the data, so a user who writes and then reads from a replica may not see their own write. The fix is not more replicas but choosing where each read goes: read your own recent writes from the primary, or wait for the replica to catch up. Synchronous replication removes the lag, at the cost of making every write wait for the replicas.

Sharding: spreading the writes

2,500 requests a second, 80% of them writes. Replicas don't help, because every write goes to the primary. Sharding splits the keys over several primaries, each owning a slice found by consistent hashing.

Sharding is the way to scale writes, and it is hard to undo: the shard key decides which data lives together. A query that needs data from every shard has to ask every shard, and a hot key lives on one shard however many there are. With skewed traffic, the busiest shard, not the average, sets the limit.

Queues: answering before the work is done

1,000 requests a second, half of them writes, and each write takes the database 40 ms. The app server waits for every write, so writes have a slow tail.

A queue lets the system answer at once and do the work later, smoothing bursts and decoupling the caller from slow work. But the work still has to be done. If the consumers handle less than the average arrival rate, the backlog grows without limit, and for every write in the queue, the rest of the system can't yet see it.

Failures: what a crash costs

Machines die. This design has 3 app servers and a database with one replica, at 1,000 requests a second. In the run panel, you can crash a machine part-way through.

Redundancy only helps if something switches to it. Stateless app servers are easy: the load balancer stops sending to the dead one. A database primary holds state, so promoting a replica takes time, and any writes it hadn't yet copied may be lost.

There is a harder version of this choice. If the network splits the primary from some replicas, the replicas can either keep answering with what they have, which might be stale (choosing availability), or refuse until they hear from the primary (choosing consistency). That is the CAP theorem: during a network partition you get one or the other. Its everyday companion, PACELC, adds that even without a partition, replicas trade latency against consistency, which is exactly the replica lag you tuned above. Raft, from the consensus lesson, chooses consistency: a minority side refuses writes.

One more piece: a rate limiter at the front turns away requests over a set rate at once, with an error the client can retry. During a spike, that protects everything behind it from collapsing, so the rest of the traffic still gets fast answers. Try one in the sandbox.

• A system is as fast as its busiest part. Find the bottleneck, scale that, then look for the next one.
• A cache takes load off the database, but its copies go stale unless writes invalidate or update them.
• Read replicas scale reads and lag behind; sharding scales writes and makes hot keys and cross-shard queries harder.
• A queue replies before the work is done. The consumers must keep up with the average rate, or the backlog grows.
• Redundancy needs a switch-over: load balancers route around dead app servers, failover promotes a replica. In a partition, choose consistency or availability.

Put it together in the case studies, each with traffic, targets and a budget, and practise the numbers that come first in the estimation drills.