Then I sat through a mock interview where every question was a small, ordinary-looking scenario, and the interviewer kept asking "okay, and then what?" until my answer fell apart. I did fine on some. On one of them I was confidently, completely wrong.

So this post is two things. It's my notes on the vocabulary (mostly from the system-design-primer and karanpratapsingh/system-design, both free and both excellent). And it's the questions I was asked, with what I said, what I should have said, and the lesson I took away. If you're preparing for interviews, try answering each question yourself before you read mine.

Scalable doesn't mean "handles a lot of traffic"

That was my working definition, and it's too loose to be useful. The Primer has a much sharper one: a system is scalable if adding resources gives you a proportional increase in performance.

Add a second server and handle roughly twice the load? Scalable. Add a second server and get 10% more? Then something else is the bottleneck, and in my experience it's nearly always the database.

Image: karanpratapsingh/system-design

Slow is not always a scaling problem

This one sounds obvious when you write it down, but I've watched real teams get it wrong.

A performance problem means the system is slow even for one user. A scalability problem means it's quick for one user and slow when ten thousand show up at once. They need opposite fixes. More servers won't help a slow query, and tuning a query won't help when you're simply out of capacity.

The question: Your dashboard takes 4 seconds to load, even at 2 a.m. when nobody's using it. Your manager says "let's add more servers." What do you say?

I got this one. Nobody is using it, so load isn't the problem. It's a performance issue: profile the request, run EXPLAIN on the queries, and find out where those 4 seconds actually go.

If you put a cache in front of a slow query, you haven't fixed anything, you've just hidden it. Find the cause first.

Latency and throughput pull against each other

Latency is how long one request takes. Throughput is how many requests you get through per second. The analogy that made it click for me: latency is how long one car takes to cross the city, throughput is how many cars cross per hour. Adding lanes raises throughput, but no single car arrives sooner.

The Primer's rule of thumb is to aim for the highest throughput you can get with acceptable latency. I nodded along to that rule. Then I got this question.

The question: Your logging service writes every event to disk the moment it arrives. An engineer changes it to write in batches of 1,000, and throughput jumps 10x. The next day the product team complains that events show up on their dashboard several seconds late. Is it a bug?

This is the one I got wrong. I said the disk was probably too slow for big batches, and suggested checking the IOPS and the indexes.

The disk was fine. The interviewer asked me to think about the first event that lands in a new batch. It just sits there, waiting for 999 more to arrive. At 500 events a second, that's a two-second wait. At night, at 10 events a second, it's a hundred seconds. Nothing is broken. That is exactly what batching does: you buy throughput and pay for it in latency.

The usual fix is to flush when the batch is full or when a time limit passes, whichever comes first. Say, 1,000 events or 200 ms. Kafka producers work this way. The time limit isn't a technical decision either. You ask the product team what delay they can live with, and that number becomes your limit.

What I took away: when something gets slower right after you "sped it up", stop looking at where the request does work and start looking at where it waits.

Scaling up vs scaling out

Scaling up (vertical) means a bigger machine: more CPU, more RAM, faster disks. It's simple, usually needs no code changes, and your data stays in one consistent place. The downsides: there's a ceiling, the price climbs steeply as you approach it, upgrades usually mean downtime, and when that one machine dies, everything dies with it.

Scaling out (horizontal) means more machines behind a load balancer. You get redundancy, you can add capacity at peak and remove it at night, and you can upgrade one box at a time. You pay for that in complexity: keeping servers identical, data that can drift out of sync, and a lot more connections hitting whatever sits behind them.

Most resources treat horizontal scaling as the obviously correct answer. That habit cost me on this question.

The question: A startup runs everything on one big database. CPU is at 80%, and traffic will double in three months. The CTO says, "Just buy the biggest machine AWS sells." Do you agree?

My first answer was "no, scale horizontally, it's cheaper and more available." The interviewer asked, reasonably, how exactly you scale a database horizontally. My second answer was better: read replicas for reads, writes to the primary, and a conversation about sync versus async replication. When asked what happens if the load is mostly writes, I said that's when you'd look at sharding.

What I never did was answer the actual question. Is the CTO wrong?

Not really. Check first whether it's a performance problem, because a missing index can easily eat half your CPU. If it isn't, a bigger machine is a perfectly sensible bridge. It's one click, needs no code changes, and buys you months to plan the harder work. Replicas only help with reads (every replica still has to apply every write), and sharding is powerful but expensive, so it comes last. The usual order is: optimise the queries, scale up, add caching and read replicas, and shard only when you have to.

What I took away: don't dismiss the simple option just because it isn't the impressive one. "Scale up now to buy time, scale out properly later" is usually what a senior engineer says.

Stateless servers, and the question I couldn't stop thinking about

Horizontal scaling only works if any server can handle any request. If server A keeps your session in its own memory and your next request lands on server B, you're suddenly logged out. So app servers have to be stateless: sessions go into a shared store like Redis or a database, and uploaded files go to shared storage, not a local disk.

The question: You run three app servers behind a load balancer. Users keep getting logged out at random, and their carts sometimes show up empty. Nothing has crashed. What's going on?

I spotted it straight away: the servers are stateful. But I stopped at "make them stateless", and that's the goal, not the answer. The interviewer wanted the how. Move sessions and carts into Redis or the database. Or, as a stopgap, turn on sticky sessions at the load balancer, accepting that users lose their state if their server dies. Or put the session in a signed token.

While learning this, a doubt kept nagging me. If every session now lives in one Redis, haven't we just moved the problem? If Redis restarts, it's in-memory, so everyone gets logged out. If it crashes, nobody can stay logged in at all.

Yes. We've moved it. The state didn't go away, it just has one home now. The fixes stack up from cheap to expensive:

• Turn on Redis persistence, so a restart reloads sessions from disk instead of starting empty.

• Run a replica, with something like Redis Sentinel to promote it if the primary dies.

• Shard across several Redis nodes if one can't keep up.

• Or skip the shared store and use signed tokens (JWTs), accepting that you can't easily revoke one before it expires.

It also helps to ask what losing a session actually costs you. It's an annoying re-login, not a lost payment. You probably need fast recovery, not perfect durability.

Since then I ask myself one question whenever state moves into a shared component: what happens when this dies?

The nines, and the arithmetic behind "remove single points of failure"

Availability is just uptime divided by total time, quoted in "nines". The two worth memorising:

Each extra nine cuts the allowed downtime by ten, and costs a lot more than the last one.

The more useful idea is how availability combines. If a request needs two components in sequence, you multiply their availabilities, so the total is worse than either one. If two components sit in parallel and either can serve the request, the system is only down when both are down at once, so the total is far better.

In sequence:  99.9% × 99.9%               = 99.8%
In parallel:  1 − (0.001 × 0.001)         = 99.9999%

That's the maths behind "remove single points of failure". Every hop you add to the request path drags availability down, and every redundant copy pushes it back up. (One catch: the parallel formula assumes the copies fail independently. Two replicas in the same data centre can easily go down together.)

The question: Your request path is load balancer (99.99%) → app server (99.9%) → database (99.9%), each a single instance. What's the overall availability? If you could make only one change, what would it be?

The arithmetic was fine: about 99.79%. Then I said I'd double the app server and the database, which isn't one change. When pushed to pick one, I chose the app server, but for the wrong reason.

Here's the thing I missed. On paper it doesn't matter which you double: both are 99.9%, so either takes you to about 99.89%. What matters is what each change costs. The app server is stateless, so you clone it and you're done. The database holds data. A second copy immediately raises questions: how writes reach both copies, what happens when the replica lags, who decides which one is primary, and what happens if both think they are.

I also suggested that putting Redis in front of the database would stop it being a single point of failure. The interviewer asked what happens when someone places an order while the database is down. Obviously, it fails. A cache helps reads that hit it. Writes and cache misses still need the database.

What I took away: before saying "just add another one", ask whether the thing holds data. If it does, you're signing up for replication, consistency and failover, and you should say so out loud.

High availability is not the same as fault tolerance

Two terms I used interchangeably until this week:

• High availability: minimal interruption. When something fails, there may be a brief blip while a standby takes over.

• Fault tolerance: zero interruption, because everything is fully redundant. It costs significantly more.

There's a related point: reliability and availability aren't the same either. A reliable system is available, but you can build a highly available system out of unreliable parts, as long as you have enough redundancy.

The question: Option A is a primary with a standby: failover takes 30 seconds and costs X. Option B is fully redundant with zero interruption, at 3X. Which would you pick for a card-payment gateway, and which for an internal analytics dashboard?

Payments get B, because every second of downtime is a failed charge and a customer who might not come back. One thing I've learned to add: even then, retries need to be idempotent, so a request caught mid-failover never charges someone twice.

For the dashboard, I realised you don't even need A. A single server is fine, as long as the data is protected. Buffer incoming events in a queue, back up the database, and accept that managers occasionally wait a few minutes. Keeping the service up and keeping the data safe turn out to be separate problems with separate price tags.

I did call option B "high availability" in my first answer, though. Small slip, but interviewers notice.

What I'm taking into the next topic

Three habits, all learned by getting something wrong:

1. Before I say "add another X", I ask whether X holds data.

2. I don't dismiss the boring option. Sometimes scaling up really is the right call for now.

3. When something gets slower after a speed-up, I look for where the request waits.

Next up is DNS, the first hop of every request and a surprisingly common source of interview follow-ups. If you have a favourite scenario question on any of this, drop it in the comments. I'd love to be grilled on it.