It is almost never your code.
A page feels slow and everyone blames the application. Trace an actual request and the application is usually a rounding error next to an extra handshake, a cache that missed, or eight thousand kilometres of ocean. Flip the switches and watch where the milliseconds really go.
Illustrative figures for a normal web application. Absolute numbers vary; the ratios between them are remarkably stable across almost every system.
Under every abstraction there is a rack in a room
"The cloud" is a rental agreement, not a technology. What you are renting is a slice of a machine bolted into a steel frame in a building with two of everything.
The rack, measured in U
A standard rack is 42U tall, where one U is 44.45 mm. Everything — servers, switches, storage — is built to that height so it can be swapped by anyone, from any vendor, in the dark, at three in the morning.
Two feeds, always
Every server has two power supplies fed from two independent strips on separate circuits, backed by separate UPS batteries and separate generators. Losing one side should cause nothing at all — and testing that is why generators are run monthly.
Top of rack, then spine
Each rack has its own switches at the top; those uplink to a spine layer that connects every rack to every other. Machines in the same rack talk in microseconds, across the building in under a millisecond.
Zone, region, continent
An availability zone is one or more buildings with independent power and cooling. A region is a cluster of zones a few milliseconds apart. Across regions you are paying for the speed of light, and no engineering removes that.
Cold aisle, hot aisle
Racks face each other so cold air is drawn in from one side and exhausted into a shared hot corridor. Every watt a server consumes becomes a watt of heat someone has to remove, which is why the cooling bill tracks the compute bill.
Why regions are a design decision
Putting the database in one region and the application in another is the single most common self-inflicted performance disaster. It looks fine on a diagram, costs 150 ms per round trip, and there is no configuration flag that fixes it.
Eight layers, and what breaks when each one goes
Redundancy is not the same as availability
Two of something only helps if the failure is detected, the switchover works, and the spare was not already broken. Most outages happen in that gap rather than in the component itself.
One more than you need
Size the system for peak load, then add one spare unit. Simple, and it fails the moment two things break in the same hour — which is why critical systems use N+2 or 2N.
Both running, all the time
Better than a standby, because a spare that has never served traffic is a spare nobody has proved works. If both sides carry load continuously, a failure just means the other side gets busier.
Why clusters come in odd numbers
Three, five, seven. A cluster needs a majority to agree it is still the real cluster. With an even number, a network split can produce two halves that each believe they are in charge — and both write.
The worst outcome is not downtime
Two halves of a database both accepting writes is far more expensive than being down, because the damage is silent, spreads into backups, and has to be untangled record by record.
Things nobody duplicated
The domain registrar. The TLS certificate that expires on a Sunday. The one engineer with the production credentials. The CI pipeline. All of these have taken down companies whose servers were perfectly redundant.
An untested failover is a hypothesis
Mature teams fail things over on purpose, during working hours, on a schedule. The alternative is discovering your assumptions during a real incident with customers watching.
State is what makes infrastructure difficult
Web servers are easy: they hold nothing, so you can add ten more and delete them afterwards. Everything painful in infrastructure comes from the components that remember things.
Interchangeable by design
An application server that stores nothing locally can be created, destroyed, restarted and replaced with no consequence. This is the whole reason autoscaling works — and it stops working the moment someone writes a file to local disk.
Databases cannot simply be cloned
Add a second database and you now have a consistency problem: which copy is right, what happens to a write that arrived at both, and how does a reader know it is not reading the past.
The usual compromise
One machine accepts writes, several serve reads, and replicas lag behind by milliseconds to seconds. Fine for a product catalogue, disastrous for a bank balance the user just changed.
The cheapest performance win, and the loudest bug
A cache turns a 14 ms query into a 2 ms lookup. It also means there are now two answers to every question, and every hard cache bug is the same bug: nobody agreed when the copy becomes wrong.
Move the work out of the request
Anything the user does not need an answer to — emails, thumbnails, reports — goes on a queue and happens later. This is the single largest latency improvement available to most applications.
Push state to the edges
Keep as much of the system stateless as possible and concentrate the memory in as few, well-run places as you can. Every additional component that remembers something is another thing to back up, replicate, patch and eventually restore under pressure.
A backup is a restore you have not tested yet
Everyone has backups. Far fewer have ever restored one under time pressure, with the person who set it up on holiday, into an environment that no longer matches.
Three copies, two media, one off-site
The oldest rule in the field and still correct. The off-site copy is what survives fire, flood and a mistake applied to your whole primary environment at once.
How much data may be lost
Recovery point objective: the distance back to the last good copy. Hourly snapshots mean accepting the loss of up to an hour of work — a business decision, not a technical one.
How long you may be down
Recovery time objective. Restoring ten terabytes over a network takes hours no matter how good the plan is, and that number should be measured once rather than estimated forever.
Immutable copies
Attackers now delete backups first, and often sit quietly for weeks so the corruption is already inside the older copies. The answer is write-once storage that even an administrator cannot erase, plus enough retention to reach back past the intrusion.
Mirrors copy your mistakes
A replicated database faithfully reproduces the accidental delete on every copy within a second. Replication protects against hardware failure; only backups protect against people and software.
Restore drills
Pick a random backup each quarter and restore it into a scratch environment with a stopwatch running. The number that comes out is your real RTO, and it is usually a surprise.
Observability, and the people carrying the pager
Logs, metrics, traces
Logs say what happened, metrics say how much and how often, traces follow one request across every service it touched. You need all three: metrics tell you something is wrong, traces tell you where, logs tell you why.
Latency, traffic, errors, saturation
If you can only watch four numbers per service, watch these. Nearly every incident announces itself in one of them before a customer notices.
Averages hide the problem
An average response time of 200 ms can mean everyone got 200 ms, or that 95% got 50 ms and 5% got four seconds. Alert on the 95th and 99th percentile, because that tail is a real group of real users.
The most dangerous failure mode
An alert that fires often and means nothing trains people to ignore the alert that matters. Every page should be actionable, urgent and rare — otherwise delete it.
Someone is awake for this
A rota, a runbook per alert, a defined severity scale and an escalation path. The runbook exists because at 3am nobody reasons well from first principles.
Blameless postmortems
Write down the timeline, the contributing causes and the fixes, without naming a culprit. Teams that hunt for someone to blame get quieter incidents, later detection and worse outages — every study of this finds the same thing.
Two tables worth memorising
The first explains almost every performance decision ever made. The second explains almost every contract argument.
| Operation | Roughly | Relative to the first row |
|---|---|---|
| CPU L1 cache read | 1 ns | 1× |
| Main memory read | 100 ns | 100× |
| SSD random read | 100 µs | 100,000× |
| Same-rack network round trip | 0.2 ms | 200,000× |
| Same-region round trip | 1 ms | 1,000,000× |
| Spinning disk seek | 5 ms | 5,000,000× |
| Europe to North America | 80 ms | 80,000,000× |
| Europe to Australia | 160 ms | 160,000,000× |
| Availability | Downtime per year | Per month | What it takes |
|---|---|---|---|
| 99% | 3d 15h | 7h 18m | One server and someone who notices in the morning. |
| 99.9% — three nines | 8h 45m | 43m | Redundant components, monitoring, an on-call rota. |
| 99.95% | 4h 22m | 21m | Automated failover, no maintenance windows that drop traffic. |
| 99.99% — four nines | 52m | 4m 21s | Multi-zone, tested automatically, deploys that cannot take the system down. |
| 99.999% — five nines | 5m 15s | 26s | Multi-region active-active, and an organisation built around it. Extremely expensive. |
Each additional nine costs roughly an order of magnitude more than the last, and most of that money is spent on process rather than hardware. Choosing three nines deliberately is engineering; promising five without measuring is marketing.
Twelve words from the incident channel
- Latency vs throughput
- How long one thing takes, versus how many things fit through per second. Improving one often makes the other worse.
- Round trip time
- The time for a packet to reach the other end and come back. The floor for every network interaction, set by physics.
- Cold start
- The delay when a server or function has to be created before it can answer. The hidden cost of scaling from zero.
- Blast radius
- How much breaks when one thing breaks. Good architecture is mostly the deliberate shrinking of this.
- Backpressure
- A system telling its callers to slow down instead of collapsing. Without it, overload turns into a cascading failure.
- Circuit breaker
- Logic that stops calling a failing dependency for a while, so one sick service does not exhaust everyone else's connections.
- Idempotent
- Safe to do twice. Essential, because in a distributed system every retry may already have succeeded silently.
- Blue-green deploy
- Running the new version alongside the old and switching traffic over, so rollback is a routing change rather than a rebuild.
- Infrastructure as code
- The environment defined in files under version control rather than assembled by hand. The difference between rebuilding in an hour and in a fortnight.
- SLO and SLA
- An internal target versus a contractual promise with penalties. The SLO should always be stricter than the SLA.
- MTTR
- Mean time to recovery. Usually more useful to improve than mean time between failures, because failures are inevitable.
- Runbook
- The written procedure for a specific alert. Written calmly in daylight, for someone who is neither.