A field guide to distributed-systems failure

A single machine fails honestly. It crashes or it doesn’t, the process is up or down, the disk is there or gone. You can reason about it because the states are discrete and the failure is total.

A distributed system offers no such mercy. It fails partially. One node is slow but not dead. A message arrives and its acknowledgment doesn’t. A replica believes it’s the leader while another replica believes the same thing. The space between working and broken is where you’ll spend your career.

Most engineers arrive from single-machine work carrying the assumption that failure is an exception, something that happens now and then and gets handled in a catch block. In a distributed system failure is the steady state. Right now, in your cluster, something is retrying, something is timing out, something is half-committed. The useful question isn’t what happens if a component fails. It’s which one is failing at this moment, and whether anything noticed.

#The partial-failure problem

The defining hazard is that a component can fail in a way its callers cannot see. A service returning errors is easy, because you get errors. A service that accepts your request, does nothing with it, and never replies is a nightmare, because from the outside that looks exactly like a service which is merely slow.

Which is why timeouts are load-bearing rather than a nicety. A call without one can hang forever, and a thread blocked forever is a thread you’ve lost. Stack enough of those and a single slow dependency drags down a service that was otherwise fine. That’s the cascading failure everyone quotes, where the thing that breaks is never the thing that caused it. I’ve watched a 200ms p99 turn into a dead pool because one downstream call had no deadline on it and the connection just sat there, open, polite, useless.

#Idempotency, or “did that actually happen”

Here’s a question with no clean answer. You sent a request, the connection dropped, and no response came back. Did it succeed? You genuinely cannot know. The server may have processed it and failed to reply, or never received it. From where you’re standing those are the same event.

The only sane response is to make the operation safe to repeat. An idempotent operation gives the same result whether it runs once or five times, which turns “I’m not sure if that happened, so I’ll do it again” from a hazard into a shrug. Attach a unique key to each logical operation, have the server deduplicate on it, and retries become free.

def charge_card(idempotency_key, amount):
    # already processed this exact operation? return the prior result
    existing = ledger.get(idempotency_key)
    if existing:
        return existing  # safe to retry, no double charge

    result = payment_gateway.charge(amount)
    ledger.put(idempotency_key, result)
    return result

Without the key a retry double-charges your customer. With it, the client can retry as aggressively as it likes and the worst case is a wasted round trip. Note that the retry behavior didn’t change at all. The operation became safe to retry, which is a different problem and a much cheaper one to get right.

#Containment beats prevention

You can’t prevent failure here. You can only decide how far it spreads, and most of the discipline is drawing boundaries. Bulkheads, circuit breakers, isolated pools. When a component goes bad the blast radius should be that component.

A circuit breaker is the cleanest version of the idea. Once a dependency starts failing you stop calling it for a while rather than hammering it with requests that were going to fail anyway. You fail fast and locally instead of slowly and globally, which protects the struggling dependency and frees your own resources to serve the requests that never needed it.

The systems that survive production aren’t the ones that never fail. They’re the ones where a dead node is a non-event, three others pick up its work, and the only evidence is a blip on a dashboard nobody had to wake up for.