10 Must-Know System Design Failure Modes

It's easy to explain how a system works when everything goes right. The harder — and more valuable — question is: what happens when it does not?

Distributed systems don't fail in infinite ways — they fail in a small number of recognizable patterns, and the same ones keep appearing in postmortems at every company. Once you can name a pattern, you can design against it before it ever becomes an incident. None of these need exotic infrastructure to hit — most show up the first time a system carries real traffic.

1. Single Point of Failure

Any component on the request path with no redundancy — one load balancer, one primary database, one message broker, one auth service. When it dies, everything behind it becomes unreachable, even though those instances are perfectly healthy.

Clients ──► [Load Balancer] ──► App Server 1, 2, 3 (all healthy)
                  ✗ dies
Clients ──► ✗ ──► unreachable, despite 3 healthy app servers

Fix: Redundancy + automatic failover. Run at least two instances of everything on the critical path, put health checks in front of them, and spread instances across availability zones so one datacenter problem can't remove an entire tier. Failover must be tested — a standby that's never taken traffic is a guess, not a plan.

A useful habit when reviewing any architecture: trace each request path and ask which single box, if deleted, stops all traffic. That's the box to fix first.

2. Cascading Failure

One slow dependency rarely stays contained. A database slows from 20ms to 5s → the service calling it holds its (fixed-size) thread pool open waiting → it stops answering its own callers → they block too. The slowness travels backwards up the call chain until the whole system looks down, even though only one component actually degraded.

Client → Service A → Service B → Service C → DB (slow: 20ms → 5s)
                                     ↑ threads held open
                          ↑ B's threads exhausted, stops responding
             ↑ A's threads exhausted, stops responding

Fix: Containment.

• Aggressive timeouts — a caller gives up rather than waiting forever.
• Circuit breakers — stop sending traffic to a dependency that's already failing.
• Bulkheads — give each dependency its own connection pool, so one sick downstream can't consume every thread the service owns.

3. Retry Storm

Retries are usually good — they stop being good when everyone retries at the same moment. A service comes back after a blip; every client that failed during the outage retries at once, often multiple times each. The recovering service now sees several times its normal load, falls over again, generating a fresh round of retries. The retry logic becomes the outage.

3 clients × 3 retries each = 9x load on a service that just barely recovered → falls over again

Fix:

• Exponential backoff with jitter — spreads attempts across a window instead of synchronizing on the same second.
• Retry budget — caps retries as a percentage of total traffic.
• Circuit breaker — clients stop retrying a service that's clearly down.

4. Cache Stampede

Caches hide how much load your database really receives. When one popular key expires, every request that would have been a cache hit becomes a miss at the same instant. All of them run the same expensive query against the database simultaneously — which suddenly takes the full read volume it's been shielded from for months.

Cache key expires → 1,000 concurrent requests all miss → all 1,000 hit the DB with the same query

Fix:

• Request coalescing (single-flight) — the first miss recomputes the value; the rest wait on that in-flight result instead of each triggering their own DB query.
• Refresh hot keys shortly before expiry — keeps them warm, avoiding the miss entirely.
• Add jitter to TTLs — stops thousands of keys from expiring together in the first place.

5. Hot Partition

Sharding assumes traffic spreads evenly across shards. Real traffic doesn't. Pick a low-cardinality shard key (country, tenant, day) and one shard receives most of the writes while others idle. That shard saturates first — and adding more shards doesn't help, because the skew follows the key, not the machine count.

Shard key = country → Shard 1 (US): 92% of writes | Shard 2, 3: idle

Fix:

• Choose a high-cardinality, evenly-distributed key — e.g., a hash of the user ID.
• For genuinely "celebrity" entities that are hot no matter what: salt the key into sub-partitions, or give those entities dedicated capacity.
• Track per-shard metrics, not cluster averages — an average hides the one node that's on fire.

6. Replication Lag

Read replicas make reads cheap, but they're always slightly behind the primary. Under load, that gap grows from milliseconds to seconds. A user updates their profile, the write commits on the primary, the next read lands on a lagging replica — the change appears to have vanished. Nothing is technically broken, but the bug report is still real.

Client writes → Primary (committed) → Replica (3s behind) → Client reads → gets STALE value

Fix: Be explicit about which reads can tolerate staleness.

• Read-your-writes routing — route to the primary for a short window after a write by that same user.
• Track replication position — only serve a read from a replica that has caught up past the write's position.
• Analytics/feed reads can safely stay on replicas — they don't need freshness.

7. Duplicate Processing

Any retried request may execute twice. The first attempt reaches the server and succeeds, the response is lost to a timeout, and the client retries the same operation. For a read, harmless. For a payment, an email send, or an inventory decrement, the user is charged twice and the ledger is wrong.

Client → charge $100 → Server processes, charges $100 → response lost (timeout)
Client retries → charge $100 → Server charges $100 AGAIN → $200 total charged

Fix: Idempotency. Client attaches a unique idempotency key to each logical operation; server records that key with the result. When a request arrives carrying a key it's already processed, the server skips the work and returns the original response — same request, same outcome, any number of attempts.

8. Queue Backlog

Queues absorb bursts, which is exactly why they hide problems. If producers publish faster than consumers process — even by a small margin — the backlog grows continuously. Every health check stays green while the age of the oldest message climbs from seconds to hours, and by the time anyone notices, the data being processed is badly out of date.

Producers: 10,000 msgs/sec  |  Consumers: 2,000 msgs/sec  →  queue grows by 8,000/sec, forever

Fix: Make the imbalance visible and bounded.

• Alert on consumer lag and oldest-message age, not just queue length alone.
• Autoscale consumers on lag.
• Bound the queue so it applies backpressure, and shed low-value work under sustained overload instead of accepting everything and falling further behind.

9. Poison Message

One malformed message can stop an entire queue. The consumer picks it up, throws, and never acknowledges it. The broker redelivers it, the consumer throws again, and the loop repeats forever. Throughput drops to zero while every valid message behind the bad one waits — and the crash loop can take the whole consumer group down with it.

[bad msg] [msg2] [msg3] ... → consumer crashes on bad msg → redelivered → crashes again → forever
                                                                              ↑ msg2, msg3... never processed

Fix:

• Cap delivery attempts, then move the message to a Dead Letter Queue so the consumer can continue past it.
• Alert on anything landing in the DLQ.
• Keep enough context on the message to fix the underlying bug and replay it later.

10. Split Brain

During a network partition, two nodes can both conclude they are the primary. Each accepts writes from the clients that can still reach it, and the two copies of the data diverge. When the partition heals, there's no correct way to merge them — both sides accepted writes assuming they were authoritative.

Network partition:
  Node A (thinks it's primary) ← writes from Client 1
  Node B (thinks it's primary) ← writes from Client 2
  → data diverges, no clean merge once partition heals

Fix: Make leadership require a majority.

• Quorum-based writes — only the side holding more than half the nodes can commit.
• Consensus-based leader election (e.g., Raft) — prevents two leaders being elected in the same term.
• Fencing tokens — let storage reject writes from a leader whose lease has already expired.

Patterns Worth Pairing Together

A few of these failure modes naturally show up as a package when you propose a design:

• Add a cache → think about stampede protection up front.
• Add sharding → name your shard key and why it won't go hot.
• Add retries → pair them with backoff and jitter.
• Add replicas → decide which reads can tolerate lag.

Anyone can draw boxes on a diagram — the useful skill is explaining how those boxes behave on the worst day of the quarter.

Quick Reference Table

Summary

Distributed systems fail in a recurring, nameable set of ways: missing redundancy, failures propagating through tight coupling (timeouts/thread pools), synchronized client behavior (retries, cache expiry), uneven data/traffic distribution, staleness from replication, non-idempotent side effects, and disagreements about leadership during partitions. Each one has a well-understood fix — redundancy, timeouts/circuit breakers/bulkheads, backoff with jitter, request coalescing, better shard keys, staleness-aware read routing, idempotency keys, lag-based backpressure, dead letter queues, and quorum-based consensus. Designing a system well means assuming each of these will eventually happen, and deciding in advance how the system should behave when it does.