## Load balancing for system design interviews, explained

Load balancing for system design interviews, explained

Every system design interview reaches the moment you draw a second server. The instant there are two, something has to decide which one a request goes to — and that box is the load balancer. "Put a load balancer in front" is the easy half of the sentence; the real question is the one the interviewer asks next: which algorithm, at which layer, and what happens when a server behind it dies mid-request? Load balancing is not one idea but a set of deliberate choices about how traffic is spread, how unhealthy instances are detected and removed, and how the balancer itself avoids becoming the thing that takes the whole system down.

The reason it comes up in almost every round is that a load balancer is how a system earns two properties every design cares about: scalability (add more servers and traffic spreads across them) and availability (a server fails and traffic routes around it with no one noticing). But a balancer is also a new component in the hot path of every request, so it introduces its own questions — what it inspects to make a routing decision, how it knows a backend is healthy, whether a user must keep hitting the same instance, and what stands behind the balancer so it is not a single point of failure. This guide walks the choices a strong candidate names out loud: where the balancer sits (Layer 4 vs Layer 7), the algorithms that spread requests, and the operational story — health checks, failover and sticky sessions — that the follow-ups are really about.

Why load balancing comes up in system design interviews

Load balancing rarely gets its own whiteboard prompt. It appears inside almost every design — a URL shortener, a news feed, a payment system, a chat backend — at the moment you say "we'll run multiple application servers." You draw a box in front of them, and the interviewer leans in: not whether to balance load, but how. Which algorithm, what the balancer inspects, how it detects a dead backend, and what happens to a request that was in flight when a server failed. Load balancing is a favourite probe precisely because the box is easy to draw and everything true and hard lives in how it behaves under failure.

The interviewer rarely lets "add a load balancer" stand on its own. "You're round-robining across four servers and one of them starts returning 500s on every request. Round robin keeps sending it a quarter of your traffic. Walk me through how the balancer notices, what it does about it, and what happens to the requests already in flight to that server." That question — detecting and routing around failure — is the one a memorised "use a load balancer" answer can never survive, and the one this topic is really testing.

The rounds on Zynter are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: you name an algorithm, and the next sentence out of the interviewer's mouth attacks its weakest point — an uneven load, a slow backend starving, a dropped session, or the balancer itself failing.

What the interviewer is really testing

A load balancing answer is read on the same four dimensions as any design decision. Problem solving — did you pick an algorithm that fits the traffic shape, or reach for round robin because it is the one you remember? Communication — can you say in one sentence why least connections beats round robin when request costs vary wildly? Technical execution — do you handle health checks and the balancer as its own single point of failure, or did you leave a new one in the diagram? Time management — did you spend your words on the failure and scaling story, not on reciting that a load balancer "distributes traffic"? Reading this guide teaches the mechanisms; it does not teach you to defend one while someone pushes on the dead backend.

Where the load balancer sits: Layer 4 vs Layer 7

Before the algorithm, name the layer the balancer operates at, because it decides what the balancer can even see.

Layer 4 (transport layer) balances on IP address and TCP/UDP port. It forwards packets without parsing the request, so it is fast and cheap and protocol-agnostic — it does not know or care whether the bytes are HTTP, gRPC or a raw TCP stream. The cost is that it cannot make decisions based on content: it cannot route /api/search to the search pool or read a cookie. AWS Network Load Balancer and HAProxy in TCP mode are Layer 4.

Layer 7 (application layer) terminates the connection and parses the full request — HTTP method, path, headers, cookies. That unlocks content-based routing: send /images/* to one pool and /checkout to another, pin a session with a cookie, terminate TLS, or rate-limit per path. The cost is overhead — it does more work per request — and it must understand the protocol. AWS Application Load Balancer, NGINX and Envoy are Layer 7.

Say the distinction crisply: Layer 4 is fast and blind — it moves packets by IP and port; Layer 7 is slower and smart — it reads the request and routes on its content. For most microservice and web designs, Layer 7 is the default because the routing flexibility is worth the overhead; reach for Layer 4 when you need raw throughput, non-HTTP protocols, or the lowest possible latency.

Load balancing algorithms: how traffic actually gets spread

Once the layer is settled, the algorithm decides which healthy backend each request goes to. Name a few and, more importantly, when each fits.

• Round robin — hand each request to the next server in rotation. Simple and even when every request costs about the same. Its weakness is exactly that assumption: a server handling one slow request still gets its turn for the next one.
• Weighted round robin — give beefier servers a larger share of the rotation. The fix for a heterogeneous fleet where some instances have more CPU or memory than others.
• Least connections — send the next request to the server with the fewest active connections. The right default when request durations vary a lot (some return in 5 ms, some hold a connection for seconds), because it tracks real load instead of assuming uniformity.
• Least response time — least connections, plus the backend's observed latency, so a server that is technically idle but slow stops getting piled on.
• IP hash / consistent hashing — hash the client (or a key) to a server so the same client reliably lands on the same instance. This is how you get session affinity without a cookie, and consistent hashing specifically minimises how many keys move when a server is added or removed. (Consistent hashing is its own interview topic for exactly this reason.)
• Power of two choices — pick two servers at random and send the request to the less loaded of the two. A cheap trick that gets most of the benefit of least connections without the balancer tracking global state, and it avoids the "everyone stampedes the one idle server" problem a naive least-connections can cause.

The sentence that scores: round robin assumes every request is equal, so I'd reach for least connections (or power of two choices at scale) when request costs vary, and consistent hashing when I need the same client or key to keep landing on the same instance. Naming why you moved off round robin is what separates a real answer from "use round robin."

System Design
HLD · Core concept
Pick and defend a load balancing algorithm out loud

Practise on Zynter →

Health checks, failover and the balancer itself

The algorithm only ever picks from servers it believes are healthy, so the hard part — and where most follow-ups land — is how the pool stays accurate and what happens when a backend, or the balancer itself, fails.

Health checks keep the pool honest:

• Active health checks — the balancer periodically probes each backend (a GET /health every few seconds); a configured number of consecutive failures removes the instance from the pool, and a run of successes adds it back. Catches a dead server before real traffic hits it.
• Passive health checks — the balancer watches real traffic and marks an instance unhealthy after observed errors (consecutive 5xx responses or connection failures). Catches problems a shallow probe misses, because it reflects what users actually experience.
• A robust setup uses both: active probes to catch outright failure fast, passive observation to catch the subtler "healthy but wrong" case. The request already in flight to a failing server is handled by a retry to a healthy backend (safe only for idempotent requests — a point worth saying out loud).

Failover and the balancer as a single point of failure. The trap in a naive diagram is that the load balancer you added to improve availability is now the one box whose failure takes everything down. The answer the interviewer is listening for:

• Run the balancer as a redundant pair (active–passive or active–active) with a floating/virtual IP that fails over to the standby in seconds.
• Use DNS or anycast in front for a layer of balancing across multiple balancers in multiple zones, so no single balancer — or availability zone — is critical.
• Sticky sessions (session affinity) pin a user to one backend (via a cookie or IP hash) so in-memory session state stays reachable. State the cost plainly: stickiness undermines even balancing and means a server's failure drops its users' sessions — which is exactly why the better answer is usually to make servers stateless and push session state to a shared store (like Redis), so any server can serve any request.

The point to state plainly: health checks keep the pool accurate, retries keep a single failure invisible, a redundant pair keeps the balancer from being the failure, and stateless servers keep stickiness from forcing your hand. Those are four different jobs, and a strong answer names which one each mechanism does.

System Design
HLD · Core concept
Defend your health-check and failover plan out loud

Practise on Zynter →

How to talk about load balancing in an interview

The probes from every section rhyme. Master these five and you can defend a load balancing decision in any design, not just this checklist.

• "You've added a load balancer — Layer 4 or Layer 7, and why?" → Layer 7 by default so I can route on path and cookie and terminate TLS; Layer 4 when I need raw throughput or a non-HTTP protocol and don't need to read the request. (problem solving)
• "Which algorithm, and why not plain round robin?" → Least connections when request costs vary, power of two choices at large scale, consistent hashing when I need the same client or key to keep hitting the same instance; round robin only when requests are genuinely uniform. (technical execution)
• "A backend starts failing every request — what happens?" → Active and passive health checks pull it from the pool after consecutive failures, in-flight idempotent requests retry to a healthy server, and it is added back only after it passes probes again. (technical execution)
• "Your load balancer just became a single point of failure — fix it." → A redundant active–passive (or active–active) pair with a floating IP, plus DNS or anycast across zones so no single balancer or AZ is critical. (communication)
• "Do you need sticky sessions here?" → Prefer stateless servers with session state in a shared store so any server serves any request; reach for stickiness only when in-memory state is unavoidable, and name the cost — uneven balancing and lost sessions on a server failure. (problem solving)

Each maps to a debrief dimension. Reading them teaches the answers; it does not teach you to deliver them while someone interrupts with "but that server is already overloaded" — that only comes from saying them out loud.

How to prepare: a short plan

You do not need a new topic for every design. You need the load balancing vocabulary automatic, so when a design grows a second server you reach for the right choice without thinking.

• Day 1: Learn the layers (Layer 4 vs Layer 7) and the algorithms (round robin, weighted, least connections, least response time, consistent hash, power of two choices) until you can state each one's one-sentence trade-off from memory.
• Day 2: Drill the operational story — active vs passive health checks, retries and idempotency, the balancer as a single point of failure and its redundant-pair fix, and sticky sessions vs stateless servers. These are where the follow-ups live.
• Day 3: Place a load balancer inside a real design (a web app, an API, a chat backend): pick a layer and an algorithm, add health checks, make the balancer redundant, and decide on session state. Then do it out loud, timed, with someone pushing on the dead backend and the failed balancer.

This explainer feeds the broader Top 25 system design interview questions — load balancing is a decision inside almost every one of them, so getting it automatic pays off across the whole list.

Frequently asked questions

What is the difference between Layer 4 and Layer 7 load balancing?

A Layer 4 load balancer operates at the transport layer and routes traffic based on IP address and TCP/UDP port without inspecting the request contents, which makes it fast, cheap and protocol-agnostic. A Layer 7 load balancer operates at the application layer: it terminates the connection and parses the full request — HTTP method, path, headers and cookies — which lets it make content-based routing decisions such as sending a URL path to a specific pool, pinning a session with a cookie, or terminating TLS. The trade-off is overhead: Layer 7 does more work per request. Most web and microservice designs default to Layer 7 for the routing flexibility and reach for Layer 4 when they need raw throughput or a non-HTTP protocol.

Which load balancing algorithm should I use?

It depends on the shape of your traffic. Round robin is fine when every request costs about the same, because it just cycles evenly through the servers. When request durations vary a lot, least connections is usually better because it sends work to the server with the fewest active connections rather than assuming uniformity, and least response time refines that by also accounting for latency. At large scale, power of two choices gets most of the benefit of least connections without tracking global state. Use IP hash or consistent hashing when you need the same client or key to keep landing on the same server, for session affinity or cache locality. The interview-worthy move is explaining why you moved off round robin, not just naming one.

What are health checks and why do they matter?

Health checks are how a load balancer keeps its pool of backends accurate, because every algorithm assumes the servers it picks from are healthy. Active health checks periodically probe each backend (for example a GET /health every few seconds) and remove an instance after a configured number of consecutive failures, adding it back once it passes again. Passive health checks watch real traffic and mark an instance unhealthy based on observed errors such as consecutive 5xx responses or connection failures. A robust setup uses both — active probes to catch outright failure quickly, passive observation to catch the subtler "healthy but returning errors" case — and retries in-flight idempotent requests to a healthy server.

Is the load balancer a single point of failure?

It can be, and that is one of the most common follow-ups, because the load balancer you added to improve availability becomes the one box whose failure takes everything down if you leave it standalone. The standard fix is to run it as a redundant pair — active–passive or active–active — with a floating or virtual IP that fails over to the standby in seconds. For a stronger guarantee you put DNS or anycast in front so traffic can be balanced across multiple load balancers in multiple availability zones, meaning no single balancer or zone is critical. Naming this unprompted signals that you noticed the new point of failure you just drew.

What are sticky sessions and should I use them?

Sticky sessions, or session affinity, pin a user to a specific backend server — usually with a cookie or by hashing the client IP — so that in-memory session state on that server stays reachable across requests. The cost is real: stickiness undermines even load distribution, and if that server fails, its users' sessions are lost. For that reason the usually-preferred answer in an interview is to make the servers stateless and push session state to a shared store such as Redis, so any server can serve any request and stickiness is unnecessary. Reach for sticky sessions only when in-memory state is genuinely unavoidable, and name the trade-off when you do.

How do I practise a system design interview out loud?

Reading about load balancing is not the same as defending an algorithm choice while an interviewer pushes on a failing backend. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a design on a shared canvas, and a voice interviewer follows up one question at a time ("one server is 500-ing on every request, what does round robin do?", "your load balancer just died — now what?"), then gives you a debrief on problem solving, communication, technical execution and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can name round robin and least connections. The round is won on defending the dead backend out loud.

Pick an HLD system design round, put a load balancer in your design on the canvas, and defend it out loud to a voice interviewer that keeps asking "which algorithm, and why not round robin?" and "that backend is failing every request — what happens to the traffic?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise an HLD system design round →
Browse all interview tracks