Scalability & Load Balancing

In a placement system-design round, the first thing an interviewer probes is: "your service is getting 10x traffic — what breaks, and how do you grow?" Getting scalability and load balancing right is the backbone of almost every answer.

The core idea

Scalability is a system's ability to handle growing load — more users, requests, or data — without a collapse in performance. You grow in one of two directions. Vertical scaling (scale up) means putting a bigger machine underneath: more CPU, RAM, faster disk. Horizontal scaling (scale out) means adding more machines and splitting the work between them. A load balancer is the traffic cop that sits in front of a pool of servers and distributes incoming requests so no single server is overwhelmed.

Beginner — vertical vs horizontal

Vertical scaling is the simplest: no code changes, just a bigger box. But it has a hard ceiling (the biggest machine money can buy) and a single point of failure. Horizontal scaling has no real ceiling and gives you redundancy, but your app must become stateless — any server should be able to handle any request. That usually means moving session state out to a shared store like Redis.

Intermediate — load-balancing algorithms

A load balancer picks which server gets each request:

• Round robin — cycle through servers in order. Simple and fair when servers are equal.
• Weighted round robin — give beefier servers a bigger share.
• Least connections — send to the server with the fewest active connections; good for long-lived or uneven requests.
• IP hash — hash the client IP so the same client keeps hitting the same server (sticky sessions).

Load balancers also run health checks, pulling a dead server out of rotation automatically.

Advanced — layers, availability, and bottlenecks

Load balancing happens at Layer 4 (transport — routing by IP/port, very fast) or Layer 7 (application — routing by URL, headers, cookies, enabling smart rules). Scaling the web tier is easy; the database usually becomes the bottleneck, which is why read replicas, caching, and sharding follow. The load balancer itself must not be a single point of failure — you run it in an active-passive or active-active pair.

Worked example

An app on one server maxes out at 2,000 requests/sec. Traffic climbs to 8,000 req/sec. Vertical scaling can't get you 4x affordably. Instead you put four stateless app servers behind a load balancer using least-connections, move sessions to Redis, and add health checks. Now the tier handles ~8,000 req/sec with a spare server tolerating one failure.

When to use

Reach for horizontal scaling + a load balancer whenever you need high availability or growth beyond one machine — essentially every web-scale service. Use vertical scaling early, for quick wins, or for hard-to-distribute components like a primary database.

Tricks and mnemonic

Remember "UP is a bigger box, OUT is more boxes." Up hits a ceiling and a single point of failure; out scales forever but demands statelessness.

Quick revision of how systems grow under load and how traffic is spread.

• Scalability = handle growing load without performance collapse.
• Vertical (up) = bigger machine — simple, but has a ceiling and single point of failure.
• Horizontal (out) = more machines — unlimited growth + redundancy, but app must be stateless.
• Load balancer distributes requests and runs health checks to drop dead servers.
• Algorithms: round robin, weighted round robin, least connections, IP hash (sticky).
• L4 balancing routes by IP/port (fast); L7 routes by URL/headers (smart).
• The database is usually the next bottleneck after the web tier scales out.

Cover the answer, recall, then check. 8 cards.

Q1. What is scalability?
A1. A system's ability to handle growing load (users/requests/data) without a collapse in performance.

Q2. Vertical vs horizontal scaling?
A2. Vertical = bigger single machine; horizontal = more machines sharing the work.

Q3. What is the main limitation of vertical scaling?
A3. A hard ceiling (biggest machine available) and a single point of failure.

Q4. What must an app be to scale horizontally cleanly?
A4. Stateless — any server can handle any request; state lives in a shared store.

Q5. What does a load balancer do besides distributing requests?
A5. Runs health checks and removes unhealthy servers from rotation.

Q6. Name three load-balancing algorithms.
A6. Round robin, least connections, IP hash (also weighted round robin).

Q7. Difference between Layer 4 and Layer 7 load balancing?
A7. L4 routes by IP/port (fast); L7 routes by URL/headers/cookies (smart).

Q8. After the web tier scales out, what usually becomes the bottleneck?
A8. The database.

Worked Example

Problem: A web app runs on one server handling ~1,200 requests/second. Peak traffic is projected to reach 5,000 req/s. How would you scale it, roughly how many servers are needed, and how are requests distributed?

Solution: A single machine cannot serve 5,000 req/s, so scale horizontally — add more servers behind a load balancer — rather than only buying a bigger box (vertical scaling has a hard ceiling).

Estimate the fleet size:

• Needed capacity / per-server capacity = 5,000 / 1,200 ≈ 4.17 → 5 servers.
• Add headroom for spikes and failover (N+1 or N+2) → provision 6 servers.

A load balancer (L7) spreads requests using round-robin or, better, least-connections so busy servers get fewer new requests. Crucially, make the servers stateless — store sessions in a shared store (Redis) — so any server can handle any request and instances can be added or removed freely.

Answer: ~5-6 stateless servers behind a least-connections load balancer.