Distributed systems & scaling

• Monolith vs microservices tradeoffs (don't reach for microservices by default). ★
• API gateway and service discovery roles.
• CAP theorem: under a partition choose Consistency or Availability; CP (MongoDB, HBase) vs AP (Cassandra, CouchDB); caveat — CAP says nothing about scalability. ★
• Eventual consistency and why distributed systems accept it.
• Horizontal vs vertical scaling, statelessness, load balancing. ★
• Resilience patterns that stop a partial failure from cascading: timeouts (bound every call), retries with exponential backoff + jitter (idempotent ops only; jitter avoids a retry storm), circuit breaker (open after N failures → fail fast → half-open to test recovery), and bulkhead (isolate a thread/connection pool per dependency).

02 Curated reading

• The System Design Primer ↗

  essential 40m — The canonical free survey of scaling building blocks.
• Discord — How we store trillions of messages ↗

  optional eng blog 20m — AP datastore (Cassandra→ScyllaDB) at real scale.
• Amazon Builders' Library — Timeouts, retries, and backoff with jitter ↗

  optional eng blog 18m — Resilience patterns for distributed calls.

03 Knowledge check

What gets asked on this topic — tap a card for how to approach it, the follow-ups, and the trap. Company tags are best-effort & sourced.

• ▸ Commonly asked mid concept very common Explain the CAP theorem. Under a partition, what are you actually choosing between?

  CAP says that when a network partition happens, a distributed data store must choose between Consistency (every read sees the latest write) and Availability (every request gets a non-error response). You can't have both during a partition; without a partition you get both.

  So it's really a choice made *when partitioned*. CP systems (e.g. HBase, MongoDB in its default config) refuse or block to stay consistent; AP systems (e.g. Cassandra, CouchDB) keep serving and reconcile later (eventual consistency). Important caveat: CAP says nothing about latency or scalability — it's strictly about behavior under partition.

  Follow-ups they push on

  • Why is the 'pick 2 of 3' framing misleading?
  • What does PACELC add to CAP?

  Red flag Saying 'pick 2 of 3' as if you choose freely. Partition tolerance is mandatory in a distributed system; the real choice is C vs A only when a partition occurs.

  source: system-design-primer — CAP theorem ↗
• ▸ Commonly asked senior concept very common Monolith vs microservices — what are the real tradeoffs, and why not default to microservices?

  A monolith is one deployable: simpler local dev, easy refactors across boundaries, in-process calls, one transaction — at the cost of coupled deploys and scaling the whole app together. Microservices give independent deploys, team autonomy, and targeted scaling — but you pay with distributed-systems tax: network failures, eventual consistency, distributed transactions/sagas, harder debugging and tracing, and heavy ops.

  The seasoned answer: don't reach for microservices by default. Most teams should start with a well-modularized monolith and split out services only when a clear scaling, team-ownership, or deploy-cadence boundary justifies the added operational cost.

  Follow-ups they push on

  • What forces a split — scaling, team size, or deploy cadence?
  • What is a 'distributed monolith' and why is it the worst of both?

  Red flag Starting greenfield with microservices for resume-driven reasons, inheriting distributed-systems complexity before you have the scale or teams to need it.

  source: Martin Fowler — Monolith First ↗
• ▸ Commonly asked mid concept very common Horizontal vs vertical scaling — and why does statelessness matter for scaling out?

  Vertical scaling means a bigger machine (more CPU/RAM) — simple but bounded by the largest box and a single point of failure. Horizontal scaling means more machines behind a load balancer — effectively unbounded and fault-tolerant, but only if requests can hit any node.

  That's why statelessness matters: if a server keeps user session state in local memory, the load balancer must pin a user to one node (sticky sessions), which breaks failover and uneven load. Push state to a shared store (Redis/DB) so any node can serve any request, and horizontal scaling becomes trivial.

  Follow-ups they push on

  • What breaks if you keep sessions in local server memory?
  • How do load balancers route — round robin, least connections, hashing?

  Red flag Storing session state in process memory and then trying to scale horizontally — you're forced into sticky sessions, which undermine failover and balancing.

  source: system-design-primer — Scalability ↗
• ▸ Commonly asked mid concept common What is eventual consistency, and why do distributed systems accept it?

  Eventual consistency means replicas may temporarily disagree, but if writes stop, they converge to the same value given enough time. AP systems accept it as the price of staying available and low-latency under partitions and across regions.

  Why accept it: strong consistency requires coordination (consensus, quorums) on every write, which adds latency and reduces availability when nodes can't reach each other. For many features — a like count, a social feed, a shopping cart — a few seconds of staleness is fine, and the availability/latency win is worth it. For money movement you choose strong consistency instead.

  Follow-ups they push on

  • Give a feature where eventual consistency is fine and one where it's not.
  • What is read-your-own-writes consistency?

  Red flag Using eventual consistency for invariants that must hold immediately (e.g. account balances). Match the consistency level to the business need.

  source: AWS — Eventual consistency ↗
• ▸ Commonly asked mid concept common What roles do an API gateway and service discovery play in a microservices system?

  An API gateway is the single entry point for clients: it routes to the right service and handles cross-cutting concerns — auth, rate limiting, TLS termination, request aggregation, and sometimes response shaping — so each service doesn't reimplement them and clients don't need to know the internal topology.

  Service discovery lets services find each other's network locations as instances scale up/down and move. A registry (Consul, Eureka, or Kubernetes DNS/Services) maps a logical service name to current healthy instances, so callers resolve a name instead of hardcoding IPs. Together they decouple clients from the shifting set of backend instances.

  Follow-ups they push on

  • Client-side vs server-side discovery — what's the difference?
  • What concerns belong in the gateway vs each service?

  Red flag Putting business logic in the gateway. It handles routing and cross-cutting concerns; domain logic stays in the services.

  source: microservices.io — API gateway & service discovery ↗
• ▸ Commonly asked senior design common Why retry failed calls with exponential backoff AND jitter? What goes wrong without jitter?

  Retries handle transient failures, but naive retries cause two problems. Exponential backoff (wait 1s, 2s, 4s…) stops a struggling service from being hammered every few milliseconds while it tries to recover.

  Jitter (randomizing each wait) prevents a thundering herd: if many clients fail at the same instant and all back off by the exact same schedule, they retry in synchronized waves that keep knocking the service over. Adding randomness spreads the retries out. Pair this with retry budgets/circuit breakers and only retry idempotent or idempotency-keyed operations.

  Follow-ups they push on

  • Why only retry idempotent operations?
  • What does a circuit breaker add on top of backoff?

  Red flag Backoff without jitter — synchronized clients retry in lockstep, creating a self-reinforcing herd that prevents recovery.

  source: AWS Builders' Library — Timeouts, retries, and backoff with jitter ↗
• ▸ Commonly asked senior concept common What is a circuit breaker and how does it protect a distributed system?

  A circuit breaker wraps calls to a dependency and tracks failures. In closed state calls pass through; once failures cross a threshold it opens and fails fast (returns an error or fallback immediately) instead of piling up calls on a sick service. After a cooldown it goes half-open, lets a trial request through, and closes again if it succeeds.

  It prevents cascading failures: without it, a slow dependency exhausts the caller's threads/connections waiting on timeouts, which then takes the caller down, propagating upstream. Failing fast contains the blast radius and lets the dependency recover.

  Follow-ups they push on

  • Walk through closed → open → half-open transitions.
  • How does this complement timeouts and bulkheads?

  Red flag Relying on retries alone with no breaker — retries against a failing dependency amplify load and accelerate the cascade.

  source: Martin Fowler — CircuitBreaker ↗
• ▸ Commonly asked senior concept common What is consistent hashing and why do distributed caches and databases use it?

  With plain hash(key) % N, changing the number of nodes N remaps almost every key — catastrophic for a cache (mass misses) or a sharded DB (mass data movement). Consistent hashing maps both keys and nodes onto a ring; a key belongs to the next node clockwise. Adding or removing a node only relocates the keys in that node's arc — about 1/N of keys — instead of nearly all of them.

  Virtual nodes (each physical node placed at many ring positions) smooth out uneven distribution. This is why Cassandra, DynamoDB, and Memcached-style caches use it.

  Follow-ups they push on

  • What do virtual nodes solve?
  • How much data moves when you add the (N+1)th node?

  Red flag Using modulo hashing for a sharded cluster, so adding one node reshuffles nearly all keys and stampedes the backing store.

  source: system-design-primer — Consistent hashing ↗
• ▸ Commonly asked senior concept occasional Why are leader election and quorum used in distributed coordination?

  Many tasks need exactly one node in charge (assigning work, ordering writes) to avoid conflicts — so the cluster elects a leader via a consensus protocol (Raft, Paxos, ZooKeeper/ZAB). If the leader dies, a new one is elected.

  To agree despite failures, decisions use a quorum — a majority (N/2 + 1). Requiring a majority for writes and reads guarantees any two quorums overlap, so the system never commits two conflicting decisions and can tolerate a minority of nodes failing. This is the backbone of consistent distributed stores and coordination services.

  Follow-ups they push on

  • Why a majority specifically? (overlapping quorums prevent split decisions)
  • What is split-brain and how does quorum prevent it?

  Red flag Allowing writes without a majority quorum, enabling split-brain where two partitions both think they have a leader and diverge.

  source: The Raft Consensus Algorithm ↗
• ▸ Commonly asked mid concept common Compare load-balancing algorithms: round robin, least connections, and consistent hashing. When does each shine?

  Round robin sends each request to the next server in rotation — simple and fine when requests are uniform and servers identical, but blind to actual load. Least connections routes to the server with the fewest active connections — better when request durations vary, since it adapts to real load instead of assuming uniformity.

  Consistent hashing (hash the client/key to a server) keeps a given key/session on the same server — essential for cache affinity or sticky routing, and it minimizes remapping when servers are added/removed. Round robin for stateless uniform work; least connections for variable work; consistent hashing when affinity/locality matters.

  What a strong answer covers

  • Round robin: simple rotation, ignores load; good for uniform requests.

  • Least connections: adapts to variable request durations.

  • Consistent hashing: routes a key to a stable server (cache/session affinity).

  • Weighted variants account for heterogeneous server capacity.

  Quick self-check

  Requests have highly variable processing times. Which LB algorithm adapts best to real server load?

  • Round robin
  • Least connections
  • Random
  • First-available-by-name

  Follow-ups they push on

  • When does round robin distribute poorly?
  • Why does consistent hashing help cache hit rates behind a load balancer?

  Red flag Defaulting to round robin when request costs vary wildly — a few expensive requests pile onto one server while others idle. Least connections adapts better.

  source: Cloudflare — What is load balancing? ↗
• ▸ ★ must-know Commonly asked senior concept common Why is an idempotency key essential for a client retry after a timeout, and what subtlety makes timeouts dangerous?

  A timeout is ambiguous: when a client's request times out, it cannot tell whether the server never received it, processed it but the response was lost, or is still processing. So a retry might be a true retry or an accidental duplicate of a request that already succeeded.

  That's why a non-idempotent operation (charge a card, place an order) needs an idempotency key: the client sends a stable key, the server dedupes on it, and a retry of an already-applied request returns the original result instead of re-applying it. Without the key, the safe-looking retry can double-charge. The subtlety: the failure you can see (timeout) hides whether the side effect happened.

  What a strong answer covers

  • A timeout doesn't tell you if the operation succeeded — it's inherently ambiguous.

  • Retrying a non-idempotent op risks a duplicate side effect.

  • An idempotency key lets the server dedupe and return the first result.

  • Idempotent methods (GET/PUT/DELETE) are safe to retry without a key.

  Follow-ups they push on

  • Why can a request that 'timed out' have actually succeeded server-side?
  • How does this connect to at-least-once delivery in messaging?

  Red flag Treating a timeout as a definite failure and blindly retrying a charge/order. The request may have completed; without an idempotency key you double-apply it.

  source: AWS Builders' Library — Making retries safe with idempotent APIs ↗
• ▸ Commonly asked mid concept common What is the difference between latency and throughput, and why can optimizing one hurt the other?

  Latency is how long a single operation takes (time per request); throughput is how many operations complete per unit time. They're related but distinct — a system can have high throughput and high latency at once.

  They trade off because techniques that raise throughput often add per-request latency: batching many requests amortizes overhead (more throughput) but each request waits for the batch to fill (more latency); deep queues keep workers busy (throughput) but messages wait longer (latency). The discipline is to measure latency as a distribution (p50/p95/p99), not a mean, since tail latency is what users feel, and to choose the tradeoff per workload.

  What a strong answer covers

  • Latency = time per operation; throughput = operations per unit time.

  • Batching/queuing raise throughput but add per-request latency.

  • Report latency as percentiles (p95/p99), not averages — tails matter.

  • Little's Law links them: concurrency ≈ throughput × latency.

  Quick self-check

  Why is p99 latency usually more informative than mean latency?

  • It's easier to compute
  • It captures the slow tail that real users experience, which the mean hides
  • It's always equal to the maximum
  • It ignores outliers entirely

  Follow-ups they push on

  • Why report p99 instead of the mean?
  • How does batching trade latency for throughput?

  Red flag Reporting only average latency. A good mean can hide a terrible p99 that real users hit; and maxing throughput via batching can quietly wreck per-request latency.

  source: Hello Interview — Latency vs throughput ↗
• ▸ Commonly asked senior design common Design read scaling for a heavily-read database. How do replication and the read-your-writes problem interact?

  For read-heavy load, add read replicas: writes go to the primary, which asynchronously replicates to replicas that serve reads, spreading read load and adding redundancy. The catch is replication lag — a replica may be milliseconds-to-seconds behind, so a user who just wrote can read a replica and not see their own change.

  Fix the read-your-writes experience by routing a user's reads to the primary for a short window after they write, pinning their session to the primary, tracking a write timestamp/LSN and only reading replicas caught up past it, or using synchronous replication for the critical path (at a latency cost). Layer caching and, if writes also dominate, consider sharding.

  Follow-ups they push on

  • What is replication lag and how do you measure it?
  • When would you shard instead of (or in addition to) adding replicas?

  Red flag Sending a user's immediate post-write read to an async replica and showing them stale data ('I just saved it — where did it go?'). Route recent writers to the primary or track their write position.

  source: system-design-primer — Replication & federation ↗
• ▸ Commonly asked senior trick occasional Trick: a service adds aggressive client retries to improve reliability and the whole system gets less reliable under load. What happened?

  This is a retry storm / metastable failure. When a dependency slows or briefly fails under load, every client retries — often multiplying traffic 3x or more right when the service is least able to handle it. The added load keeps the service overloaded, so it keeps failing, so clients keep retrying: a self-sustaining feedback loop that doesn't recover even after the original trigger passes.

  Fixes: bound retries with a retry budget (cap retries as a fraction of traffic, not per-request), add exponential backoff with jitter, use circuit breakers to fail fast, and only retry idempotent operations. Retries help with isolated transient blips; unbounded retries under systemic load amplify the failure.

  What a strong answer covers

  • Mass retries multiply load exactly when the service is already struggling.

  • Creates a self-sustaining (metastable) overload that outlasts the trigger.

  • Fix: retry budgets, backoff + jitter, circuit breakers, retry only idempotent ops.

  • Retries help isolated blips, not systemic overload.

  Quick self-check

  Aggressive unconditional retries make a system LESS reliable under load because:

  • Retries use more memory per request
  • They multiply traffic to an already-struggling service, sustaining the overload
  • Retries always hit a different server
  • They disable the load balancer

  Follow-ups they push on

  • What is a retry budget and why cap retries as a fraction of total traffic?
  • How does a circuit breaker break the feedback loop?

  Red flag Adding per-request retries everywhere as a blanket reliability boost. Under correlated failure they amplify load into a retry storm — bound them with budgets, backoff, and breakers.

  source: AWS Builders' Library — Timeouts, retries, and backoff with jitter ↗
• ▸ Commonly asked senior concept common What is sharding (horizontal partitioning), and why is choosing a good shard key the hard part?

  Sharding splits one logical dataset across multiple databases/nodes by a shard key, so each shard holds a subset and the system scales writes and storage beyond one machine. Reads/writes route to the shard owning the key.

  The shard key is the hard part because a bad one creates hotspots — picking a low-cardinality or monotonically-increasing key (like a timestamp) funnels traffic to one shard, defeating the point. You want a key that spreads load evenly *and* keeps commonly-joined data co-located so you avoid expensive cross-shard queries. Cross-shard transactions and re-sharding as you grow are the recurring pains, which is why teams delay sharding until replicas and caching are exhausted.

  What a strong answer covers

  • Sharding = horizontal partitioning by a shard key across nodes.

  • Scales writes/storage past a single machine.

  • Bad shard key → hotspots (monotonic or low-cardinality keys are traps).

  • Cross-shard joins/transactions and re-sharding are the ongoing costs.

  Follow-ups they push on

  • Why does a timestamp or auto-increment id make a poor shard key?
  • How does consistent hashing reduce re-sharding pain?

  Red flag Sharding on a monotonically increasing key (timestamp/sequence id) so all new writes hit the newest shard — a hotspot that recreates the single-node bottleneck.

  source: MongoDB — Sharding and shard keys ↗