SYSTEM DESIGN · QUESTION 12
How would you design a rate limiter?
On this page
A rate limiter decides whether a request is allowed under a policy such as N requests per identity per time interval. I would place it at an API gateway or service boundary, choose an algorithm such as token bucket based on burst behavior, and use atomic state updates when limits are shared across instances. The design must define identity, response headers, storage failures, and whether temporary inconsistency is acceptable.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A rate limiter protects a service by controlling how quickly a client can make requests. It needs a rule (for example, a refillable allowance), a key that identifies whose requests are counted, and state that is updated safely. A single-process limit is simple, but a distributed service needs coordinated state or a deliberate consistency trade-off. A rejected request should tell the client what happened and when retrying may make sense.
A USEFUL COMPARISON
Picture it this way
Imagine a venue handing out a limited number of entry tokens and adding a few back over time. A guest can enter while holding a token; once none remain, they wait. A token bucket works similarly and allows short bursts, while other algorithms shape traffic differently.
FOLLOW THE JOURNEY
The limiter sits before protected service work
Show the request decision, shared counter/state, and allow or reject paths.
A distributed implementation may place state in a shared store or use local approximations. The choice changes consistency, latency, and failure behavior.
A client request reaches a gateway rate limiter, which atomically checks policy state; allowed requests continue to the service and rejected requests receive a throttling response.
How to read this visual
A distributed implementation may place state in a shared store or use local approximations. The choice changes consistency, latency, and failure behavior.
UNDER THE HOOD
How it works, step by step
Define the policy
Choose identity, limit, interval, burst allowance, and exceptions.
Why it matters: A numeric limit is meaningless without knowing who and what it applies to.
Choose an algorithm
Select fixed window, sliding window, token bucket, or leaky bucket.
Why it matters: Algorithms differ in boundary bursts, smoothness, and state needs.
Make the decision
Check and update limiter state atomically or define an acceptable approximation.
Why it matters: Concurrent requests must not all consume the same remaining allowance.
Respond clearly
Allow work or return 429 with retry information where available.
Why it matters: Clients need a useful and predictable recovery signal.
Plan for failure
Decide whether dependency failure is fail-open, fail-closed, or limited locally.
Why it matters: The choice balances protecting the service against availability.
OUTSIDE THE TEXTBOOK
A practical example
A public API may allow a burst of 20 requests and refill one token each second for each API key. The gateway checks and consumes a token before calling the application. When none remain, it returns 429 and a retry hint. A separate service-wide emergency limit can protect capacity even if individual clients stay within their quotas.
GO A LITTLE DEEPER
Technical details that matter
- Fixed windows are simple but can allow nearly twice the limit around a window boundary.
- Sliding logs are accurate but store request timestamps; sliding counters approximate a rolling window with less state.
- Token bucket allows bursts up to bucket capacity while controlling average rate; leaky bucket smooths outgoing work.
- In distributed deployments, atomic increments or scripts can coordinate state, but network latency, hot keys, and store availability matter.
- Use a monotonic time source where possible and define behavior for clock differences and expiration.
if limiter.allow(api_key):
return forward_request()
return Response(
status=429,
headers={"Retry-After": retry_seconds},
)This pseudocode shows the boundary contract, not a complete algorithm. The allow operation must update shared state safely if multiple server instances enforce the same policy.
MAKE A GOOD CHOICE
When is this useful?
- Protecting expensive APIs from accidental or abusive bursts.
- Fairly allocating shared service capacity among tenants.
- Applying separate coarse global limits and finer per-identity quotas.
Concurrency limits or queues may better protect long-running work; rate limits govern arrival frequency, not the number of jobs currently executing.
WATCH OUT FOR
Common misunderstandings
- Choosing a limit without defining identity or burst behavior.
- Using non-atomic read-then-write counters across concurrent requests.
- Treating rate limiting as a complete abuse-prevention or authorization system.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Shared state adds latency and another failure dependency.
- Approximate distributed limits may temporarily exceed a strict quota.
- Overly strict policies can block legitimate clients behind shared network identities.
WHAT TO REMEMBER
Key takeaways
- 1A limiter enforces a policy at a well-defined request boundary.
- 2Algorithm choice controls burst and smoothing behavior.
- 3Distributed updates need atomicity or an explicit approximation.
- 4Define 429 behavior and fail-open/closed policy.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
When would you use token bucket over fixed window?
Token bucket supports bounded bursts while controlling the average rate. Fixed window is simpler but can admit a boundary burst across adjacent windows.
How do you enforce limits across multiple servers?
Use atomic shared state, partitioned ownership, or local approximations with a documented consistency trade-off. The right choice depends on strictness and latency.
Should the limiter fail open or closed?
It depends on the protected resource and risk: fail-open preserves availability but may expose capacity; fail-closed protects the service but can reject legitimate traffic during a dependency outage.
SAY IT OUT LOUD
How to explain this in an interview
“I would define the client identity and burst policy first, then place the limiter at an API gateway or service boundary. A token bucket is a good choice when clients may burst but should stay within an average rate. In a multi-instance service, the token update needs atomic shared state or a consciously approximate design. Rejected calls should receive a 429 response, and the system needs an explicit fail-open or fail-closed policy.”
Connect this question to the roadmap topic it supports.