Design a rate limiter

Design a server-side rate limiter for a large API platform. It should limit requests per user/IP/API key according to configurable rules, across many API servers.

1. Clarify the requirements

Before drawing anything, ask questions. These are the answers an interviewer would typically give:

Client or server side?

Server side (API gateway or middleware).

What do we limit by?

By user ID, IP or API key, with flexible rules per endpoint.

Scale?

1 million requests per second across 200 API servers.

Latency budget?

It must add less than ~1 ms to each request.

What if the limiter fails?

Requests must still be served (fail open).

Tell the user?

Yes: rejected requests return an error and headers explaining the limit.

2. Estimate the scale

Assumptions: 1M requests/second; each counter entry ~100 bytes; 10M active keys.

• Memory for counters (10M keys × 100 B): about 1 GB10M × 100 B = 1 GB: fits comfortably in a Redis cluster.

3. The high-level design

1. Clients and your APIWithout a limiter, requests go straight to the API servers, and one runaway client can take them down.
2. A rate limiter in frontEvery request passes through rate-limiting middleware first. Over the limit, the client gets HTTP 429 with a Retry-After header instead of reaching your servers.
3. Shared counters in RedisEvery limiter instance must see the same counts, so counters live in Redis: a sliding-window counter per user or IP, updated atomically with INCR + EXPIRE (or a Lua script).
4. The rest of the systemBehind the limiter, a normal web stack: cache in front of the database.
5. Watch it in productionMetrics and alerts close the loop: 429 rate per rule, Redis latency, and requests that slip through.

4. The components and what each one does

Load balancer Load balancer

Spreads allowed requests over stateless API servers.

API servers App servers

Stateless: any server can handle any request, so you scale out horizontally. Checks the auth token on every request.

Rate limiter Rate limiter

Middleware in front of the API. Rules (like 5 posts a minute per user) are loaded from config files into memory. Over the limit → 429 Too Many Requests.

Redis counters Cache · Redis

Sliding-window counters per user/IP, atomic INCR + EXPIRE in a Lua script so two limiters can't both let a request through. Sharded by user ID across a Redis cluster.

App cache Cache

Hot reads served from memory.

App database SQL database

Primary + read replicas with failover; sharded by user ID as it grows.

Metrics & alerts Metrics & alerts

Dashboards and alerts on p99 latency, error rate, queue depth and cache hit ratio, so a bottleneck shows up before it becomes an outage.

5. Deep dive: algorithm & distributed counting

Which algorithm do you use and why? How do 200 servers share counters without race conditions?

• Token bucket allows bursts; sliding window counter is accurate and memory-cheap.
• Keep counters in Redis; use atomic operations or a Lua script to avoid race conditions.
• Sync issues: don't rely on sticky sessions; use the central store.

6. What interviewers listen for

• A specific algorithm, with its trade-offName the algorithm and why it fits (bursts allowed? memory? accuracy?).
• Race conditions handled (atomic ops / Lua / locks)Two servers doing read-then-write on a counter can both let a request through.
• Rules stored and cachedRules live in config files/DB and are cached by limiter workers.
• 429 response with rate limit headersClients need to know they were limited and when to retry.
• Fail open if the limiter store is downIf Redis is down, it's usually better to let traffic through than reject everyone.
• Multi-region: local counters, eventual syncGlobal limits with low latency use local counters synchronized eventually.
• Database replication and failoverA single database is a single point of failure. Replicas give redundancy and read capacity; failover promotes a replica when the primary dies.
• Partition (shard) data as it growsOne machine eventually can't hold the data or the write load. Say how you'd split it and by which key.

7. Follow-up questions

1. The Redis cluster holding your counters becomes unreachable. What does the API do?
   • Fail open (allow requests) to keep the API up
   • Fall back to local in-memory limits
   • Alert and replicate Redis
2. With a fixed 1-minute window of 100 requests, a client sends 100 at 0:59 and 100 at 1:01. What happens and how do you fix it?
   • 200 requests in 2 seconds pass: the window-edge problem
   • Use a sliding window (log or counter) or token bucket

The design in a nutshell

• Clients → API gateway/middleware with the rate limiter → API servers.
• Counters in Redis (INCR + EXPIRE, or a Lua script for atomic token bucket / sliding window counter), keyed by user/IP/API key + rule.
• Rules in config, cached by workers; 429 + X-Ratelimit headers; fail open on store failure; monitor rejected rate.

Related lessons: Algorithm 1: Token bucket · Algorithm 5: Sliding window counter · Rate limiting in a distributed environment