Rate Limiter System Design (Token Bucket)

A rate limiter caps how many requests a client can make in a window. The three algorithms you need to know are token bucket, sliding window, and fixed window. Here's how each works, when to use it, and a token-bucket implementation.

Backend Engineerbackendrate-limitingsystem-design

What a rate limiter does

A rate limiter protects a service by rejecting requests from a client once they exceed an allowed rate (e.g. "100 requests per minute"). It's the answer to abuse, runaway scripts, and fair-use quotas - and a staple of system design interviews.

The three algorithms

Algorithm	How it works	Trade-off
Token bucket	A bucket holds up to N tokens, refilled at a steady rate. Each request spends one token.	Allows short bursts up to capacity. Feels nice to clients. The usual default.
Fixed window	Count requests per fixed clock window (e.g. per minute).	Simple, but allows a 2x burst at the window boundary.
Sliding window	Track timestamps of recent requests (or a weighted count).	Smooth + accurate, but more memory/compute.

Token bucket implementation

import time

CAPACITY = 10          # max burst
REFILL_PER_SEC = 10.0  # steady-state rate

_buckets = {}  # client_id -> (tokens, last_ts)

def allow(client_id: str) -> bool:
    now = time.monotonic()
    tokens, last = _buckets.get(client_id, (CAPACITY, now))
    tokens = min(CAPACITY, tokens + (now - last) * REFILL_PER_SEC)  # refill
    if tokens >= 1:
        _buckets[client_id] = (tokens - 1, now)
        return True
    _buckets[client_id] = (tokens, now)
    return False

The key idea: you don't run a timer to add tokens. You lazily refill on each request based on elapsed time - cheap and exact.

Designing it for real

Distributed: an in-memory dict only works on one node. For a fleet, store the bucket in Redis (atomic via a Lua script or INCR + TTL) so all nodes share state.
Identify the client: API key, user id, or IP - pick the key you actually want to limit on.
Respond clearly: return 429 Too Many Requests with a Retry-After header so clients back off politely.
Per-route limits: expensive endpoints get tighter limits than cheap ones.

Which to pick

Default to token bucket for API gateways - it absorbs bursts gracefully. Use sliding window when you need strict, smooth fairness and can afford the bookkeeping. Avoid plain fixed window when boundary bursts matter.

Want to try it hands-on? HeyDevJob gives you this exact setup in a live cloud workspace in your browser - edit it, run it, and see it work. Free, nothing to install.

Try it in a workspace →

What you'll practice

Implementing a token-bucket limiter with lazy refill
Returning 429 + Retry-After correctly
Reasoning about burst vs steady-rate trade-offs across the three algorithms

FAQ

What is the best rate limiting algorithm?

Token bucket is the usual default - it allows short bursts up to a capacity while enforcing a steady average rate, which feels fair to clients. Use a sliding window when you need stricter, smoother fairness.

How do you rate limit across multiple servers?

Keep the counter/bucket in a shared store like Redis and update it atomically (e.g. a Lua script or INCR with a TTL), so every node enforces the same limit instead of each tracking its own.

What status code should a rate limiter return?

429 Too Many Requests, ideally with a Retry-After header telling the client how long to wait before retrying.

Keep learning

Return the Correct HTTP Status CodeBackend project Repair Broken API PaginationBackend project Add CORS Middleware to an Express APIBackend project Backend roadmapStep by step to hired Backend interview questionsSTAR answers All Backend projectsProjects hub

Learn it by doing. Open this in a live cloud workspace, make the change yourself, and keep a record of the work you can share.

Open the workspace →