"Design an API gateway" sounds like plumbing, but interviewers use it to see whether you understand cross-cutting concerns: what belongs at the edge and what belongs in each service.
Functional: route requests to the right backend service, terminate TLS, authenticate callers, apply rate limits, and transform or aggregate responses when needed.
Non-functional: very low added latency (single-digit milliseconds), high availability, horizontal scaling, and safe config changes without restarts.
The load balancer sends the request to any stateless gateway node.
The node matches the path and host against a routing table held in memory.
Middleware runs in order: authentication (validate a JWT or API key), rate limiting (token bucket backed by a shared store such as Redis), request validation.
The request goes to a healthy instance found through service discovery, with a timeout and a retry budget.
The response passes back through logging, metrics, and optional response shaping.
Config distribution: routes live in a config store; nodes watch it and swap the in-memory table atomically, so a bad push can be rolled back fast.
Rate limiting accuracy vs latency: a local counter per node is fast but loose; a central Redis counter is accurate but adds a hop. A common compromise is local buckets that sync periodically.
Failure isolation: per-backend timeouts, circuit breakers and bulkheads stop one slow service from exhausting gateway threads.
Keep it thin: business logic in the gateway becomes a deployment bottleneck. Aggregation for mobile clients can live in a separate backend-for-frontend layer.
Stateless nodes scale linearly; the shared rate-limit store and the auth key cache are the usual hot spots, so cache public keys locally and shard counters by API key.
Draw the request path first, then hang each concern on it. It keeps the discussion structured when the interviewer jumps between topics.
If you want to practise talking through designs like this out loud, Linkjob.ai can help. Disclosure: I work on Linkjob AI.
Further reading: Linkjob AI resource directory