Fail open or fail closed? What your rate limiter does when Redis dies

At 02:14 on a Tuesday, a Redis failover took eleven seconds. Not a catastrophe. The primary went away, a replica got promoted, and eleven seconds later everything was fine again.

Except the API returned 500s for four minutes.

The rate limiter sat in front of every route. When it could not reach Redis it threw, the error bubbled up through the middleware, and every request died with it. Nobody decided that should happen. It is just what the code did when nothing told it otherwise.

That is the thing about rate limiter fail open vs fail closed: almost every team has a policy, and almost none of them chose it. You inherited the default of whatever library you installed, and you will find out what that default is during an incident.

So let's actually choose.

Fail open vs fail closed: the two modes every rate limiter has

The vocabulary is borrowed from door locks and it is a good metaphor. When the power cuts, a fail-open door unlocks so people can get out. A fail-closed door stays locked so people cannot get in.

For a rate limiter, when the counter store is unreachable:

• Fail open means allow the request. Traffic keeps flowing without enforcement.
• Fail closed means reject the request. Enforcement holds, service does not.

Search this topic and you will get a fairly confident consensus: fail open, because a limiter outage should not become an API outage. It is reasonable advice and it is right more often than it is wrong.

It is also incomplete in three specific ways, and each one has bitten real production systems:

1. It assumes the answer is the same for every route. It is not.
2. It assumes your store fails by going down. Usually it fails by going slow, which your fail-open branch never catches.
3. It assumes the limiter failed for reasons unrelated to your traffic. Sometimes your traffic is what killed it.

Let's take them in order.

What is your limiter actually protecting?

Here is the question that settles the policy, and it is not a question about availability. It is a question about what the limiter is for. A limiter guarding a login endpoint and a limiter guarding a search endpoint are different machines wearing the same uniform.

What the limiter guards	Example routes	Cost of failing open	Cost of failing closed	Sane default
Abuse and security	Login, password reset, OTP and SMS sends, signup, coupon redemption	Unbounded credential stuffing, an SMS bill you will remember, fraudulent signups at machine speed	Those routes are unavailable for the duration	Closed
Capacity and fairness	Search, list endpoints, read APIs, report generation	One noisy client can crowd out the others until the store recovers	Your entire read path is down, which is a bigger outage than the one you are having	Open
Revenue and metering	Anything billed per call or drawn from a credit balance	You give away product you sell, with no record of it	You refuse service to customers who already paid	Neither. See below.

The security row is where the popular advice does the most damage. An unavailable login endpoint is embarrassing and recoverable. You will be back in minutes and your users will retry. An unlimited login endpoint is a credential-stuffing window, and there is no rolling back the attempts that succeeded. Those two outcomes are not on the same scale, so they should not get the same default.

The rule: fail open where the cost of the failure is reversible, fail closed where it is not. Availability is a good tiebreaker, not a first principle.

"Can't I just fail open everywhere and watch the alerts?"

You can, and for a read-heavy internal API that is a perfectly defensible call. The problem is the response time of the human in that loop. Your alert fires, someone acknowledges it, someone opens a laptop. Call it minutes if your on-call is sharp. An automated attacker measures its opportunity in requests per second, and it does not need your permission to use all of them. If the fail-open window is minutes wide, "watch the alerts" is not a control. It is a notification that the thing already happened.

"Down" is the easy failure. "Slow" is the one that gets you.

Here is the part almost nobody tests, and it is the reason that Tuesday incident lasted four minutes instead of eleven seconds.

You write your fail-open branch. You test it the obvious way: docker stop redis, fire a request, watch it sail through, feel good. That test passes because a hard-down Redis refuses the TCP connection in microseconds. Your client throws immediately, your catch block runs, the policy works exactly as designed.

Production rarely gives you that. Production gives you a Redis that is reachable and slow: mid-failover, stalled on an fsync, swapping because it crossed maxmemory, or blocked behind someone's KEYS * in a console. There is no error. There is no exception. There is just latency, and your fail-open branch sits there unexecuted while every request waits.

Work it through. Say your limiter runs on a pool of 200 request handlers and Redis starts taking 5 seconds to answer. At 100 requests per second of incoming traffic, your entire pool is occupied inside two seconds and everything behind it queues. You configured fail open. You are having a total outage. Nothing about your policy was wrong, it just never got the chance to run.

This is not hypothetical. Kong shipped a fault_tolerant flag for exactly this purpose, and users still filed an issue where an unreachable Redis produced 500s anyway: the connection timed out, and the code then tried arithmetic on a nil value it never expected to have. The issue is old and closed, and every gateway has a version of this scar. The lesson is not "Kong is bad", it is that a fail-open setting and a fail-open code path that actually runs are two different things.

A fail-open policy with a default client timeout is not a fail-open policy. It is a slower outage.

So set the timeout explicitly, and set it against your latency budget rather than against Redis's feelings:

const Redis = require('ioredis')

const redis = new Redis({
  host: process.env.REDIS_HOST,
  port: Number(process.env.REDIS_PORT || 6379),

  // Give up dialing fast. The default is 10s, which is ten seconds
  // of holding a request handler hostage.
  connectTimeout: 200,

  // The one that matters: abandon a command that hangs. There is no
  // default here, so without it a slow-but-alive Redis is waited on
  // indefinitely and your fallback never triggers.
  commandTimeout: 50,

  // Don't spend a request's budget on retries. Fail and move on.
  maxRetriesPerRequest: 1,

  // Critical: while disconnected, don't silently buffer commands
  // and resolve them later. Reject now so the fallback runs now.
  enableOfflineQueue: false,
})

redis.on('error', (err) => {
  // Log, don't throw. An unhandled 'error' event will take the process down.
  console.warn('redis:', err.message)
})

Pick commandTimeout by asking what you can afford. If your p99 API latency budget is 300ms and the limiter is allowed 15% of it, you get roughly 45ms. A limiter that answers in 50ms or not at all is a limiter you can reason about. A limiter that might take 5 seconds is a load-bearing outage waiting for a bad night.

The correlated failure nobody plans for

Fail open rests on an assumption worth saying out loud: the limiter failed for reasons unrelated to the traffic it was limiting. A bad deploy, a network partition, a full disk. Under that assumption, letting traffic through is obviously right, because the traffic was never the problem.

Now consider the other case. Traffic spikes. Connection counts climb, the counter store gets hot, memory pressure builds, and Redis falls over because of the surge. Your limiter fails. Your policy says fail open. You have just removed the brake from a car that is already going too fast, at the precise moment you built the brake for.

The arithmetic is worth doing with your own numbers, because it is usually worse than intuition suggests. Suppose 500 clients are each capped at 10 requests per second, so your ceiling is 5,000 rps and your database is provisioned comfortably above that. During a fail-open window, the ceiling is no longer 5,000. It is whatever those clients want to send. If a fraction of them are batch jobs that will happily push 200 rps each, thirty clients alone can put 6,000 rps on a database that has never seen it. Add client-side retries, which fire exactly when things are slow, and offered load can multiply again while you watch.

How wide is my fail-open window, really?

Longer than the outage, which surprises people. The window is detection time plus your retry and circuit-breaker cooldown plus recovery. If your breaker stays open for 30 seconds after the store comes back, that is 30 more seconds of unlimited traffic hitting a database that just finished being overwhelmed. Size the cooldown deliberately, and consider having recovery ramp back into enforcement rather than snapping.

Stop choosing. Build a ladder.

The framing of "open or closed" is a trap, because it presents a binary at the exact moment you have the most options. A limiter that cannot reach its shared store is not out of information. It still knows the configured limit, and it still knows what this process has seen. That is enough to be useful.

Four rungs, in order:

1. Ask the shared store, with a tight explicit timeout. Covered above. This is the rung most people skip.
2. Trip a circuit breaker. After N consecutive failures, stop asking for a cooldown period. Without this, every single request pays the full timeout, and your "fast failure" is 50ms of dead weight per request forever.
3. Fall back to a local counter. Each worker enforces limit / worker_count on its own, in memory. Imprecise, and dramatically better than nothing.
4. Only now, apply the policy. Open or closed, per route. By this point it governs a much narrower set of cases.

Here is the whole thing. It is deliberately a plain fixed-window counter of the kind Redis documents, so the failure handling stays readable. If you want the atomic sliding-window version, that is a whole post of its own and we wrote it: building an atomic sliding-window rate limiter in Redis and Lua.

const WORKERS = Number(process.env.WORKER_COUNT || 1)
const FAILURE_THRESHOLD = 5
const COOLDOWN_MS = 10_000

const breaker = { failures: 0, openUntil: 0 }
const local = new Map()

// Rung 3: per-worker share of the global limit.
function localAllow(id, limit, windowMs) {
  const share = Math.max(1, Math.floor(limit / WORKERS))
  const now = Date.now()
  const bucket = local.get(id)

  if (!bucket || now >= bucket.resetAt) {
    local.set(id, { count: 1, resetAt: now + windowMs })
    return { allowed: true, source: 'local' }
  }

  bucket.count += 1
  return { allowed: bucket.count <= share, source: 'local' }
}

// degraded: 'local' | 'open' | 'closed'  -- set it per route.
async function allow(id, limit, windowMs, { degraded = 'local' } = {}) {
  const now = Date.now()

  // Rung 2: breaker is open, don't even try.
  if (now < breaker.openUntil) {
    return degraded === 'local'
      ? localAllow(id, limit, windowMs)
      : { allowed: degraded === 'open', source: 'policy' }
  }

  try {
    // Rung 1: the authoritative path.
    const count = await redis.incr(`rl:${id}`)
    if (count === 1) await redis.pexpire(`rl:${id}`, windowMs)

    breaker.failures = 0
    return { allowed: count <= limit, source: 'redis' }
  } catch (err) {
    breaker.failures += 1

    if (breaker.failures === FAILURE_THRESHOLD) {
      breaker.openUntil = now + COOLDOWN_MS
      console.warn(`limiter: breaker open ${COOLDOWN_MS}ms (${err.message})`)
    }

    // Rung 4: degrade, don't collapse.
    return degraded === 'local'
      ? localAllow(id, limit, windowMs)
      : { allowed: degraded === 'open', source: 'policy' }
  }
}

Wire the policy where the route is defined, so it is a visible decision instead of a global default nobody remembers making:

// Irreversible damage if unlimited. Refuse rather than guess.
app.post('/login', limit({ limit: 5, windowMs: 60_000, degraded: 'closed' }))

// Protects the database. Stay up, enforce roughly.
app.get('/search', limit({ limit: 100, windowMs: 60_000, degraded: 'local' }))

// Internal, low stakes, availability wins outright.
app.get('/status', limit({ limit: 1000, windowMs: 60_000, degraded: 'open' }))

During a Redis stall, the log should read like this: a short burst of timeouts, one breaker message, then silence while the local counters carry the load.

redis: Command timed out
redis: Command timed out
redis: Command timed out
redis: Command timed out
redis: Command timed out
limiter: breaker open 10000ms (Command timed out)

Be honest about what rung 3 costs you. limit / WORKERS is only correct if your load balancer spreads traffic evenly, which it does not, and if WORKERS is accurate, which autoscaling makes a moving target. A client unlucky enough to land repeatedly on one worker gets throttled early. During a store outage, "some clients see limits tighter than advertised" is a trade most teams will take over both alternatives. Just do not describe it as exact, least of all in your public docs.

If you want the deeper argument for why concurrency is often a better control than request rate in the first place, Jon Moore's Strange Loop talk is the one to watch. It reframes the whole problem through queueing theory and Little's Law:

The case both modes get wrong: metered billing

Now the row from that table nobody writes about. If your limiter is not just protecting capacity but also counting things you charge for, both policies are wrong and no amount of tuning fixes it.

Fail open and you serve requests you never recorded. That is not a rounding error, that is product you sell, given away, with no audit trail. Fail closed and you return errors to customers who paid you in advance. One of those shows up in revenue, the other in your inbox.

The way out is to stop conflating two jobs that happen to share a code path:

• Enforcement decides whether this request proceeds. It must be immediate, and it can be approximate.
• Accounting decides what the customer owes. It must be accurate, and it can be late.

Once you separate them, a third mode appears, and it is the right one: serve and settle. Admit the request, write a durable usage event locally, and reconcile when the store comes back.

Three things make it work, and all three are load-bearing:

1. The usage event must be durable before you respond. An append to a local log or a queue, not an in-memory array that dies with the process. If the event is not durable, you have simply failed open with extra steps.
2. Every event needs an idempotency key. Reconciliation will replay, and a request id keeps a retried batch from double-billing a customer for one call.
3. Overspend during the outage is a policy decision, so decide it in advance. A customer who blew past their quota while you could not enforce it can be forgiven, billed, or capped, and there is no universally correct answer. There is a universally wrong one: deciding case by case, during the incident, in a support thread.

This is genuinely more work than a boolean, which is the honest reason most homegrown limiters never do it and most usage-based APIs quietly lose a little money every time their counter store hiccups. It is also the seam where "we'll just use Redis" stops being cheaper than buying the layer. If you are deciding where quota should live in the first place, we went through that in where a multi-tenant quota actually belongs.

What changes if you don't run the store yourself

Fair warning, this is the part where we talk about our own product. Here is the honest version: using a managed key and quota layer does not delete this decision. It moves it one hop. You are no longer choosing what to do when Redis is unreachable, you are choosing what your service does when the validation call is unreachable, and that code is still yours to write.

What it does change is which failures you own. Store failover, replication, per-region enforcement, and keeping counters consistent stop being your pager's problem. The ladder above collapses to rungs 1 and 4: set a tight timeout on the call, decide your degraded behavior per route.

For the specifics, ReqKey puts rate limits on the consumer rather than the key. A consumer's rateLimit is { "limit": 100, "window": 60 }, meaning 100 validations per 60 seconds, enforced as a sliding window and shared by every key that consumer owns. There are no key-level rate limits. Credits and rate limits are independent axes: credits meter how much, the rate limit meters how fast, and a consumer with unlimited credits can still be throttled.

Two documented behaviors matter for the failure conversation. First, a throttled request is free: the docs state that a rate-limited request consumes no credits and no rate-limit quota, so a client recovers as soon as it slows down. Second, the status codes mean genuinely different things, and the docs are blunt that 402 and 403 are not outages. Retrying them will not help until the underlying problem is fixed.

let res
try {
  res = await fetch('https://api.reqkey.com/key/validate', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.REQKEY_ROOT_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ key: customerKey, credits: 1 }),
    signal: AbortSignal.timeout(300), // your budget, not their mood
  })
} catch (err) {
  // Unreachable or too slow. THIS is your fail policy, and it is your code.
  return applyDegradedPolicy(route) // 'open' | 'closed' | 'serve and settle'
}

if (res.status === 429) {
  // Too fast. Retrying later works.
  const retryAfter = res.headers.get('Retry-After')
  return reply(429, { retryAfter })
}

if (res.status === 402) return reply(402, { error: 'Out of credits' })
if (res.status === 403) return reply(403, { error: 'Key or consumer disabled' })
if (res.status >= 500) return applyDegradedPolicy(route)

const data = await res.json()
if (!data.valid) return reply(401, { error: 'Invalid key' })

// data.creditsRemaining / data.creditsLimit are null for unlimited consumers.
return proceed(data)

Note the branch order: /key/validate returns HTTP 200 with { "valid": false } for an unknown key, so status alone is not enough. Always read the valid field.

And an honest limitation, since a comparison that only lists strengths is marketing: rate-limit windows are enforced per region. End users are pinned to one region so they experience the configured limit, but a client deliberately spraying requests across regions can reach up to limit × regions. If you are limiting for abuse prevention rather than fairness, that is a real property to design around, and you should know it before you pick a tool rather than after.

Key takeaways

• Pick the policy from what the limiter protects, not from a blog post's default. Reversible damage (a crowded database) argues for fail open. Irreversible damage (credential stuffing, an SMS bill, fraudulent signups) argues for fail closed. Go set /login to closed today.
• Set an explicit command timeout or your fail-open branch is decoration. Stores usually fail by getting slow, not by refusing connections, and a default multi-second timeout means every handler blocks before your policy ever runs. Budget the limiter a small slice of your p99 and enforce it in the client config.
• Add a circuit breaker or you pay the timeout on every single request. Fast failure repeated a million times is just a slower system. Stop asking after N failures, and size the cooldown knowing it extends your unenforced window past the outage itself.
• Degrade before you decide. A per-worker counter of limit / workers is imprecise and far better than both alternatives. Reserve the raw open/closed choice for the routes where approximate enforcement is genuinely meaningless.
• If you meter billing, split enforcement from accounting. Enforcement can be approximate and must be immediate. Accounting must be accurate and can be late. Serve the request, durably record the usage event with an idempotency key, reconcile after, and decide your overspend policy while nothing is on fire.

Test the failure, not just the feature

The single most useful thing you can do after reading this is not a config change. It is an experiment: put your limiter under load, make its store slow rather than dead (a proxy that delays packets works fine), and watch what your API actually does. Most teams discover their fail-open policy never fires. Better to find that out on a Wednesday afternoon than at 02:14.

If you would rather not own that failure mode at all, that is roughly the pitch for ReqKey: consumer-level rate limits and credit metering behind one validation call, with throttled requests costing nothing. The free tier includes 100,000 requests a month, which is enough headroom to run the slow-store experiment above against a real integration and see exactly how your service behaves before it matters. Your degraded-path code is still yours to write. At least this way it is the only part you have to get right.