Fail-open auth, or when a missing environment variable disables your security

Our proxy's API-key enforcement was governed by an environment variable, and the launch-era logic read like this:

ENFORCE_API_KEYS === 'true'   → reject requests without a valid key
anything else                 → let everything through

"Anything else" is doing an enormous amount of work in that second line. Unset: auth off. Empty string: auth off. True with a capital T, enforce, a typo, a deploy that dropped the variable, a new environment that never had it: auth off. Every failure mode of configuration management — and configuration management is nothing but failure modes — resolved to silently disable security. Nothing logged it, nothing alerted, and the proxy behaved identically from the outside either way, right up until someone noticed traffic that should have been rejected being served.

A URL path segment is not a credential, which is why the flag exists at all. The proxy's routing puts a user id in the path — /{userId}/{provider}/... — and user ids leak: they sit in dashboard URLs, in shell history, in screenshots. Before key enforcement, anyone who learned one could log traffic into that account, burn its plan limits and pollute its dashboard. The Bearer key is the identity; the path is an address. Enforcement is what makes that sentence true, and the flag decided whether enforcement ran.

The default was not stupid; that is the instructive part. At launch, observe-only was correct — you introduce keys, watch adoption, and only then start rejecting, because enforcement before adoption bricks every existing integration. The defect was never the posture. It was that the failure path and the chosen posture were the same state. A config mistake produced a valid-looking configuration, and "we chose not to enforce" became indistinguishable from "the flag fell out of the deploy".

The fix: make every state explicit, and fail closed without bricking#

The flag is now required and two-valued, with the failure path pointing the other way:

'true'  / '1'   → enforce
'false' / '0'   → observe-only (the explicit opt-out)
anything else   → ENFORCE, and log the misconfiguration loudly

Choosing not to enforce is still available — as an explicit, spelled-out state someone had to type. What no longer exists is a state where absence chooses. And note what "fail closed" means here, because the blast radius was chosen deliberately: a misconfigured flag does not brick the Worker or reject everyone — a keyless request under a broken flag gets the same well-formed 401 envelope it would get under 'true', while customers with valid keys are untouched. The worst case of a lost variable is "unauthenticated traffic is refused", never "all traffic breaks", and never — the old world — "everything is silently allowed". One more Workers-specific wrinkle: there is no startup hook to warn from, so "loud at startup" becomes loud on the first request each isolate serves, once per isolate — a busy Worker must not drown its own logs in its own warning.

The second inversion: a failed lookup is not a failed key#

The audit found a subtler cousin. Verifying a key means a database lookup, and the original code treated a failed lookup — Supabase down, network blip — like a failed key. Under enforcement that meant a transient outage answered 401 invalid key to customers whose keys were fine.

Trace what a 401 does downstream: the SDK treats it as permanent and does not retry; the developer reads "invalid key", concludes their credential is broken, and starts rotating healthy keys in the middle of our outage — destructive action, on their side, taken because our error told them to. The fix is that "we could not tell" is its own answer: a definitive bad key gets a 401; an inconclusive lookup under enforcement gets a retryable 503 that says plainly the verification backend did not answer, the request was not forwarded, and the caller's keys are fine. An error message is an instruction the caller will follow. Status codes are the API for that instruction, and sending "your credential is wrong" for "our database was briefly down" is an instruction to break something that works. Two failure directions, two different answers — the same argued-per-case reasoning as the cache-minimum table that failed closed in the wrong direction, and the same insufficient-data honesty as the dash, applied to auth instead of money.

Two design notes from the same work, for anyone building similar: the credential is self-identifying (a vgl_ prefix), because on OpenAI-shaped traffic the Authorization header already carries the provider's key — a proxy credential must be distinguishable from a provider credential sharing the same header, and must be stripped before forwarding while the provider's is preserved. And only the key's hash is ever stored or queried; the raw key is shown once at creation. Plain SHA-256 is right here precisely because API keys are high-entropy random strings — a slow KDF defends against dictionary attacks passwords face and keys do not, and would add latency to every proxied call for nothing.