Nine metric definitions that were quietly wrong

A metric has two parts: the number, and the sentence next to it saying what the number is. The arithmetic can be perfect while the sentence is false, and that combination is worse than a broken calculation — a broken calculation looks broken, while a wrong definition looks like a fact. We audited every figure on our own dashboard against what its code actually computes. Nine definitions failed. Here they are, worst first.

1. "Banked so far · actually saved since you turned it on." The number under that label was the savings on eligible calls inside a 30-day window, summed over agents optimisation is switched on for right now. Three lies in one sub-line: an agent enabled 60 days ago silently lost its first 30 days; switching one agent off removed its history from the total; and so a "cumulative balance" could decrease with no caching loss anywhere. A rolling window of currently-enabled agents wearing a lifetime label. This is the defect class to fear most, because every individual number it ever showed was correctly computed — only the definition was wrong, and no unit test on the arithmetic could catch it. The fix made the copy a pure function tested against the constant that sets the window, so the sentence cannot drift from the sum.

2. Unpriced spend reported as $0.00. The proxy fails closed on a model it has no verified rate for: cost_usd is written as NULL, because zero would be a claim and null is an absence. That contract held all the way to the database and died there — every aggregate wrapped the column in coalesce(sum(cost_usd), 0), and Postgres sum() skips NULLs. Route traffic to a model that shipped after the rate table's last verification and the dashboard reported a confident $0.00 for real spend. The invariant that replaced it: a total that omits rows must state its denominator, and past a threshold of unpriced share it stops reporting a number at all. The full argument is in why our dashboard shows a dash instead of $0.00.

3. A 10x cost blowout rendered green. The trend tile derived its colour from its direction: up-arrow, green. On a cost dashboard, a bill going from $100 to $1,000 displayed "+900.0%" in cheerful green. Direction and sentiment are different axes, and each metric now declares which direction is good news.

4. The model comparison contradicted itself. The card computed which model was cheaper correctly, then derived the adjective from the same sign a second time — inverting it whenever the winner was the second operand. With model A at $0.12 and model B at $0.10, it read "B is 20% more expensive", naming the cheaper model and then insulting it, on the page whose only purpose is choosing between models.

5. Nav badges counted from one page of results. The error badges in the sidebar were computed by filtering a LIMIT-50 query ordered by recency. On a busy account, all fifty rows were recent findings of one type, every other badge computed to zero, and the more traffic an account had, the more reliably its always-visible navigation lied. Counts now come from an exact aggregate, never from a page.

6. "12 failures" directly above "Nothing has failed." Same shape, adjacent elements: the header count came from an exact RPC, the section body from the LIMIT-50 page, and when the fifty most recent rows happened to contain no failures the page rendered a red 12 badge over a green all-clear tick. Two sources of truth on one screen will eventually disagree on that screen.

7. One-hour cache writes priced at the five-minute multiplier. Anthropic bills a 1-hour cache write at 2x base input and a 5-minute write at 1.25x. The cost function had a parameter for the 1h count; no live call site ever passed it, so every 1-hour write was priced at 1.25x — a 37.5% understatement on that line, on the tier the proxy itself requested by default. The fix carries the two tiers as separate named values so that summing them wrongly is unwriteable, and refuses to price at all when the response withholds the split on a call where a 1h write was possible. Details in the write-tier post.

8. Every Opus call priced 3x high. A substring fallback — includes('opus') → $15/$75 — written when Opus meant claude-3-opus, silently priced every current Opus model at three times its real $5/$25 rate. A guess against a moving model catalogue is a liability wearing a convenience costume; the fallback is deleted and an unknown model now prices to null.

9. An agent written off as "too small to cache" that was eligible all along. The per-model cache minimums table sat 2–8x too high on five current models after an over-correction in the safe-looking direction. One agent measured 1,415 stable tokens against a stored minimum of 2,048 and was reported ineligible — on a model whose real published minimum is 1,024. A minimum set too high produces no visible error, just a false verdict that looks exactly like a true one. "Fail closed" needs its direction argued per case, not assumed.

The common thread: not one of these was an arithmetic bug. Every one was a definition — a label, a source, a multiplier choice, a fallback — and definitions are where dashboards lie, because nothing type-checks a sentence.