My test report printed "0/96, 0% pass rate". The truth was my account was out of credit

Something you can do right now

Open any script you have that prints a score — an eval, a CI check, an audit tool, a health check — and ask it one question:

If the infrastructure fails (out of credit, expired key, rate limited, no network), what does this script print?

If the answer is a score, a ratio, or a pass count, you own a false-red-light generator.

Not "might go wrong." Structurally guaranteed — because it put "couldn't measure" and "didn't pass" in the same box.

It took me a full day to see this, and I had to see it twice, on two different scripts.

Where it started: I'd just finished something I was confident about

I develop with Claude Code and had accumulated 89 custom skills (think "specialised tools for the AI" — each has a name and a self-description). Those descriptions get packed into the model's opening memory, and that memory has a character limit. Go over it and descriptions get dropped, names kept — and that tool never surfaces on its own again.

I squeezed 24 of those descriptions shorter, saving 2,754 characters, retiring nothing. (The full write-up of that, with the A/B test: I cut 41 AI tools' self-descriptions in half.)

The scary part of trimming is cutting a trigger word: that description is the basis on which the model decides whether to invoke the skill, so cutting the wrong phrase makes it silently stop firing, with no error. So I wrote a reconciliation script that lists every token present in the old version and absent in the new, forcing me to judge them one by one. Eight had genuinely lost trigger words. All restored.

Then I wanted harder evidence: behavioral tests. I already had a question bank — 4 cases per skill (should-trigger, strict, should-not-trigger, boundary), 96 total.

I had believed that bank didn't exist. I'd even written in a handoff doc: "most of these 24 have no test cases, so 'will it still trigger?' can only be answered by token reconciliation."

That sentence was wrong. All 24 had cases.

What caught the wrong sentence wasn't me re-reading it. It was running a full audit before pushing, and noticing that one check had gone from "pass" to "blind", and had taken 100 seconds. It had been triggered into actually running by my 24 changed skills. I chased that state change, and found the question bank had been there all along.

A state change in an audit item is itself a signal — often more informative than its green light. pass→blind, fast→slow, warning count 9→10 — every one of those deserves a "why did that change?" If I hadn't chased it, I'd have kept a false sentence ("there's no bank to test against") while the bank sat right there.

Hole #1: an empty wallet, rendered as a 0% pass rate

So I ran the 96 tests.

┌─────────────────────────────────────────────────────────┐
│              Skill Eval Report (advisory)               │
├─────────────────────────────────────────────┬───────────┤
│ ❌ action-gating-surface-disclosure            │  0/4 (0%) │
│ ❌ architecture-completeness-guardian          │  0/4 (0%) │
│ ❌ audit-cross-repo                            │  0/4 (0%) │
                        ⋮  (all 24 like this)
├─────────────────────────────────────────────┼───────────┤
│   TOTAL                                       │ 0/96 (0%) │
└─────────────────────────────────────────────┴───────────┘

⚠️  The following skills scored < 50% — consider reviewing:
   (all 24 listed)

If I only read that table, the conclusion is unambiguous: I just broke all 24 tools and should roll back immediately.

Scroll down a few hundred lines and every single case's raw output is the same sentence:

ERROR: Anthropic API error 400: {"type":"error","error":{"type":"invalid_request_error",
"message":"Your credit balance is too low to access the Anthropic API..."}}

Zero successful API calls. Total spend: US$0.0000.

An empty wallet, rendered as "none of these 24 tools trigger any more."

The root cause is one line. The per-case runner looked like this:

try {
  const r = await evalCase(c, systemPrompt, skillName)
  runs.push(r)
} catch (err) {
  runs.push({ pass: false, raw: `ERROR: ${err.message}` })   // ← here
}

Any exception is recorded as pass: false. Network down, expired key, empty account, a bug in my own code — all of it becomes "this skill did not pass."

And the harder half to notice: **that pass=0/96 line was already printing in the round before my changes.** The false red light had been there a long time. Nobody read it — it's advisory (doesn't block the push), so every audit round printed one line and every round skipped past it.

The fix: "couldn't measure" needs its own box

My system already had a convention: a check result is three states, not two —

• 🟢 I looked, it's clean
• 🛑 I looked, I found something
• ⚠️ I was unable to look (blind)

Blind means "this round's green light doesn't count", not "there's a problem." I'd written that convention into six sentinel scripts. This eval didn't implement it.

Three changes:

1. Classify each exception as "infrastructure is down" vs "the skill genuinely didn't trigger." Credit / key / rate limit / overload / DNS / network → blind.
2. Blind cases don't enter the denominator. All-blind → print a blind banner and exit code 2, print no pass rate at all.
3. The per-case ✗ passed 0/3 detail doesn't print for blind cases either — that display is the false red light.

Both directions have to be blocked; blocking one just swaps a false red for a false green. I added reverse tests: "the model answered but didn't trigger" and "my own JSON parsing broke" must not be classified as blind — otherwise a real failure gets laundered into "we didn't measure it," which is worse than a false red.

Eleven self-tests pass, three of them "prove it starts red" (using the day's real error text as fixtures, not strings I made up — strings I make up get contaminated by my own imagination).

End to end, verified against the day's real failure condition: before the fix, 0/96 (0%) plus 24 ❌; after, a blind banner and the actual reason — and those two lines of fake numbers vanished from the audit output.

Hole #2: a gate that had structurally never judged anything

After topping up the account, I ran it again. This time it worked:

│   TOTAL                                       │ 90/96 (94%) │

All 24 passed (six at 3/4, the rest 4/4). That is the evidence I wanted.

Then the same script printed, below the table:

⚠️ Blind: the eval runner produced output but no n/m results could be parsed — not counted as a pass.

It printed a beautiful table, and then said it couldn't read the results.

Here's why. The outer script — the one that decides pass/fail — called the inner scorer with execFileSync, then used a regex to scrape lines like Testing <name>... 3/4.

Those lines are written to stderr. And execFileSync returns only stdout when the child succeeds.

On top of that, the scorer's last line is process.exit(0), with the comment "Advisory: always exit 0" — it always succeeds.

Put those together:

This gate could not read a result on any run where the scoring succeeded. The only "pass" it ever reported was the empty run — "no skills changed this time" — which exits in 287 milliseconds. It had never once actually judged pass or fail.

This is more insidious than the first hole. The first gives you a wrong answer. The second never gives an answer at all, but because it reports "blind" rather than "failed," it looks like an honest gatekeeper.

Fix: switch to spawnSync and take both pipes — stdout is the JSON the machine reads (the single source for the verdict), stderr is the table the human reads. And write the real lesson into the comment:

"The success path and the failure path receive different information" is itself the reason this script was blind.

The old code did out = stdout + stderr on failure (both pipes) and out = stdout on success (one pipe short). Every test had been written against the failure scenario, so it looked correct forever.

Both holes are the same disease

Side by side:

The shared shape: "couldn't measure" and "measured, it's broken" are encoded as the same thing — or encoded as a thing that can never happen.

I've written before about what a green light actually proves. These two are the other half of that family: red lights and blind states deserve the same suspicion. A check reporting 0% and a check reporting 100% both need you to ask "what did it actually see?"

I've also written about tools' output not being the world's facts. This post is the concrete case: 0/96 is the tool's output. The world's fact was my credit card balance.

Five things you can take away

1. Ask the question. For every script that prints a score: "if the infrastructure is down, what do you print?" If the answer is a score, fix it.

2. Three states, not two. 🟢 clean / 🛑 found something / ⚠️ unable to look. The third state needs its own exit code, must not occupy the denominator, and must not print a pass rate.

3. Block both directions. Infrastructure error → blind; but a genuine failure must not be laundered into "didn't measure." Doing only the first swaps a false red for a false green, which is worse.

4. Take your fixtures from what the target system actually emits. My self-tests use the day's real error text. If I'd written my own "simulated insufficient credit" string, it would have matched my regex perfectly — both came out of the same head — and that test would be a mirror, not a test.

5. Chase state changes. pass→blind, fast→slow, warnings 9→10. The thread I pulled on this whole thing was noticing one check took 100 seconds — when the previous round it took 287 milliseconds.

The same disease shows up outside evals. In a retrieval benchmark I ran a week later, the report printed a median score and I let it answer a question that medians structurally cannot answer — write-up here: I turned both knobs on my on-prem Chinese RAG all the way up.

Postscript: the number

90/96 (94%). All 24 trimmed tool descriptions cleared the behavioral threshold.

288 API calls, US$0.30.

I'd estimated "about US$0.02" — off by an order of magnitude, because I estimated "tokens per question" in my head and never counted cache writes and hits (2.54 million cache-hit tokens, as it turned out). Which is this post's theme again: the number you estimate and the number you measure are two different things.