I turned both knobs on my on-prem Chinese RAG all the way up. It didn't get a single extra question right

Something you can do right now

If you're about to "increase the retrieval depth" or "swap in a newer reranker" to improve your Chinese RAG accuracy, spend ten seconds on this first:

Take one question you got wrong, and compute where the passage containing the correct answer ranks in your raw vector similarity.

No reranker. No full pipeline. One embedding call and one round of cosine similarity.

That number decides everything downstream:

• If it ranks outside your candidate depth → going deeper might help (only might — see below)
• If it ranks inside your candidate depth → going deeper definitely won't, and your bottleneck is somewhere else

Without that probe I'd have swept top-k = 10 / 20 / 30 / 50 / 80 / 100, six configurations, 29 questions each, tens of seconds per question — about an hour. After the ten-second probe, those six configurations collapsed to one.

Where this started: a to-do I wrote for myself

I was evaluating a "the data never leaves the building" knowledge base for a client, and wrote up my research as a handoff doc for the next session. The to-do was specific:

Step 1: top-k 10 → 50 (zero cost, zero VRAM) — a public benchmark on 3,493 traditional-Chinese questions shows a weak embedding plus a reranker reaching 98%+ at k=100, beating every single-stage retriever. ⚠️ Don't just crank it to the max (a SIGIR 2025 paper reports that over-reranking degrades results); there's a sweet spot and you have to sweep for it. Step 2: bge-reranker-base → bge-reranker-v2-m3 (+1.3GB, same architecture, same API, no prompt changes)

It reads as completely reasonable. External benchmarks behind it, a counter-example warning, a clear execution order.

Both steps ended up as "don't." And that "zero cost" in step 1 is wrong.

First cut: the cheap adjacent question

Before touching anything I looked at the table from my previous round:

96% is 24 out of 25. And the miss is always the same question.

That sentence matters, because it caps the question "how much can deeper candidates buy me?" — at most one question, i.e. 4 percentage points.

So the question changed from "which k is best?" into something far cheaper: did the reranker ever even see the correct answer for that one question?

I wrote a ten-second probe: for each of the 25 questions, where does the passage containing the correct answer rank by raw vector similarity?

One table cut six configurations down to one:

• Depth 10 → correct answer present in candidates for 23/25 (ceiling 92%)
• Depth 30 → 24/25 (ceiling 96%)
• Depth 50 and 80 → still 24/25, because that question sits at 83. Identical ceiling to 30 ⇒ no need to run them
• Depth 100 → 25/25 (ceiling 100%)

And check the calibration: depth 10's ceiling is 92%, measured 92%. Depth 30's ceiling is 96%, measured 96%. Both match to the decimal.

At that moment I derived a conclusion: "whatever's in the candidate set, the reranker pulls into the top 5 ⇒ this pipeline's bottleneck is how deep we fish, not how good the reranker is."

The derivation itself is sound (the miss set must contain the out-of-candidates set, and the two have equal size, so they're the same set).

Then I ran depth 100, and it falsified that conclusion.

Depth 100: ceiling 100%, measured 96%

The correct answer ranks 83rd, so it did make it into the 100 candidates. The reranker saw it. And then ranked it outside the top 5.

So the right statement isn't the one I derived — it's a more useful one:

Candidate depth is necessary, not sufficient. For that question to land, two things must both hold: ① you fish deep enough to see it, and ② the reranker is willing to rank it near the top. Depth 30 fails at ①. Depth 100 clears ① and fails at ②.

Incidentally: "zero cost" is wrong. A reranker is a cross-encoder, and its cost is linear in the candidate count — send it 100 candidates and it computes "how relevant is this passage to this question" 100 times. Going from depth 30 to 100 took each question from 6.8 to 37.7 seconds: 5.5× the latency, for zero extra correct answers.

"Zero VRAM" is right. "Zero cost" is not. And what it spends is precisely the one budget in this setup that had already bottomed out.

Second cut: swap in a "newer" reranker

If the bottleneck is the reranker, replace it. My to-do said bge-reranker-v2-m3.

First discovery: that model isn't available on my stack. I run ONNX through fastembed on CPU, and its cross-encoder list has six models — no v2-m3. Only two are multilingual: the bge-reranker-base I'm already on (a 2023 base-tier model), and jinaai/jina-reranker-v2-base-multilingual (1.11 GB, 2024 generation, marketed on multilingual).

Actually using v2-m3 would mean swapping the whole service from ONNX to PyTorch. That's a separate project. So I swapped in the jina model — same API, one parameter changed, no prompt edits.

It lost in three directions at once:

1. K=1 dropped 12 points (3 questions). K=1 is the column that most affects the quote — it decides how many documents the client has to feed the LLM, which decides whether they need to buy that GPU.
2. The worst drop is on pure Chinese prose (88.9% → 66.7%). A multilingual-first 2024 model losing to a 2023 base-tier model on traditional Chinese prose.
3. It's also 1.2× slower.

But the fourth reason is the fatal one, and I nearly missed it.

The fourth reason I nearly missed

My previous round had turned up a bonus finding: the reranker's score itself works as a "refuse to answer without calling the LLM" gate. Below a threshold, return "not found" — skip generation entirely. Fast, and structurally incapable of making things up.

For this client that matters more than usual, because making things up is exactly what they're afraid of.

The scoring script prints a "median top-1 score" line each round:

• bge-reranker-base: answerable 5.26 / unanswerable −2.14
• jina: answerable 0.84 / unanswerable −0.76

That looks like "the gap got smaller." What I wrote in the report at the time was "this doesn't let us say jina made the gate worse [unknown]."

But the statistic that finding needs isn't the median. It needs the minimum score among answerable questions against the maximum among unanswerable ones — because two distributions can have very different medians and still have overlapping tails.

The per-question scores were already in the scoring run's JSON. I computed it afterwards at zero extra cost:

Switching to jina means: K=1 down 12 points, latency up 1.2×, and then you also forfeit the entire refusal defence.

And the median made it look like it merely "got smaller." That's the most valuable lesson of this round: the statistic a report prints is not necessarily the statistic your conclusion needs. I turned it into a command (reads the stored scoring JSON, zero cost) and left a test inside it: fixtures of answerable [9, 9, 9, −1] and unanswerable [0] — medians differ by 9, looks trivially separable; the truth is min −1 < max 0, overlap. That test exists so that if anyone ever switches it back to medians, it goes red.

What's actually wrong with that one question: I tried a third road too, and it's also closed

I had one hypothesis left. Looking at the top-5 for that question across four configurations, I noticed all five slots in every configuration were filled by the same class of file — my own blog drafts. Derivative articles discussing the same incidents, not the canonical records of those incidents.

That matches a published finding: what hurts RAG isn't "irrelevant," it's "relevant but wrong." It also matches my own earlier observation that this corpus writes about the same event a dozen times over.

So I added an exclusion rule, filtered the entire blog-source directory out of the candidates, and re-ran that question.

Still missed. And the correct answer didn't even make the filtered 30 candidates — excluding the derivative material just pushed other non-answers up.

Three roads walked:

That question asks "what has to be done to a temp table containing personal data for it to count as safe, and which incident forced that?" — and the file with the answer is anchored on a database table name. The question doesn't contain that table name; only the answer does. It's a pure semantic-bridging task, and my embedding model puts it 83rd.

This round proved the embedding is the bottleneck, by elimination — every knob outside the embedding was turned to the stop with no effect. My original plan wrote the precondition for "swap the embedding" as "only swap once you've proven embedding is the bottleneck." That precondition now holds; it just got proven by exclusion.

My next cut will be query rewriting (have the LLM expand the question into a hypothetical answer first, then embed that) rather than swapping the embedding outright — much cheaper, and among the open-source projects I've read, the one that handles Chinese most seriously does its work on the query construction side (part-of-speech weighting, positional proximity weighting), not by changing models.

A red flag that goes straight into the quote

This round also turned up something I wasn't looking for.

To get a same-day control, I re-ran the depth-30 configuration. Recall was identical (96%, same question missed) — comparability confirmed.

But latency was 6,834 ms per question. A week earlier I'd measured the same configuration at 18,152 ms.

Same model, same corpus, same question bank, same laptop. A factor of 2.66.

Two consequences:

1. I'd been comparing a fresh depth-100 number (37,651 ms) against last week's 18,152 ms and computing "2.07×." Wrong. The correct same-day ratio is 5.51×. A latency denominator has to be produced in the same round as its numerator — the same discipline as "pin the corpus to a commit," which I'd applied to the corpus and not to latency.
2. Worse: the conclusion I'd written a week earlier was "18 seconds per question — unusable for interactive Q&A." That sentence cannot be measured today (6.8 seconds). 6.8 seconds is still slow, but "slow" and "unusable" are completely different things in a client quote — and that sentence was headed into the proposal.

I don't know the source of the variance yet (thermal throttling? something else running? a package version?). All I can state is that corpus, question bank, and model are ruled out, because all three were re-verified identical the same day.

So the rule now is: any latency number going into a quote must have its own control, measured in the same round, on the same machine, with no other load.

Net result

Four scoring runs, three probes, roughly an hour of machine time. What it bought:

Don't touch either knob. Stay on bge-reranker-base with candidate depth 30.

That sounds like nothing happened. In practice I got three things:

1. A number I can commit to: on a 25-question benchmark, 96% retrieval at K=5, 88% at K=1, plus a refusal gate that doesn't depend on the LLM.
2. A number I can't commit to: 100% retrieval. Measured as unbuyable — deeper candidates don't buy it, and neither does the newest reranker I can actually run. That's far more valuable than "I think we can probably get there," because it lets me write a ceiling into the proposal.
3. Two numbers not to quote: any local CPU latency (2.66× day-to-day variance), and any presumption that "a newer model will be better."

Plus one observation I can sell: derivative copies inside a client's corpus (drafts, meeting notes, several versions of the same policy) will systematically outrank the canon — my own blog drafts pushing my own canonical records out of the top 5 is the live example. But don't sell "curate the corpus" as a cure-all, because in the same round I measured it doing nothing for that question.

Four things you can take away

1. Before turning any expensive knob, find a ceiling you can compute in ten seconds. "At most how many questions can this buy?" routinely makes an entire sweep plan disappear.
2. "Newer, bigger, multilingual-first" does not mean "better on your corpus." Marketing labels aren't evidence. I put a 2024-generation multilingual reranker in and it lost to a 2023 base-tier model on traditional Chinese prose.
3. The statistic a report prints isn't necessarily the statistic your conclusion needs. A median can't answer "are these separable?" Ask which statistic your claim rests on, then go compute that one.
4. Don't quote a latency number without a control. The same configuration varies 2.66× day to day on my machine, and it nearly put the word "unusable" into a client proposal.

If you're doing this kind of measure-before-you-change work, I've written others in the same family: my test report printed "0/96, 0% pass rate" — the truth was my account was out of credit, on why a red light needs as much suspicion as a green one; and the A/B test where I cut 41 tool descriptions in half, which is the same discipline applied to a change I was confident about — an absolute number can't tell you whether something dropped, so you have to go build the control arm.