LLM self-evaluation missed 100 percent of the parser mistakes it was supposed to catch in a new 18-run stress test of enterprise RAG pipelines. That's the uncomfortable finding from a controlled experiment that put three GPT models against two PDF parsers, with questions designed to expose exactly the kind of structural failure cheap parsers hide. The cascade approach (cheap first, deep parsers only when escalated) is still the right pattern. The LLM as last-line-of-defence is not.
The cost gap behind every parsing decision
PyMuPDF parses a page in five milliseconds and effectively nothing per call. A vision LLM on the same page takes ten seconds and burns roughly 10,000 times more compute. That gap is the entire reason a RAG pipeline cannot just run the heaviest tool on every page. PyMuPDF is correct most of the time, and the cost of guessing wrong on a single page is one wrong answer downstream, not a broken system. So the default is cheap.
The cost of cheap only shows up in failure: a flat table that becomes 13 separate lines, a figure that becomes an image placeholder, a flattened layout that loses every column anchor. The LLM reading the answer prompt has no way to know the parse is bad unless something tells it.
Case A: Table 3 of the Attention paper
The experiment used the "Attention Is All You Need" paper as a controlled failure case. Question one asked for the value of h in the base model row of Table 3, on page 9.
PyMuPDF returned the row as 13 separate text lines. The LLM got the answer right (h equals 8) and the cell right (line 25), but the schema's self-evaluation flag fired anyway. The model flagged the risk: it had to count column positions across a flattened header to map line 25 to the right column. Escalation triggered.
Azure Document Intelligence was then called for the same page. It returned Table 3 as a structured markdown grid, one row per line, pipe-anchored. The LLM now trusted the structure without counting. Same answer, structurally clean. Cost: $0.003 and four seconds.
Case B: Figure 1 and the vision tier
Question two needed the encoder-decoder architecture diagram on page 3. PyMuPDF gave the surrounding prose and a placeholder row for the figure. The LLM read the caveat, recognised that Azure DI would not help here (a diagram is not a table), and escalated to a vision-language model.
The vision model returned a structured description: six encoder layers with multi-head self-attention, residual connections, layer norm, and position-wise feed-forward, mirrored in the decoder with masked self-attention and a cross-attention bridge. The LLM produced a complete answer that named residual placements and the cross-attention layer the prose alone had missed. Cost: 10 to 30 seconds and a few cents for the one image. The other 14 pages of the paper never saw a vision model.
The 18-run reality check
The two walkthroughs above are the best case. The author then ran three questions of growing structural difficulty across gpt-4o-mini, gpt-4o, and gpt-4.1, on PyMuPDF and Azure DI. Eighteen runs total. Four produced wrong answers.
The finding cuts the other way from intuition. All four wrong answers came with self-evaluation flag set to true, meaning the model asserted the parse was clean while fabricating a number from an adjacent column. The only time the flag fired as false was on a correct answer, by the strongest model, on a question that did not need rescuing.
Replacing the binary with a 0.0 to 1.0 confidence score did not help. Mean score on correct answers: 0.919. Mean score on wrong answers: 0.920. The structural verdict is bimodal disguised as continuous, and a 0.95 threshold caught three wrong answers at the cost of nine false alarms on fifteen correct ones, a 60 percent false alarm rate. No usable threshold exists.
The stable signal that actually works
Asking the LLM for a one-sentence rationale alongside the flag changed the picture. The rationale text was concrete and consistent across runs: "headers split across multiple lines," "each cell on its own line," "blank cells preserved." The model can describe the parse shape reliably. What it cannot do reliably is decide whether that shape is bad enough to escalate.
Run the same prompt three times on the same input, and the rationale stays stable while the flag and score swing. The fix is to read the rationale with a deterministic rule ("contains 'split across multiple lines' or 'each cell on its own line' or 'blank cells'") and escalate on the rule, not on the model's verdict.
A separate judge to catch what self-eval cannot
The architecture the experiments point to has two final layers. After generation, a separate LLM (or NLI model) verifies that every claim in the answer is supported by the cited chunks. That judge is fresh, has no incentive to defend its own generation, and is what catches the fabrications self-evaluation missed. The cost is one extra LLM call. The reliability gain is the difference between a stress test with zero catches and one with most of them.
What this means for off-the-shelf RAG
The implication for teams buying or building RAG off the shelf is sharp. A tool that returns confident answers without exposing a parse-quality audit row, or without a separate judge in the loop, is shipping exactly the failure mode the stress test exposed. The $0.003 cost of a single Azure escalation on a contested page is a rounding error against the cost of one hallucinated compliance claim in a regulatory filing, or one wrong number quoted in a board deck. Auditability, not raw accuracy, is the new moat.
What to watch next
The cascade order is the takeaway: deterministic checks on the parse output first (cheap, milliseconds), then intent-aware routing on the question, then score-gap and context-gap checks at retrieval, then LLM self-eval as a last-line signal, then a fresh judge after generation. The next test in the series applies the same pattern to cross-references, where a "see page 7" pointer inside a document becomes the new failure shape the cascade has to catch.