How ECE works
A new paper from Dekun Yang introduces a fact-checking framework that does something most automated fact-checkers refuse to do: it says "I don't know" when the evidence is weak. The approach, called Evidence Chain Evaluation (ECE), is a selective prediction system built around a tool-using verification agent that pulls evidence from web search, scholarly search, and executable checks before issuing a verdict.
The system returns a structured output with a verdict, a confidence score, and source-level metadata that explains how strong the underlying evidence actually is. Instead of forcing a true or false label on every claim, ECE is allowed to abstain, and the framework tracks both the cases it answers and the cases it defers.
The design is a direct response to a known failure mode in LLM fact-checking. Models can achieve strong accuracy on benchmarks while still issuing confident verdicts on claims where the supporting evidence is sparse, internally inconsistent, or simply missing. ECE trades a small amount of coverage for a much higher accuracy on the claims it does answer.
The numbers
On ECE-Bench, the framework hits 91.6 percent standard accuracy, 93.7 percent coverage, and 97.8 percent selective accuracy on the claims it does not defer. The improvement on selective accuracy is the meaningful one. It tells you that when the system chooses to answer, it is almost always right.
ECE does not beat the strongest retrieval baseline on aggregate calibration metrics like Expected Calibration Error, Brier score, or AURC. What it does is offer a clearer selective-prediction trade-off, which is the property that actually matters for high-stakes deployments.
The deferred cases cluster in the right places. Six of the 95 claims in the benchmark were deferred, and five of those came from sources that the framework's own evidence tier ranks as low reliability (L4). That is exactly the behavior you want: abstention concentrated on epistemically weak claims, not on hard claims that happen to be wrong.
Why this matters
For anyone deploying LLM fact-checking in a real product, the value of selective prediction is that it lets you bound the failure rate. A system that is right 98 percent of the time on the cases it answers, and that tells you honestly when it is not confident, is a system you can put in front of users with a confidence threshold attached.
The paper's framing is that abstention functions as a safety-oriented mechanism for handling epistemically weak evidence, and the experimental results support that read. The framework is not a full solution to the factuality problem, but it is a useful operational pattern that other verification agents can copy.
The code is open source at cheshireyang/ECE.git, which is itself a small signal: a benchmark is only as useful as it is reproducible, and the author is betting that the community will run the experiment again.