Addressing the Gaps in LLM Factuality Checkers

Tool-using large language model agents frequently leverage the Model Context Protocol (MCP) to call search tools, query databases, inspect structured records, and pull metadata into a single unified answer. Traditional verification systems check whether a claim is supported by available evidence once that evidence has been pooled together. However, tools like RAGAS faithfulness typically fail to indicate which specific tool output supports each individual claim or whether that matches the source cited by the answer.

To tackle this limitation, researchers developed ProvenanceGuard, detailed in a recent paper shared via the Hugging Face Blog. The primary failure mode addressed by this approach is cross-source conflation, which occurs when a claim is true somewhere in the evidence pool but incorrectly attributed to the wrong source.

Understanding Cross-Source Conflation

In data-sensitive settings, incorrect attribution can be just as damaging as an incorrect fact. For instance, a customer support agent might state that an account record includes a specific refund window, even though that policy is actually detailed in a separate policy document rather than the specific account record cited by the answer. When evidence is pooled together blindly, the claim appears supported, but the attribution remains flawed.

A similar risk emerges in clinical environments, where patient-specific medication details retrieved from a history tool can become misleading if presented as findings from general medical literature. Because answers carry explicit or implicit provenance, maintaining the connection between individual claims and their exact sources is vital for accurate verification.

How ProvenanceGuard Verification Works

ProvenanceGuard functions as a post-generation verification layer positioned on top of black-box MCP agents. Instead of collapsing evidence into an anonymous context pool, the system carries source identities all the way through the pipeline by reading captured MCP traces, including tool outputs and source IDs, without retraining the underlying agent.

The sequential workflow breaks the generated answer into specific claims, locates the most relevant source for each claim, checks whether that source actually supports the claim, compares the source against the named or implied source in the answer, and finally emits both a per-claim verdict and a global allow or block decision.

Experimental Setup and Performance Results

During evaluation, the research team utilized local models in a controlled offline setup. MiniLM helped identify relevant sources, a DeBERTa NLI verifier model checked support, and a local language model decomposed the text into specific claims. Literal values such as numbers, dates, and identifiers were strictly checked to prevent plausible-sounding hallucinations from passing.

Testing involved 281 real traces from a medical agent utilizing patient records and research articles. Human experts reviewed 361 claims from 40 answers. Experts noted that 139 claims should not pass, and ProvenanceGuard successfully caught 138 of them while holding 67 supported claims for review or repair, reflecting its conservative decision policy.

Handling Swaps and Repair Loops

In a controlled test focusing on wrong attribution where the named source was swapped in 50 cases while leaving supporting evidence intact, ProvenanceGuard caught all 50 swaps. While distinguishing between many similar sources remains an ongoing challenge, the system demonstrated strong baseline performance.

Furthermore, when blocked answers were connected to a RARR-style repair loop, the full-trace run successfully resolved all 173 blocked answers, demonstrating how verification layers can actively assist in generating safe, source-grounded revisions.

Sources

• Hugging Face BlogGetting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents