RAG evaluation: grounding, faithfulness, and citation traceability
Retrieval-augmented generation fails in quiet ways: confident answers with no support in the source, citations that do not say what the answer claims, and silent drift as the corpus grows. We evaluate RAG systems for grounding and traceability and publish a signed scorecard.
Why RAG systems fail in production
A RAG system can look right in a demo and be wrong in production. The retriever misses the relevant passage, the model answers anyway, the citation points at a document that does not support the claim, and the whole thing drifts as the corpus grows and changes.
These failures are quiet. They do not throw errors. They just erode trust one confident wrong answer at a time.
What we measure
We score the dimensions that decide whether a RAG answer can be trusted.
- Faithfulness: is the answer actually supported by the retrieved context
- Context precision and recall: did retrieval surface the right material
- Answer relevance: does the answer address the question asked
- Citation traceability: does each cited source really back the claim it is attached to
- Drift: does quality hold as the corpus grows
Tooling
We build on Ragas for the RAG-specific metrics and layer custom test sets on top for the failure modes specific to your domain and your corpus.
How we score and baseline
We score against your current production system, with confidence intervals on every score and a reproducibility pack (model version, harness version, prompt version, and seed) so the run can be repeated a year later and still mean something.
The shape of a typical engagement
A representative engagement is retrieval evaluation over a knowledge base that spans years of documents, for example a compliance corpus, scoring grounding, citation traceability, and drift across the full source set. The shape above is illustrative, not a specific client engagement.
The signed scorecard you receive
The engagement ends with a public, signed scorecard in the same format as every Lattice/AI engagement, with the failing suites listed alongside the passes.
Who signs the report.
Lattice/AI is an independent AI evaluation firm, founded and led by Rahul Srivastava. For seventeen years he has made it his work to find how systems fail before the people who rely on them do, and he built Lattice/AI to bring that discipline to AI.
His research on AI safety is published and peer-reviewed, he contributes to Microsoft PyRIT, the open framework Microsoft built for adversarial AI testing, and his findings have carried enough weight for a major model vendor to ship a fix in response. What you are hiring is judgment proven in the open, with a name attached to it, not a logo.
Every engagement ends in a signed, dated report that shows what failed next to what passed. Here is the format we ship: a sample scorecard.
Questions buyers ask.
What is RAG evaluation?
Measuring whether a retrieval-augmented generation system gives answers that are actually grounded in its sources, retrieves the right material, and cites honestly, rather than just sounding plausible.
How do you measure whether answers are grounded?
We score faithfulness (is the answer supported by the retrieved context) alongside context precision and recall, answer relevance, and citation traceability, using Ragas and custom domain test sets.
What is citation traceability and why does it matter?
Citation traceability checks that each cited source genuinely supports the claim attached to it. It matters because a confident answer with a citation that does not back it up is worse than no citation at all.
Can you evaluate a RAG system over a private or regulated corpus?
Yes. We evaluate retrieval and grounding over private and regulated knowledge bases, and the engagement is designed to respect the confidentiality of the corpus.
How is this different from general LLM evaluation?
General LLM evaluation scores the model's own behaviour. RAG evaluation scores the retrieval plus generation pipeline together: whether the right material is found and whether the answer stays faithful to it.
Tell us what’s under contract.
Three sentences is usually enough. Tell us what you are trying to ship, when it has to land, and the thing that scares you about it.