The 9 suites every agent must survive.
Production-grade evaluation for LLMs and agents, built on the tooling the field already trusts (Promptfoo, Ragas, Inspect, PyRIT, LangSmith) and extended with the custom test sets your domain needs. We design the suite before any agent code lands, score in shadow against your real production traffic, and publish a signed scorecard with confidence intervals on every score and a baseline comparison against your current system.
Methodology
/ 4 phasesSuite design (two modes)
We design the suite in two modes depending on what each evaluator is for. Compliance evaluators that map to a regulatory framework (EU AI Act, ISO 42001, CBUAE 2/2026, your internal model risk policy) are written before any agent code is reviewed, with the threshold agreed in writing so the audit remains independent. Capability evaluators that score whether the agent is good enough at the thing it is supposed to do are built error-first, growing from structured error analysis on real or recorded production traffic in weeks 3 to 5. The foundation under both modes is the production tooling the field already uses: Promptfoo for prompt regression, Ragas for RAG metrics when retrieval is in scope, Inspect for general LLM evaluation, PyRIT or Garak for adversarial work, and LangSmith for trajectory inspection. We document the threat model explicitly, freeze the compliance test set at v1, and pin every dependency so the run can be reproduced a year later (model SHA, harness version, prompt version, random seed). See field note 004 for why we run both modes rather than picking one.
Baseline + shadow build
If you have a prod-shadow pipeline, we wire the agent into it. If you do not, we evaluate against recorded production traffic or a synthetic distribution calibrated to your real patterns. We baseline three things in parallel: your current production system, a naive non-LLM baseline (rule-based or retrieval-only), and any prior agent version. Sample sizes are set so every suite score lands with a 95% confidence interval at ±2 percentage points.
Adversarial + cost
We run the adversarial suites and score cost-per-decision and drift relative to the canonical run. Every regression is surfaced in writing with a confidence interval on the delta, so the reader knows whether the change is real or inside the noise. When ground truth is human-labelled, we publish the inter-rater agreement score (Krippendorff's alpha) alongside the suite score so the reader can tell how reliable the labels themselves are.
Gate review + scorecard
We sign and publish the scorecard at a public URL with a signed footer and a date. The failing suites stay on the page alongside the passes, and so do the methodology version, the test-set hash, and the model SHA the run was scored against. Anyone reading the page can re-run the suite against the same inputs and verify the result independently, which is the whole point of publishing it.
Engagement shape
/ what you sign up for- 01An eval suite built on the production tooling the field already uses (Promptfoo, Ragas, Inspect, PyRIT, LangSmith), with custom test sets layered on for the failure modes specific to your domain
- 029 dimensions of evaluation, canonical and adversarial, with frozen test sets and explicit threat-model documentation
- 0314-day shadow regression report against real production traffic, or against recorded equivalent traffic if no shadow pipeline exists, with confidence intervals on every score
- 04A three-way baseline so every score has something honest to be measured against (your current production system, a naive non-LLM baseline, and the prior version of the agent)
- 05Public scorecard, signed and dated, on latticeevals.com/evals, with the methodology version, test-set hash, and model SHA published alongside the result so anyone can reproduce it
- 06Continuous monitoring playbook handed over to your ops team, with documented severity 1–3 escalation criteria
Why we design the suite first
The suite is the contract. Writing the eval before the agent forces us to name what we will accept and what we won’t, in a document signed by the same people who will later have to ship the agent. By the time the agent exists, the threshold is already on paper.
Designing the suite first also reveals which problems are well-posed and which aren’t. A surprising number of agent projects we audit fail at this step. The team can describe what they want the agent to do, but they cannot write the test that would tell them if it did. We will not build past that point.
What "in shadow" means
Shadow means the agent reads real production traffic and produces real outputs, but the user never sees them. We score the agent’s outputs against ground truth and against your existing system’s outputs, in parallel. Two weeks of this catches the distribution shifts that synthetic evals cannot.
If your stack does not have a shadow pipeline, that is a common situation rather than a disqualifier. We have two fallbacks. The first is to evaluate against recorded production traffic, which costs you a sampling job but no live engineering work. The second is to construct a synthetic distribution calibrated to your real traffic patterns by sampling from your logs. Both fallbacks are less sharp than live shadow, and the scorecard says so explicitly.
What rigour looks like
A scorecard is only worth what its methodology lets you defend.
Data governance. Eval data is either supplied by you or sampled from your production logs. We do not retain personally identifiable information past the engagement. Where eval cases must contain PII to be realistic, we hash the PII before scoring and document the hashing scheme in the methodology pack. Eval cases are owned by you and stored in your environment unless we agree otherwise in writing.
Reproducibility. Every published scorecard pins the model SHA, the harness version, the prompt version, the random seed, and the test-set hash. Re-running the suite a year later on the same inputs should produce the same score, modulo model-provider-side drift, which is itself one of the things the suite measures.
Baselines and ablations. A pass rate of 94% on hallucination is not interpretable without context. Our scorecards report the agent score next to the baseline your current system scores, the naive non-LLM baseline, and the prior agent version. If a number on the page does not have something honest to be compared to, the comparison is the work, not a presentational extra.
Statistical confidence. Every suite score is published with a 95% confidence interval. Sample sizes are set so the interval is no wider than ±2 percentage points. Where ground truth is human-labelled, the inter-rater agreement score is published alongside.
The tooling we build on. Wherever the production-eval field has already standardized on a tool that does the job well, we use it rather than reinventing it. Promptfoo handles prompt regression and adversarial replay. Ragas covers the RAG-specific metrics, including faithfulness, context precision, context recall, and answer relevance. Inspect, from the UK AI Security Institute, handles general LLM evaluation at scale. PyRIT from Microsoft and Garak from NVIDIA cover the heavier adversarial automation. LangSmith gives us trajectory inspection when the system is agentic. The methodology pack that ships with every scorecard names every tool, every version, and every custom Lattice/AI extension we add on top.