This note is for anyone who has read our pillar page on the Evaluate practice and noticed that we say one thing about when to write eval suites, while Hamel Husain (the most cited voice in production LLM evaluation in 2026) says the opposite. We have been thinking about that disagreement, and we want to be public about where we now stand and what is changing on the page as a result.
What we said
Our /practice/evaluate page has, until today, opened the Week 1 to 2 methodology phase with the sentence:
"We write the eval suite against your domain before any agent code lands."
That sentence describes what is sometimes called eval-driven development, by direct analogy to test-driven development in traditional software engineering. You decide what good looks like in writing, you freeze that as the test, and then the team builds the system to clear the test. The argument is straightforward enough on its face: if the threshold is on paper before the system exists, the gate is structurally independent of the people who built the system.
It is a clean position. We have been teaching it on engagements. We need to revisit it.
What Hamel argues
Hamel Husain, in a direct entry in his 2026 LLM Evals FAQ at hamel.dev, takes the opposite position on eval-driven development. He argues that the practice "sounds appealing but creates more problems than it solves" for LLM systems specifically. The reasoning is worth quoting fairly. In traditional software engineering, failure modes are bounded and you can name them in advance, which makes test-driven development a productive discipline. With LLMs, the failure surface is effectively infinite. You cannot write good evaluators for failures you have not seen yet, and trying to imagine them in advance tends to produce evaluators that test things which never break, while the things that actually break in production were not on anyone's list.
His recommended alternative is what he calls error-driven development. You start by running the system against real inputs, you do structured error analysis on the outputs the system produces, and you write evaluators for the errors you actually find. The evaluator suite grows alongside the system, and it grows against real evidence rather than imagined risk.
This is not a casual opinion in a side channel. Hamel and Shreya Shankar have put more than 700 engineers and product managers through their Maven course on this methodology. The FAQ piece is one of the most-shared resources in the production-eval community right now. We have been teaching the opposite of it without engaging the critique. That is not a tenable position for a firm that brands itself on evidence over opinion.
Where we now think both positions are right
After working through Hamel's critique against the actual shape of our engagements, we have come to a position that we think is more defensible than either pure stance.
For capability evaluation, error-driven development is the right default. The capability question is whether the system is good enough at the thing it is supposed to do. For a customer-support agent, that means whether it actually resolves customer issues without making things worse. Hamel is right that the failure modes that matter here are not the ones you can imagine sitting in a kickoff meeting. They are the ones that surface in week one of shadow deployment, in week three after the model provider's silent API update, and in month two when an edge case hits production traffic for the first time. Evaluators built for imagined risk often catch nothing, because they catch what was imagined. Evaluators built from real errors catch things, because they catch what actually happened.
For safety and compliance evaluation, suite-first design is still the right default, and we are not backing down from this. The safety question is whether the system stays within lines that a regulator, a board, or an internal model risk policy has drawn. When you are evaluating an AI system against the EU AI Act, against CBUAE 2/2026, against ISO 42001, the threshold has to be agreed and on paper before the system exists. Otherwise the audit is not independent. If the team building the system also gets to define what passing means after the system is built, you do not have an audit. You have a vendor producing a self-assessment dressed up to look like one.
The refined position is this. The question is not "write the suite first" or "do error analysis first." The question is "what is this eval for?" If it is for the engineering team iterating toward a working system, error analysis on real outputs is how you build a useful eval. If it is for a regulator, a board, or a public scorecard with the firm's name signed to it, the threshold has to be agreed and written before the system gets to influence what counts as passing.
Both modes typically show up in the same engagement, often in the same week.
What this means for our engagements
In practice, our Evaluate engagements now run both modes in parallel.
Week one includes a small set of compliance evaluators written against the regulatory framework in scope, with thresholds agreed in writing before any agent code is reviewed. These are immutable for the engagement. They define what the engagement is auditing against. They are suite-first by necessity, because the alternative is the team grading itself.
Week one also includes the scaffolding for ongoing error analysis against the candidate agent. As the agent runs against real or recorded production traffic, we do structured error analysis on the outputs, and we write evaluators for the failure modes we actually find. These evaluators grow throughout the engagement. They are error-driven by necessity, because the failure surface is too large to anticipate in a single planning meeting.
The scorecard we publish at the end separates the two. Compliance evals report against the threshold that was agreed in week one. Capability evals report against the failure modes that surfaced during the engagement, with each evaluator's origin documented (compliance-driven or error-driven) so the reader can tell what each evaluator was actually measuring against. This is what we will be teaching from now on, and the /practice/evaluate page has been updated to reflect it.
A note on changing our minds in public
We are a new firm. The temptation in the first year of a firm's life is to defend every stated position publicly because the firm needs to look certain to be taken seriously. We think the opposite is closer to true.
The firms in this space whose work we take most seriously, including Apollo Research, METR, and the UK AI Security Institute, all publicly update their stated positions when the evidence moves. They publish post-mortems on their own past work. They name the things they got wrong, and they explain what they think now. That is what credible technical organisations do, and it is one of the few signals readers can use to tell whether a firm is reading the field or just selling into it.
This is the first of what we expect will be many notes that update something we previously said in writing. The brand promise is evidence, not opinions. When the evidence moves, the opinions have to move with it, in writing, with the date and the reasoning attached.