This note is for the people who have to read scorecards but do not write them. CISOs reviewing a vendor’s claims. Heads of Risk signing off on an internal deployment. Executives deciding whether to put the company name on something an AI helped produce. If you do not run evaluations for a living but you have to make decisions based on someone else’s eval results, this is for you.
The takeaway is short. An honest scorecard has five things on it. If any of the five is missing, the scorecard is incomplete, and you should ask why.
The five things every honest scorecard contains
1. The passes
The list of evaluation suites the system was tested on, and the score it got on each. This is the part everyone shows you. It looks like a green dashboard.
2. The failures
The same list, but for the suites where the system did not clear threshold. If a scorecard shows you only passes, it is not a scorecard. It is a sales sheet.
This is the most important of the five. The point of evaluation is to surface what is broken, not to congratulate the team that built the system. A scorecard that hides failures is hiding the only information you actually need.
3. The methodology
What the suite measured, why it measured those things, and what threshold the system had to clear to count as a pass. A score of 94% means nothing without knowing what the 94% is of. What cases? Selected how? Scored by whom?
The methodology section is also where you find out whether the eval was run against synthetic test cases the team wrote (low confidence in the result) or against real production traffic the system would actually see (high confidence). The difference matters.
4. The baseline
What the system is being compared against. A 94% pass rate on a hallucination suite is only interpretable when you know the baseline. Is the previous version of the same system at 92%? Is a non-AI baseline at 65%? Is a random-answer baseline at 35%? The number without the comparison is a number. The number with the comparison is information.
5. The signature
The name of the person or firm who signed the scorecard, and the date they signed it. AI systems drift. Models get retrained. Vendors update. A signature with a date tells you when the scorecard was true. If you are looking at a scorecard with no date and no name on it, treat it as a marketing artifact and ask for the signed version.
Three things to look for when something is off
A "100% pass" scorecard
In any honest evaluation of any real AI system, something fails. The system is too good at one thing and too cautious at another, or there is one adversarial suite it cannot clear, or the cost suite has a regression. When a scorecard claims 100% on everything, the eval suite was almost certainly curated to flatter the system. Ask which suites were dropped to get to that number.
No baseline mentioned
If the scorecard tells you the system scored 94% but never tells you what 94% is being compared against, the number is unanchored. Ninety-four percent of an easy benchmark is a failure. Ninety-four percent of a hard benchmark is a serious result. You cannot tell which one you are looking at without the baseline.
No date, or a date older than the model version
A scorecard from before the most recent model update is a scorecard for a different system. The model that shipped today is not the model that was scored last quarter. If the date on the scorecard is older than the model version it claims to evaluate, the scorecard is not actually evaluating the model in front of you.
The takeaway
A scorecard is a snapshot, not a forever statement. It tells you what an AI system was doing on a specific date, against a specific suite, signed by a specific person. Treat it that way.
When a vendor or an internal team shows you a scorecard, ask three questions: where are the failures, what is the baseline, and who signed it and when. If any of the three does not have a clean answer, the scorecard is not telling you what you need to know. That itself is information worth acting on.