LLM-as-judge is broken in five specific ways. Here is how we use it anyway. · field note 5

Production ML teams have adopted LLM-as-judge as the default scoring mechanism, and the bias literature is now extensive enough to be uncomfortable. Position bias around 40%, verbosity bias around 15%, self-enhancement bias, authority bias, and silent judge drift. Here are the five problems with names and numbers, and the four methodological controls we apply to get a defensible result anyway.

This note is for production ML teams who have adopted LLM-as-judge for some part of their evaluation pipeline and have started noticing that the judge's scores are not as stable as they look. The takeaway: the biases are documented, quantified, and well-known to the people who study them carefully, and there are four methodological controls that get you to a result you can sign your name to. If you are scoring with an LLM and you are not doing all four, your scorecard is reporting noise as if it were signal.

Why LLM-as-judge became the default

By 2024 the production evaluation field had a problem. Human raters were expensive, slow, and disagreed with each other in ways that made it hard to compare runs across weeks. Hard-coded reference answers worked well enough for some tasks (math, code execution, structured outputs that can be parsed and checked) but they fell apart for the bulk of LLM use cases, which involve open-ended generation against criteria that cannot be reduced to a regex.

Something had to score the open-ended outputs at production scale, and the answer that emerged was to use another LLM as the scorer. You send the candidate output and the rubric to GPT-4 or Claude or Gemini, you ask it to produce a score against the rubric, and you log the result. This is now the standard scoring mechanism across the production-eval ecosystem. Promptfoo supports it natively. Ragas builds on it for the RAG-specific metrics. DeepEval ships it as a default scorer. LangSmith integrates it into evaluation runs. If you have looked at any production evaluation pipeline in 2026, you have looked at LLM-as-judge running underneath it.

The problem is that the judge has its own biases, and those biases are now well documented.

The five documented biases

1. Position bias of roughly 40% on GPT-4. When the same two responses are presented to the judge in different orders, the judge picks the first one a significantly larger fraction of the time than chance. The Adaline 2026 analysis reports position-bias inconsistency around 40% on GPT-4. If your evaluation compares model A against model B and you always present A first, you are partly scoring A's position rather than A's output.

2. Verbosity bias of roughly 15 percentage points. Judges systematically prefer longer responses to shorter ones, even when the shorter response is correct and complete. The inflation measured in pairwise preference studies is around 15 percentage points. If model A produces 200-token answers and model B produces 80-token answers, the judge is partly measuring length when you wanted it to measure quality.

3. Self-enhancement bias of 5 to 7 percentage points. When the judge is the same model family as one of the candidates being scored (for example, GPT-4 scoring outputs from GPT-4 against outputs from Claude), the judge tends to favour outputs from its own family by a small but persistent margin. This is documented across multiple papers, including the arxiv literature on judge bias and uncertainty estimation.

4. Authority bias and rubric anchoring. Judges over-weight the framing language used in the rubric, the order in which criteria are listed, and the example responses provided. Two semantically equivalent rubrics with different surface wording can produce meaningfully different scores on the same outputs. The judge is not actually reading the rubric the way you assume it is.

5. Silent judge drift from provider-side updates. When the model behind your judge gets a silent provider-side update (which happened to GPT-4 multiple times in 2024 and continues to happen across all the frontier APIs in 2026), the judge's scoring behaviour drifts. Scores compiled before and after a silent update are not directly comparable, but most teams compare them anyway because they do not know the update happened.

A 2026 RAND study reports that no LLM judge is uniformly reliable across benchmarks, with frontier models exceeding 50% error rates on challenging bias benchmarks. That is not a small problem. It is the central problem with the scoring mechanism most production teams have already adopted.

The four methodological controls we apply

We use LLM-as-judge on most of our engagements because the alternatives at production scale are unworkable. Here is what we do to make the result defensible.

1. Position randomization. Every pairwise comparison is run twice with the order of the candidate responses swapped. If the judge agrees with itself across both runs, the comparison counts toward the score. If the judge contradicts itself, the comparison is flagged and either re-run with a different judge or escalated to human review. This neutralises position bias entirely, at the cost of doubling the API spend on pairwise scoring, which we think is worth paying.

2. Rubric-based scoring with criterion separation. Instead of asking the judge "which response is better," we score each response against a fixed rubric with explicit and separable criteria such as accuracy, completeness, format adherence, instruction following, and hallucination. Each criterion is scored independently and the judge does not see the others. This reduces verbosity bias (because length is not a criterion) and authority bias (because the rubric structure is held constant across runs and is calibrated against a small human-labelled set at the start of the engagement).

3. Rogan-Gladen calibration for confidence intervals. This is the methodological control most production teams do not implement, and it is the one that most often turns an interesting score into a defensible one. The Rogan-Gladen estimator, originally from a 1978 epidemiology paper, corrects for the known false-positive and false-negative rates of a measurement instrument, which in our case is the judge. We measure the judge's sensitivity and specificity against a small human-labelled calibration set at the start of every engagement, and every score the judge subsequently produces is corrected using the Rogan-Gladen formula. The estimator stays unbiased under distribution shift between the calibration set and the test set, which most newer bias-correction methods do not. The output we publish is a corrected score with a confidence interval that reflects both sampling error and judge unreliability.

4. Multi-judge majority voting on the calls that matter. For any comparison that crosses a passing threshold, or that appears in a signed report, or that influences a published scorecard, we run three independent judges from different model families (typically one from OpenAI, one from Anthropic, one from Google) and require majority agreement before the comparison counts. Disagreements get escalated to human review. This catches self-enhancement bias because no single model family can dominate the call, and it catches judge drift because three independently-drifting judges rarely all drift in the same direction at the same time.

When LLM-as-judge is appropriate, and when it is not

LLM-as-judge is appropriate for high-volume, low-stakes evaluation where the cost of any single wrong score is bounded. Sampling production traces for quality drift. Comparing prompt variants in development. Filtering for obvious failures in a regression suite. Anything where you need to score thousands of outputs and a few wrong scores in the mix do not change the conclusion.

LLM-as-judge is not appropriate on its own for evaluation that ends in a signed report with the firm's name on it. We do not let an LLM judge be the final authority on whether an eval suite passes for a regulatory submission. Every signed score in a Lattice/AI scorecard either has multi-judge majority agreement behind it or has been escalated to human review, and the methodology pack that ships with the scorecard says which.

The one-line summary

If you score with one LLM judge, in one direction, with an unmeasured rubric, you are publishing the judge's biases dressed up as evaluation. If you randomize position, separate the rubric criteria, calibrate the judge against a human-labelled set with Rogan-Gladen, and require multi-judge agreement on the calls that matter, you are publishing something defensible. The math is not hard. The controls take a few extra hours per engagement. Most teams skip them. We do not.