Calibrating LLM judges with human review
Validate model-based scoring against human judgments before relying on it at scale.
Build a human-scored reference sample
Include clear successes, clear failures, and close calls. Use the same rubric for people and the automated judge. Compare scores by criterion and task type, with special attention to whether the judge misses the failures that matter most to users.
Check sensitivity to presentation
Swap the order of paired answers and compare verdicts. Sample cases where length, tone, or formatting differ but substantive quality is similar. Reserve human review for ambiguous examples and rerun the reference sample after changing the judge model, prompt, or scoring rules.
Identify the errors the judge must catch
For a help-center assistant, a judge that consistently misses incorrect policy claims is unsuitable for factual acceptance even if it agrees with humans on tone. Build a reference set that includes such mistakes and report false passes by criterion. Include human rationales tied to evidence so disagreements can be investigated without guessing what the original reviewer intended.
Compare human and judge decisions by slice
Break results out by task, answer length, and whether an item was a close call. For binary decisions, count false passes and false failures. For graded decisions, inspect how far the judge score departs from the human score. Validate on a held-out reference set after tuning the judge prompt. A strong result on the development sample may reflect its specific examples.
Test whether presentation drives ratings
Reverse paired answers and check whether preferences reverse with the answers. Present equivalent content in concise and verbose forms and compare the rationale. Change formatting without altering facts to see whether the judge is responding to presentation. Prefer scoring prompts that require evidence from the answer or supplied reference, and route unstable or high-impact judgments to people.
Set a policy for using the score
Use automated scores to prioritize review or compare aggregate trends once their reliability is measured. Decide which low-confidence or high-stakes outcomes require human confirmation. Record the exact judge model, prompt, rubric, and reference set version. When any of them changes, rerun calibration and compare where false passes occur before accepting a new scoring baseline.