← All articles
Model evaluation

Writing rubrics for LLM evaluation

Translate broad quality goals into criteria that can be scored and reviewed.

Use observable scoring anchors

Replace “good answer” with distinct criteria. For a policy question, an answer might need to identify the correct rule, state a relevant exception, and cite supporting text. Define what earns full, partial, or zero credit for each criterion.

Choose which failures override the total

Some errors are too consequential to average away. A fluent support answer that promises an unavailable refund may fail the task even if it scores highly on tone. Choose those cases before scoring, then calibrate with short examples that show the difference between a minor omission and a critical error.

Score one realistic answer

A policy assistant is asked how long a customer has to return an unopened item. Award a point for the correct deadline, a point for specifying when the period starts, and a point for naming an applicable exception. An answer with the correct number but the wrong start date gets credit for the deadline and loses credit for the start date. If incorrect guidance would trigger a costly action, add a separately recorded critical-failure flag.

Separate criteria reviewers can observe

A criterion such as “grounded in evidence” should state which evidence a reviewer can consult. For example, require each substantive claim to be supported by the supplied policy, then define how to handle a missing citation for an otherwise supported claim. Avoid awarding the same point twice under different names. If a correctness mistake already counts against a criterion, decide whether a separate critical flag adds information.

Test the rubric before comparing models

Ask reviewers to score the same answers independently. Compare disagreements for each criterion and refine the scoring anchors. Keep anchor examples for full, partial, and zero credit and include an example where the answer should abstain. Recalibrate after changing the rubric, especially when a new criterion changes what counts as a passing response.

Report where scores diverge

Publish scores by criterion and task, with counts of critical failures and abstentions. A higher average can coexist with worse performance on the cases that affect users most. Show at least one representative error for each important regression. Capture rubric and prompt versions with the results so updates can be rerun against the same evidence.