← All articles
Data labeling

How to measure inter-annotator agreement

Use reviewer disagreement to improve guidelines and identify ambiguous examples.

Look beyond a single score

Begin with how often reviewers gave the same label and a table of which labels they confused. Rare classes can disappear in an aggregate score. A chance-adjusted measure such as Cohen’s kappa can help for two reviewers using the same categories, but class prevalence affects its interpretation.

Make disagreement actionable

Sample conflicts and separate unclear wording, missing context, and genuine ambiguity. Revise the instructions where a consistent decision is possible. For subjective tasks, retain multiple ratings when disagreement itself is informative. Rerun the same calibration examples after an update to see which disagreements persist.

Calculate a small example by hand

Suppose two reviewers label 100 items as positive or negative. They agree on 38 positive and 42 negative items, with 12 conflicts in one direction and eight in the other. Observed agreement is 80%. Their positive rates are 50% and 46%, giving 50% expected agreement from those marginal rates. Cohen’s kappa is therefore (0.80 − 0.50) divided by (1 − 0.50), or 0.60.

Inspect the decisions behind the metric

Alongside the score, show the confusion table and the label frequency for each reviewer. In a safety dataset, thousands of easy negatives can hide disagreements on the few cases requiring escalation. Add a targeted sample of those cases and report its results separately. Use an agreement measure appropriate for your setup, especially when different subsets of reviewers label different items.

Keep measurement independent of adjudication

Measure agreement before reviewers talk to one another or see the adjudicated answer. Preserve original judgments and record the final decision separately. If reviewers changed their minds after seeing missing context, mark that result as a workflow issue. If they still disagree with complete context, check whether your label definition is asking for a preference or an observable fact.

Use calibration to check whether a change helped

For every major disagreement cluster, revise one criterion and add contrasting examples. Test the revision on fresh cases as well as a small overlap with the earlier set. Check which label pairs improved and which did not. Avoid treating a single threshold as acceptance for every task. Boundary-sensitive extraction, subjective ratings, and binary routing decisions have different error costs.