← All articles
Model evaluation

How to design human preference labeling tasks

Collect useful side-by-side judgments with clear priorities, ties, and reviewer guidance.

Tell reviewers what earns a preference

Ask reviewers to compare answers using criteria that match the task, such as correctness, following instructions, and completeness. If style matters, explain when it should break a tie. Randomize which answer appears first and hide model names where feasible.

Make close calls visible

Allow a tie or uncertainty when the difference is too small to justify a preference. Give reviewers sample rationales that cite the prompt and the responses. When an answer is preferred for being longer, check whether the task actually needed the extra detail or whether verbosity is influencing the rating.

Compare two responses using explicit priorities

Consider a prompt asking for the deadline and required form. Answer A supplies both facts in plain language. Answer B is polished but omits the form. If completeness is required, the reviewer should choose A and record the missing form as the reason. If two answers contain the same correct information with minor differences in style, the rubric might allow a tie.

Decide what a tie and uncertainty mean

A tie can mean both answers meet the same standard, whereas uncertainty may mean the reviewer lacks the evidence to judge. Give each option a clear definition and examples. Add a separate “both fail” option when it is important to distinguish two poor answers from two good ones. Avoid counting ties as missing data when they represent a deliberate judgment.

Calibrate independently before going to production

Prepare examples where factual accuracy, completeness, and readability point in different directions. Ask reviewers to choose independently and compare their rationales. Check for ordering effects by reversing a subset of pairs. If reviewers choose the first response more often when the substantive difference is small, revisit the UI layout and your tie guidance.

Report collection details with your labels

Keep the prompt, responses, rubric version, presentation order, and reviewer decision tied to the same example. Summarize preference rates by task and by the criteria cited in rationales. A winner rate can obscure the fact that nearly all changes came from one document type or evaluation slice. Retain that breakdown so a model team can act on the result.