How to write AI data labeling guidelines
Turn a model objective into annotation rules that reviewers can apply consistently.
Start with a decision the model needs to make
Write down what each label will be used for. A support model that routes refund requests needs a different definition of refund intent than a model that summarizes complaints. Give each label an inclusion rule, an exclusion rule, and one example that looks similar but belongs elsewhere.
Test the edges before scaling
Give several reviewers the same small batch. When they disagree, record which rule each reviewer applied. If the text leaves two reasonable interpretations, improve the rule or add an uncertainty label. Keep those disputed cases as a calibration set and revisit them when the taxonomy changes.
Write a rule reviewers can apply to one example
For a support routing project, define “refund request” as an explicit request to return money already paid. “The item arrived broken” expresses a problem but leaves the desired resolution open. “Can I get my money back?” supplies the missing intent. Decide where “I want a replacement or a refund” goes before production begins. A reviewer should be able to explain each decision by pointing to one line of the guidelines.
Give each label an operational definition
Use the same fields for every label: purpose, unit of annotation, required evidence, exclusions, and precedence. For a conversation, state whether reviewers can rely on earlier messages or only the current turn. Record when an item should be skipped for missing context. Include one example of an ordinary case and one close contrast. A consistent structure makes it easier to spot missing rules.
Run a blind pilot and adjudicate disagreements
Give reviewers overlapping batches and ask them to label independently. Compare disagreements by label pair, then distinguish a missing rule from a genuine ambiguity. For example, confusion between cancellation and refund may come from how the guidelines handle orders that were already charged. Write down the adjudicated ruling and its rationale. Run a fresh set of similar examples to see whether the next group reaches the same decision.
Review changes before applying them to old labels
Assign a version to the guidelines, and attach it to each labeling batch. If a label changes, identify which earlier records could now receive a different answer. Sample those records before choosing between targeted relabeling and a full pass. A useful sign-off artifact is a short change log with the new decision rule, examples of affected cases, and a named reviewer who resolved the change.