← All articles
Data labeling

How to audit data labeling quality

Combine random sampling and targeted checks to catch labeling errors before delivery.

Sample both typical and difficult cases

Use a random sample to estimate errors in the batch as a whole. Review a separate sample of rare labels, edge cases, or tasks with unusual completion times. Report these samples separately so a deliberate oversample of difficult cases does not distort the overall estimate.

Turn defects into a correction plan

Tag each defect by type, such as a missing object, incorrect class, or inconsistent span boundary. Look for clusters by guideline version or data source. Update the instruction that caused the confusion, then review a fresh sample from the affected batch before accepting the fix.

Choose the unit and denominator up front

An image-level audit asks whether an image contains any defect. An object-level audit asks what share of labeled objects are wrong, which can miss unlabeled objects unless the auditor also reviews the whole image. Set a unit that matches the delivery requirement. Count missing, duplicate, and incorrect annotations separately so a single headline rate does not hide differences in workload or impact.

Read a sample result in context

Suppose an auditor finds six defective records in a random sample of 120. The sample defect rate is 5%. If the same audit finds eight defects in 40 handpicked edge cases, report the targeted result as 20% of that edge-case set. Combining the two into a single unweighted rate would describe your audit sample mix rather than the delivered batch.

Use defects to decide what must be reworked

Sort issues into critical problems and routine quality defects before the audit. A training set with systematic missed objects may require a batch-wide correction, even if the random sample looks acceptable overall. For isolated formatting errors, a targeted fix may be enough. Track batch ID, guideline version, defect type, and severity for each decision.

Re-audit the repair with a fresh sample

Retest examples from the defect cluster and draw a fresh random sample from repaired records. Keep the original results so you can tell whether the correction reduced one error class while introducing another. Include the sample sizes and sampling method in the handover. A later team should be able to tell what the audit checked, what passed, and what was outside scope.