How to audit data labeling quality
Combine random sampling and targeted checks to catch labeling errors before delivery.
Sample both typical and difficult cases
Use a random sample to estimate errors in the batch as a whole. Review a separate sample of rare labels, edge cases, or tasks with unusual completion times. Report these samples separately so a deliberate oversample of difficult cases does not distort the overall estimate.
Turn defects into a correction plan
Tag each defect by type, such as a missing object, incorrect class, or inconsistent span boundary. Look for clusters by guideline version or data source. Update the instruction that caused the confusion, then review a fresh sample from the affected batch before accepting the fix.
Choose the unit and denominator up front
An image-level audit asks whether an image contains any defect. An object-level audit asks what share of labeled objects are wrong, which can miss unlabeled objects unless the auditor also reviews the whole image. Set a unit that matches the delivery requirement. Count missing, duplicate, and incorrect annotations separately so a single headline rate does not hide differences in workload or impact.
Read a sample result in context
Suppose an auditor finds six defective records in a random sample of 120. The sample defect rate is 5%. If the same audit finds eight defects in 40 handpicked edge cases, report the targeted result as 20% of that edge-case set. Combining the two into a single unweighted rate would describe your audit sample mix rather than the delivered batch.
Use defects to decide what must be reworked
Sort issues into critical problems and routine quality defects before the audit. A training set with systematic missed objects may require a batch-wide correction, even if the random sample looks acceptable overall. For isolated formatting errors, a targeted fix may be enough. Track batch ID, guideline version, defect type, and severity for each decision.
Re-audit the repair with a fresh sample
Retest examples from the defect cluster and draw a fresh random sample from repaired records. Keep the original results so you can tell whether the correction reduced one error class while introducing another. Include the sample sizes and sampling method in the handover. A later team should be able to tell what the audit checked, what passed, and what was outside scope.