← All articles
Data labeling

Versioning labeled datasets and preventing split leakage

Track label changes and keep related examples together when creating dataset splits.

Split by the thing you expect to generalize across

If camera frames from one recording appear in both training and testing, the test can reward familiarity with that scene. Group related examples by recording, source document, or another stable identifier before splitting. Review near duplicates where groups overlap.

Make changes reproducible

Keep a manifest with record IDs, label schema version, annotation version, and split assignment. Record why labels were removed or revised. When comparing model runs, confirm both the evaluation data and scoring rules are the same, or report the changed versions alongside the results.

Split related examples as one unit

Suppose your dataset includes five camera angles from each warehouse shift plus clips derived from those recordings. Group records by the shift or recording session that generated them before assigning train and test partitions. Use customer or site grouping if that is the unit you expect the model to generalize across. The appropriate group depends on the production question your evaluation is meant to answer.

Check exact duplicates and near duplicates

Hash original or normalized content to identify exact matches. Review near duplicates with a method appropriate for the data, such as comparing perceptual image features or repeated passages. Inspect candidate overlaps manually when similarity alone is ambiguous. After fixing a split, rerun the overlap check against the exported files, since a later transformation can create additional duplicate records.

Make a manifest that survives relabeling

Keep a stable example ID, source group, content version, schema version, split, and a reference to the annotation. Add a reason for removals and revisions. If an example is derived from a larger record, record the parent ID. After joining sources and labels, fail the export if orphan annotation rows, duplicate IDs, or overlapping group assignments appear.

Freeze the benchmark used for comparisons

Record the dataset version, evaluation split, and rubric alongside your score. If a new release corrects benchmark labels, rerun the baseline and candidate against that release before attributing changes to the model. Keeping the old results with their original version makes the change traceable. Include a short note describing whether the new split tests unfamiliar sources, dates, or only new examples from familiar groups.