← All articles
Data labeling

Practical guidelines for multimodal data annotation

Connect image, audio, and text evidence with labels reviewers can verify.

Name the evidence behind each label

A clip may show one thing while its narration describes another. Ask reviewers whether the label should reflect the visible scene, the spoken words, or an explicit relationship between them. For temporal tasks, record the segment that supports the decision.

Handle missing and conflicting inputs

Designate a label for cases where an audio track is missing, a caption conflicts with an image, or a transcript loses important context. In quality review, replay the exact interval attached to each label and check whether another reviewer can reproduce the decision from the same inputs.

Work through a case with conflicting signals

Imagine a clip where someone says “the door is closed” while the door is still moving. A speech label can capture the statement, while a visual state label remains open until the frame that shows closure. A label such as narration matches scene should use both sources with clearly defined timestamps. Separate these labels if users need to distinguish observed state from a statement about it.

Put all evidence on a shared clock

Define how text spans, video frames, and audio segments are aligned. If a transcript uses relative time after a clip was trimmed, record the offset before reviewers annotate. On a pilot set, seek to the stored intervals and compare what plays with the label. Small offsets can be especially costly when identifying short events or the order of two actions.

Account for missing and low-quality modalities

A quiet audio track, a blurry image, and a missing transcript are different problems. Record which modality was unavailable and whether the remaining evidence supports the label. If two reviewers can reasonably reach different conclusions, specify when the task should go to adjudication. Keep a sample of partial inputs to test whether tools and guidelines let reviewers express uncertainty consistently.

Audit the linkage as well as the label

Review some records by opening only the cited evidence span. Check whether the reviewer can reproduce the label from it and whether adjacent intervals would change the answer. In exports, confirm that the evidence still points to the same media version after transcoding or transcript edits. Record linkage defects separately from wrong labels so teams can fix the correct pipeline stage.