Multilingual data annotation: locale, ambiguity, and quality control
Plan multilingual annotation with locale-aware guidelines, Unicode handling, qualified review, code-switching examples, and per-language evaluation.
Define a locale and task coverage matrix
A language label alone does not describe a usable annotation scope. Record the language, relevant regional variety, domain, writing system, and the task the model performs. A Spanish support dataset from one market may not cover terminology or expectations in another. Identify mixed-language messages, transliterated text, abbreviations, and informal spelling when these occur in the intended input. For each important slice, state whether qualified annotation and review are available. This matrix becomes the basis for collecting examples and reporting limitations, rather than assuming that a translated guideline automatically creates equivalent task coverage.
Separate shared labels from local interpretation
Keep the task ontology stable where the business meaning is shared, then document locale-specific examples and exceptions. An escalation label might retain the same definition across markets while the expressions that trigger it vary. Describe the evidence required for a label, including how much conversational context a reviewer needs. Ask locale reviewers to identify cases where the shared taxonomy cannot represent the local workflow accurately. Resolve those cases with the task owner instead of allowing silent local label changes. Record each decision in a versioned guideline with examples that explain the underlying rule.
Use reviewers who understand the actual material
Select annotators for the language variety and subject matter involved, and calibrate them on representative examples. Conversational fluency does not by itself establish expertise in a specialized product vocabulary. Use a small pilot to identify terminology gaps, missing context, and confusing instructions. Provide an escalation path for disputed meanings and named entities. Machine translation can help reviewers navigate supporting material, but a translated sentence should not replace the original evidence when the task depends on tone, ambiguity, or culturally specific usage. Preserve the source text beside any translation used during adjudication.
Make text normalization an explicit processing step
Keep an original copy of the input and document every transformation applied before annotation or scoring. Unicode normalization can make canonically equivalent character sequences comparable, but the chosen form and its timing should be consistent. Do not automatically remove diacritics, punctuation, or script distinctions if they carry task meaning. Entity offsets require particular care: normalizing a string after span annotation can invalidate character positions. Decide whether offsets refer to bytes, code points, or another documented unit, and test the complete export-to-consumer path. Visual similarity alone is not a sufficient test that two stored strings are interchangeable.
Worked example: one message, two kinds of ambiguity
Consider a synthetic support message that switches between English and Spanish while referring to an account being blocked. One reviewer labels a login problem; another labels a billing dispute because the preceding message mentions an unpaid invoice. The right repair may be to include the previous turn, not to retrain reviewers on a single phrase. Add a context policy and retain a separate field for mixed-language input if it is useful for analysis. Then test examples with and without that context. A third state such as insufficient_context can be more reliable than forcing annotators to guess a hidden cause.
Build independent quality checks for each important slice
Sample annotations within each locale and domain, with enough examples to interpret the resulting counts. A large English batch should not dominate a small but important language group in the quality report. Track disagreement reasons such as terminology, context, entity boundary, intent overlap, and guideline interpretation. Keep initial labels as well as adjudicated results. When a shared rule changes, check whether the correction affects other locales. When a local example changes, confirm that it still implements the shared task definition rather than creating an undocumented exception.
Evaluate the model beyond an overall language score
Report performance by locale and relevant task slice, along with the number of cases. Include mixed-language messages, uncommon scripts, and inputs with missing context where these are part of the deployment. For entity extraction, define boundary and normalization matching rules before scoring. For classification, inspect the confusion pairs that have different operational consequences. A language group with ten test examples cannot support the same level of confidence as one with thousands. Document those limits and collect more evidence before treating a small observed difference as a dependable ranking between models.
Treat translated and synthetic examples as separate sources
Translated examples can help develop a guideline, but they can retain the sentence patterns and assumptions of the source language. Synthetic examples may also overrepresent tidy wording or explicit statements of intent. Mark how each record was produced and avoid placing close translations of the same item across training and test sets when the intended evaluation is independent. Compare performance on naturally occurring, appropriately sourced material with performance on generated or translated material. Use the difference to target collection gaps rather than presenting a convenient synthetic test as proof of complete multilingual coverage.
Deliver a maintained language package
For each release, provide the shared schema, locale supplements, terminology decisions, transformation rules, reviewer calibration results, and split manifest. Include unresolved ambiguities and the conditions under which the application should request clarification or human review. Assign an owner to incorporate new phrases and changed workflows from production feedback. Keep evaluation examples separate from routine reviewer training material. A multilingual dataset remains useful when another team can identify which language conditions it covers, reproduce its preprocessing, and understand why the accepted labels mean the same thing across the supported workflow.