← All articles
Data labeling

OCR and document AI labeling: a practical dataset guide

Build reliable document extraction datasets with field definitions, reading order, annotation rules, quality checks, and a worked invoice example.

5 min read

Start with the decision the extracted data supports

A document extraction project should begin with the downstream workflow. A searchable archive needs readable text and reliable page references. An invoice workflow also needs the supplier, currency, total, and the association between each line item and its price. These are different annotation tasks. Write a one-page specification that names the document families, the fields the application consumes, and the action taken when a field is missing or uncertain. Decide whether the system only suggests values for review or automatically updates a business record. That distinction determines which errors deserve the most attention during labeling and evaluation.

Separate transcription, structure, and meaning

Treat these as separate layers in your schema. Transcription records visible characters. Structure records blocks, tables, reading order, and links between regions. Meaning assigns a business label such as invoice_total or purchase_order_reference. A model can transcribe 245.00 correctly and still select the subtotal instead of the amount due. Keep the original text and location alongside the interpreted value so reviewers can trace an extraction error. Do not invent unreadable characters. Provide explicit states for illegible, absent, and not applicable, because these conditions require different downstream behavior.

Write field rules before annotating a full batch

For every field, define its meaning, allowed multiplicity, location policy, and normalization policy. Specify whether a total includes tax, whether a supplier address can span several regions, and whether a date must retain the original format. Preserve the visible string separately from normalized values. An ambiguous date such as 03/04/2026 should not silently acquire a locale assumption. Include screenshots of valid and invalid selections in the guideline. Ask two annotators to label the same small pilot, then resolve disagreements by updating the rule and its examples before production labeling begins.

Make bounding regions and reading order reproducible

Define whether a region follows text tightly or includes surrounding whitespace, and how rotated text is represented. Record the page dimensions and coordinate system so boxes remain meaningful after resizing. For multi-column layouts, describe how headings, footnotes, and continuation text enter the reading sequence. Tables need stable row and column associations, including merged cells and descriptions that wrap across lines. Review a sample in the actual annotation interface rather than only checking exported coordinates. A technically valid polygon can still cover the wrong cell or omit the characters needed to verify the label.

Worked example: an invoice with two plausible totals

Consider a synthetic invoice showing subtotal 200.00, tax 40.00, delivery 5.00, and amount due 245.00 USD. The target invoice_total is 245.00, with the currency stored as USD and the original region retained. If a previous payment reduces the balance to 145.00, the project owner must decide whether the application needs invoice total, outstanding balance, or both. Add separate fields instead of stretching one label to cover incompatible meanings. Use this example during reviewer calibration. A disagreement about 245.00 versus 145.00 is a specification issue until the field definition makes the intended choice unambiguous.

Sample the variety that will reach production

Build a coverage matrix for document family, source, language, capture method, page count, and image quality. Include scans, exported PDFs, photographs, missing pages, stamps, handwriting, and unfamiliar layouts where these occur in the intended workflow. Keep all pages from the same document together when splitting data. Consider grouping recurring supplier templates or related document versions when the evaluation goal is generalization to unseen layouts. Record which source groups are absent from the test set. A score from clean single-page PDFs should not be presented as evidence for photographed multi-page forms.

Evaluate fields and complete documents separately

Use field-level precision and recall to identify missed and incorrect extractions, then add workflow-specific checks. For a synthetic test containing 100 documents, suppose 90 have a correct total and 95 have a correct currency, but only 86 have both. Report those three counts rather than inferring document success from either field alone. Define what counts as a match before scoring: exact text, normalized numeric value, or a documented tolerance. Track illegible cases separately from model errors. Where a false extraction triggers more work than an abstention, evaluate a review threshold against the cost of both outcomes.

Review disagreements without hiding their origin

Store the first annotation, reviewer decision, reason code, and guideline revision. Useful reason codes include wrong region, wrong field meaning, transcription error, omitted repeated value, and unsupported normalization. Sample accepted annotations as well as disputed ones so systematic mistakes do not escape review. When a rule changes, identify all affected examples and relabel them consistently. Keep the pre-change dataset version available for comparison. Quality reporting should show which errors were repaired, how many documents were reviewed, and which unresolved cases remain outside the release scope.

Prepare a useful handoff package

Deliver a versioned manifest, schema, guideline, sample exports, split assignments, and a quality report with counts by document family. Include a small set of difficult examples with explanations that a future reviewer can understand. Validate that every region refers to an existing page, required fields use supported states, and repeated fields retain their associations. Before accepting the dataset, run it through the consuming pipeline and inspect the resulting records. This final integration check catches mismatches between labeling assumptions and application expectations that a standalone annotation review cannot reveal.