How to Extract Structured Data from Documents with LLMs

Key takeaways

• 01

  Layout-aware OCR plus page images beats a raw text dump on invoices, forms, and scanned contracts.

• 02

  JSON Schema (with constrained decoding or repair) is the contract between the model and downstream systems.

• 03

  Every field should point at a page and box; that citation is how reviewers and auditors trust the number.

• 04

  Human review on confidence, schema failure, and amount thresholds is the production control — not a nicer prompt.

• 05

  Score field-level precision/recall on a golden set of real documents; CI should fail when a pin drops a critical field.

Extracting data from PDFs with AI means turning messy pages into records a system can post: vendor, dates, line items, totals, clauses. Classic OCR plus templates breaks when a new vendor layout appears. LLMs (text, vision, or both) generalize across layouts if you give them structure: schema, layout tokens, and a rule for what to do when a field is missing.

The unit of work is one document family and one schema. “Ingest the shared drive” is not a pipeline. Invoice OCR LLM projects fail when they try to also do contracts in the same prompt. Split the traffic.

Digital PDFs often already contain text; use it and still keep coordinates. Scans need OCR. Layout models mark tables, headers, and key-value pairs so line items stay rows. Reading order errors are a top source of swapped fields (ship-to vs bill-to).

Vision LLMs help on stamps, handwriting, and layouts OCR mangles. They cost more and still need a schema. A hybrid that OCRs first and sends page images only when native text is thin is usually the production default. Measure both paths on the same gold docs.

The schema is the product. Use JSON Schema with required fields, date formats, currency codes, and decimal types. Constrained decoding (grammar/JSON mode) reduces repair. Still validate: models emit extra keys and stringified numbers. A repair loop that re-asks only invalid fields is cheaper than regenerating the whole document.

Line items belong in arrays with their own citations. Totals should be checked against the sum of lines within a tolerance; mismatches go to review even if the model was confident. That single arithmetic guard catches a class of fluent errors evals love to miss.

A field without a box is an unverified claim. Store page, coordinates, and the snippet. The UI highlights the box when the reviewer focuses a field. Query-time ACLs still apply: reviewers only see documents their role allows, same as any other enterprise retrieval.

Route on confidence, missing citations, schema failure, new vendors, and amount thresholds. Measure straight-through processing (STP) rate and reviewer time per doc. Human edits become training and gold; they should not vanish into a spreadsheet. Shadow mode can run a new model pin next to production and compare field diffs before you change STP gates.

Score per field: precision, recall, exact match after normalization (dates, money, IBAN spacing). Score citation IoU or “box contains value.” Weight fields by risk: IBAN errors are not the same as a missing PO line note. Keep a frozen set of real vendor docs; synthetic invoices under-test stamps and skew.

CI runs on OCR/layout/LLM pin changes. A four-week implementation can freeze the schema and gold in week 1, pipeline in week 2, review UI and shadow in week 3, and handover of datasets, evals, and runbooks in week 4. The client owns the labels and the model pins. 98.7% QA pass rate on delivery work does not replace your field-level gold — it is how the implementation itself is checked.

• One document family, one JSON Schema, labeled gold docs
• Layout OCR + optional page images, split by page/region
• Schema-valid JSON with page/box citations per field
• HITL on confidence and money; field-level CI on pins

Fixed-form, high-volume, identical templates may still be cheaper with a classic template or a small specialist model. Handwriting-only archives may need a dedicated HTR stack. If you cannot show a reviewer the source box, do not auto-post payments.

Do not send documents that must not leave the perimeter to a public vision API. Deploy the extractor in the client VPC, same rule as any other private LLM path.

Step-by-step build

1. 1

   Define the schema and the document family

   Write a JSON Schema for one document type (invoice, claim, KYC form). List required fields, types, enums, and which fields are money or identity. Do not mix families in one schema. Collect 50+ labeled examples spanning vendors, languages, and scan quality.

2. 2

   Extract text, layout, and page images

   For digital PDFs, pull native text and character boxes. For scans, run OCR. Keep layout: blocks, tables, key-value regions. Store page images at a resolution the vision model can read. Split long documents by page or by detected sections.

3. 3

   Call the LLM with schema constraints

   Pass the schema, the page content (text + optional image), and instructions to emit only schema-valid JSON. Use constrained decoding or a validate-and-repair loop. For tables, extract line items as arrays rather than stuffing them into a blob.

4. 4

   Attach bounding-box citations

   For each field, record page number and box coordinates of the supporting span. If the model cannot locate a box, mark the field uncited and drop confidence. Reviewers click the box; they should not hunt the page.

5. 5

   Route to a human review queue

   Auto-accept only when schema validates, citations exist, and confidence and amount thresholds pass. Otherwise queue with the image, boxes, and model JSON. Capture reviewer edits as new gold. Never write to finance systems from unreviewed low-confidence records.

6. 6

   Evaluate and reprocess on pin changes

   Score field-level precision/recall, citation accuracy, and straight-through rate on the golden set. Pin OCR, layout, and LLM versions. When any pin changes, re-run the set in CI and reprocess production docs that failed or sat in the queue.

Common pitfalls

The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.

Frequently asked questions