How to Build Multimodal RAG

Key takeaways

• 01

  Multimodal RAG is for answers that live in pixels: scans, charts, stamps, and screenshots.

• 02

  Visual document retrieval over pages complements BM25 and text embeddings; it does not replace them for error codes and IDs.

• 03

  Keep bounding boxes at ingest so citations can highlight a figure, not only a filename.

• 04

  Send the retrieved page image to a VLM when the claim is visual; fall back to text for born-digital prose.

• 05

  Evaluate on questions whose gold evidence is a figure, or you will ship a text RAG with extra cost.

If every answer lives in a well-exported wiki, you do not need multimodal RAG. You need better chunking. Build this system when users ask about invoices, engineering drawings, slide charts, stamped forms, or screenshots of consoles — documents where the layout is the meaning.

The product contract is the same as text RAG: retrieve, ground, cite, refuse on empty. The difference is the evidence object. Instead of a 400-token passage, the evidence is a page image plus a box. If you cannot open that crop in the UI, you built a captioning demo.

Ingest at page granularity. Render PDFs at a consistent DPI, deskew scans, and store a lossless-enough web image for the UI plus a model-sized thumbnail for the VLM. Run layout detection so tables and figures have IDs. OCR is a layer, not the source of truth: keep it for keyword search and for generators that cannot take images.

Metadata is still the enterprise deal. Page number, document ACL, source URL, figure id, and whether the page is photo, chart, or text. Query-time permission filters apply to images. A VLM that has seen a restricted page can leak it in paraphrase, the same as a text model.

Visual document retrieval embeds the page (or patches of the page) so a query like 'which bar is Q3 APAC' can match a chart without waiting for OCR to spell APAC. ColPali-style late interaction and similar page encoders are the current default for scanned PDFs. For product photography, a separate image encoder and a type flag keep those vectors out of the contract index.

Never drop BM25. Users still paste invoice numbers and error codes. Fuse visual ranks with text ranks, then re-rank the top pages. A cross-encoder over page text is cheap; a VLM re-ranker is accurate and expensive — use it on the last 20 pages, not the whole corpus.

• Text-native docs: existing hybrid RAG, no page images in the prompt
• Scanned PDFs: page embeddings + OCR BM25, VLM on the hit pages
• Charts/slides: page or figure crops into the VLM, caption as a fallback
• Photos: dedicated image index, do not mix with 300-dpi text pages

When the hit is a chart or a stamp, send the image. When the hit is a paragraph of OCR, send text and save tokens. Mixed questions get both: image plus the text layer so the model can quote a number and still see the axis. Cap pages per turn; five full pages will blow the cost budget and the model's attention.

Citations must be clickable: document, page, figure id, and a box. The UI opens the crop. A faithfulness check compares the claimed number to OCR in that box or to a second VLM pass that answers 'is this claim on this crop?' Hallucinated page numbers are worse than no citation — they train users to trust a map that is wrong.

Build a golden set where at least half the questions are unanswerable from OCR alone. Score retrieval (right page in top k), citation accuracy (page and box), and answer correctness. A text-only RAG will look strong on the prose subset and fail the rest; that split is the point of the dashboard.

Watch cost. Visual embeddings and VLM generation are not the same invoice as a 7B text extract. Route text-native questions away from the VLM. Cache page embeddings. Re-embed only changed pages. If you cannot name cost per visual question, you will meet it in month two.

ReinforcedX implements multimodal RAG in the client's VPC: ingest, dual index, citation UI, and the visual golden set. Model-agnostic on both embedders and VLMs. You own the page images, boxes, evals, and runbooks. No token markup.

Four weeks covers a pilot corpus, not the entire historical scan archive. Week 1 picks the pixel-native sources, week 2 indexes a slice, week 3 shadow-modes next to text RAG, week 4 hands over. Thirty days on-call is for the first bad scanner batch, which will arrive.

Step-by-step build

1. 1

   Separate pixel-native documents from text-native ones

   Route born-digital HTML and DOCX through the text RAG path; route scans, slides, photos, and chart-heavy PDFs through a page-image pipeline.

2. 2

   Ingest pages with layout, not just strings

   Render each page to an image, run layout OCR for a text layer, and store bounding boxes for words, tables, and figures next to the page id.

3. 3

   Build visual document retrieval

   Embed pages or patches with a document-vision model (or ColPali-style late interaction) and keep a parallel BM25 index on the OCR/text layer.

4. 4

   Fuse text and visual hits

   Reciprocal-rank-fuse page hits from both indexes, filter by the user's ACL, and cap the prompt at a small number of pages plus their text layers.

5. 5

   Generate with the page in context

   Pass page images to a VLM for chart and stamp questions; pass text layers for prose; require a page number and optional bounding box on every claim.

6. 6

   Evaluate on visual evidence

   Golden questions must point at a page and box; fail the case if the model cites a different page or invents a number not in the crop.

Common pitfalls

The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.

Frequently asked questions