How to Chunk Documents for RAG
Best chunk size for RAG is measured, not guessed: structure-aware splitting, overlap, parent-child chunking, tables, and late chunking on retrieval hit rate.
To chunk documents for RAG, split along document structure (headings, pages, tables), store small child chunks for retrieval and larger parents for generation, add modest overlap and title-path prefixes, and pick a size only after you measure retrieval hit rate — there is no universal best chunk size for RAG.
What you’ll build
- A per-source splitter: headings for wikis, pages for scans, rows-plus-header for tables
- Child chunks sized for retrieval and parent chunks sized for generation
- Overlap and title-path prefixes that survive embedding without bloating the store
- A late-chunking or contextual-embedding path for long policies where you need it
- A leaderboard of chunk strategies scored on hit rate at 5 and 10, not on intuition
Before you start
- 01A representative corpus slice (policies, tickets, PDFs, and at least one table-heavy source)
- 02A golden set of real questions with the source passage marked, not just the answer text
- 03An embedding model you are willing to freeze for the experiment
- 04A hybrid retriever (vector plus BM25) so chunking is not blamed for keyword misses
- 05Page, heading, and document-id metadata on ingest
Key takeaways
- 01
The best chunk size for RAG is a measurement on your golden set, usually in the 200–800 token band for prose, not a blog default.
- 02
Structure-aware splitting beats fixed windows; parent-child (small-to-big) beats sending the same span to both the index and the prompt.
- 03
Always prepend document title and heading path so the embedding is not a context-free fragment.
- 04
Tables, code, and scanned pages are different types: header repetition, AST splits, and page-level parents respectively.
- 05
Late chunking and contextual embeddings help long policies; they do not replace a measured splitter on mixed corpora.
Best chunk size for RAG is a measurement
Search for 'best chunk size for RAG' and you will find 256, 512, and 1024 offered as facts. They are starting points. Chunk size trades retrieval precision against generation context: too small and you retrieve a sentence without the exception clause; too large and the embedding averages two topics and misses both. The number that matters is hit rate on your questions, at the k you actually send to the model.
Run a bake-off. Freeze the embedding model and the hybrid retriever. Vary only the splitter. Report hit@5, hit@10, mean chunks per document, and tokens into the prompt. If a 800-token window wins on hit@10 but drowns the prompt, use parent-child: retrieve 250-token children, expand to the 800-token parent. That is how you chunk documents for embeddings without picking a religion.
How to chunk documents for embeddings with structure
Markdown and HTML already tell you where meaning breaks: headings, lists, tables, code fences. A recursive splitter that tries those separators first, then paragraphs, then tokens, keeps procedures intact. Prepend the title path — 'HR Policy > Leave > Parental leave' — to the chunk text before embedding. Without that prefix, two 'exceptions' sections from different policies collide in vector space.
Overlap exists to save sentences that straddle a cut. Ten to fifteen percent is enough for prose. Fifty percent overlap doubles the index and rarely pays for itself once you have parent expansion. Do not overlap table rows; repeat the header instead. Do not overlap code; split on function bounds.
- Wiki/SOP: heading-bounded sections, title path on every child
- Tickets: one chunk per comment plus a parent of the whole thread
- PDFs with text: page parent, paragraph children
- Scans: page-level chunks until OCR quality is proven
Parent-child chunking (small-to-big)
Parent-child chunking stores two granularities. The child is what you embed and search: a tight passage, a table slice, a single procedure step. The parent is what you put in the prompt: the full section, the whole table, the page. Hits on two children of the same parent should collapse to one parent so you do not waste the context window on duplicates.
This is the default we ship on mixed enterprise corpora. It costs a second lookup (child id → parent id) and a bit of storage. It beats sliding windows because the parent is a semantic unit, not an arbitrary 512-token slide that still bisects a table. If your vector database cannot store a parent id, you do not have a metadata problem later — you have one now.
Tables, code, and ugly PDFs
Tables fail standard splitters. A row without its header is garbage; a header without its units is a hallucination factory. Extract tables as objects. Index each row (or row-group) as a child with the header concatenated, and keep the full table as the parent. For wide financial statements, consider a dual index: row embeddings plus a caption/title embedding.
Code belongs to the AST: function, class, or markdown section, with the file path in the prefix. Scanned PDFs belong to page-level parents until layout OCR is good enough to trust headings. If a page is a screenshot of a dashboard, you are in multimodal RAG territory — do not pretend a text chunker will save it.
Late chunking and contextual embeddings
Late chunking embeds the long document first (within the model's window), then pools token embeddings into chunk vectors so each chunk still 'knows' the rest of the document. Contextual retrieval (prepend an LLM-written context sentence to each chunk before embedding) buys similar awareness at the cost of a generation pass at ingest. Both help long, self-referential policies ('the waiting period in section 4').
They are not free. Late chunking needs an embedding model that returns usable token vectors. Contextual retrieval adds ingest latency and can inject LLM mistakes into the index. Measure them against parent-child on the same golden set. On short tickets and FAQs, they usually lose to a cheap structure-aware split. On 40-page contracts, they often win.
Measure, then freeze
Your golden set must mark the passage, not only the answer. Otherwise you cannot tell a chunking miss from a generation miss. Stratify by source type. When a strategy loses on tables but wins on wikis, keep both splitters behind a source-type flag. Re-run when you change embeddings; mixed-model indexes are a different bug.
ReinforcedX implements chunking as part of a four-week RAG delivery in your cloud: extractors, parent-child index, and the bake-off harness. You own the corpus, the eval passages, and the runbooks. We do not sell a magic 512. We will leave you with a table that says which splitter won on which source, and 30 days on-call while you add the next repository.
Step-by-step build
- 1
Inventory document types
Tag each source as wiki/markdown, paginated PDF, table/export, ticket thread, or code, and refuse a single splitter for all five.
- 2
Mark structure before you split
Preserve heading paths, page numbers, table headers, and fenced-code bounds in metadata; if the extractor flattened them, fix extraction first.
- 3
Split children for retrieval
Cut along those bounds into roughly 200–400 token children for dense search, with 10–20% overlap only where a sentence would otherwise straddle the cut.
- 4
Keep parents for generation
Index a parent (section, page, or whole table) that you return when a child hits, so the LLM sees preconditions and headers the child does not hold.
- 5
Handle tables, scans, and code as special cases
Repeat column headers on every table child, use page-level parents for scans, and split code on AST boundaries rather than line count.
- 6
Score strategies on the golden set
Compare at least three configs (fixed 512, structure-aware, parent-child) on hit@5, hit@10, and answer faithfulness; ship the winner per source type.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
One chunk size for the whole company wiki
A 512-token default tears SOP steps from their preconditions and leaves table cells without headers. Split by document type, then measure.
Fixed character windows on markdown
Character count ignores headings, fenced code, and list boundaries. You retrieve half a procedure. Use a recursive splitter that respects those marks, then cap tokens.
Embedding the child and sending the child
Tiny chunks retrieve well and generate poorly. Parent-child chunking embeds the small unit and returns the surrounding parent at generation time.
Tables as flattened prose
A CSV dumped into 400-token windows loses the column names after the first split. Keep the header row on every table chunk, or retrieve the whole table as the parent.
Never re-measuring after an embedding change
A new embedding model changes what 'similar' means. Re-chunking is optional; re-embedding and re-scoring the golden set is not.