RAG · AI Systems Architecture · Chunking
What RAG chunking actually is
Chunking is the step that determines how your documents are sliced before embedding. Get it wrong and your retriever surfaces accurate pieces of the wrong content.
Quick answer
What is the best chunking strategy for RAG? The best strategy depends on your document type. Structure aware chunking is optimal for structured documents like PDFs and financial filings. Semantic chunking wins on long narrative text. Recursive chunking is the safe default for mixed corpora. Layer contextual retrieval on top of any base strategy for the highest retrieval gain.
Every RAG pipeline has the same core problem: source documents are too long to embed as a whole. You cut them into chunks, embed each chunk, and at query time retrieve the chunks whose embeddings are closest to the query. But the quality of what you retrieve depends almost entirely on how you make those cuts.
A chunk that spans a natural topic boundary gives the embedding model conflicting signals and produces a vector that sits equidistant between two topics — useful for neither. A chunk that captures only half a sentence gives the language model insufficient context to generate a useful answer even when it is the right chunk. The strategy you use for cutting governs both failure modes.
Section 01 · Strategies
The four base chunking strategies
Each strategy encodes a different assumption about where topic boundaries live in your documents.
Structure aware chunking
Parses the document's native structure — HTML tags, PDF section headers, Markdown headings, spreadsheet rows — and cuts at those natural boundaries. A study of SEC 10-K filings found that element based chunking (using the filing's XBRL schema) halved chunk count compared to fixed size approaches while preserving each section's semantic unit. Best for: PDFs with clear heading hierarchies, structured filings, HTML documentation, and code files via an AST splitter.
Semantic chunking
Embeds sentences sequentially and inserts a chunk boundary wherever cosine similarity between adjacent sentence embeddings drops below a threshold. The boundary marks a real topic shift rather than an arbitrary token count, producing higher intra-chunk coherence. The cost: you run one embedding per sentence during preprocessing, which is more expensive than recursive or sentence based approaches. Best for: long narrative text, academic papers, and interview transcripts.
Sentence based chunking
Splits on sentence boundaries (period, question mark, exclamation point) and groups N sentences per chunk with an optional overlap window. Faster than semantic chunking and avoids the cuts that fixed size approaches make mid-sentence. Works well when sentences carry roughly equal information density. Best for: news articles, blog posts, and FAQ documents with short, self-contained paragraphs.
Recursive chunking
Applies a priority ordered list of separators — double newline, single newline, period, space — falling back to the next separator when a chunk would exceed the target size. Adapts to the document without requiring a structure parser. Most major frameworks ship this as the default. Best for: mixed corpora with heterogeneous formats and as a safe fallback when document type is unknown.
Section 02 · Decision Matrix
Matching strategy to document type
Document structure determines which strategy extracts the cleanest semantic units. Use this matrix before tuning chunk size or adding contextual retrieval.
| Document type | Recommended strategy | Why it wins |
|---|---|---|
| Structured filings (SEC, XBRL, legal) | Structure aware | Halves chunk count; boundaries match section semantics |
| Long narrative text (papers, reports) | Semantic | Cuts at genuine topic shifts; highest intra-chunk coherence |
| News and blog articles | Sentence based | Fast; respects natural prose rhythm |
| Code files and docstrings | Structure aware (AST) | Class and function boundaries are the semantic units |
| Mixed or unknown format | Recursive | Adapts via separator priority; minimal preprocessing required |
| Short Q&A / FAQ documents | Fixed size with overlap | Uniform information density; overlap prevents boundary splits |
The SEC filings result is worth anchoring on: structure aware chunking cut chunk count from roughly 1,800 to under 900 on a typical 10-K filing. Fewer chunks competing in the retrieval step means fewer irrelevant chunks in the retrieved set. That is a retrieval precision improvement you cannot buy by tuning your reranker.
Code is the other category that breaks every general-purpose approach. Splitting a Python file on double newlines produces chunks that start in the middle of a function or span two unrelated methods. An AST splitter — treating each class, function, or docstring as its own chunk — produces chunks a developer would recognize as a semantic unit. The retriever then surfaces the right function instead of the paragraph that happens to mention it.
Section 03 · Chunk Size
Why chunk size changes retrieval quality
Chunk size is a hyperparameter, not a default. The right value depends on your embedding model, your query length distribution, and your document structure.
Most teams pick 512 tokens because it is the default in the tutorial they followed. The problem is that 512 tokens is the right answer for roughly one third of corpora and wrong for the other two thirds. Here is why the number matters.
| Size (tokens) | Retrieval precision | Context to LLM | Best for |
|---|---|---|---|
| 64–128 | High | Thin — often missing context | Exact fact lookup in dense reference docs |
| 128–256 | High | Adequate for focused queries | Technical docs, short paragraphs |
| 256–512 | Good | Rich — good for synthesis | Most use cases; practical default |
| 512–1024 | Moderate | Can dilute the signal | Long-form reports when full sections matter |
| 1024+ | Low | Often too broad | Rarely useful; consider parent-child chunking |
The overlap window is the other setting most teams configure once and never revisit. Overlap prevents a retrieval miss when the answer straddles a chunk boundary — but too much overlap duplicates chunks in the index and confuses similarity scoring. A 10 to 15 percent overlap of your chunk size is a reasonable starting point. On a 256 token chunk, that is 26 to 38 tokens of overlap. If your evaluation shows high recall but declining precision as the corpus grows, the overlap window is often the reason.
Section 04 · Contextual Retrieval
Contextual retrieval: the highest-ROI upgrade
Contextual retrieval does not replace your base strategy. It adds a context window to every chunk at indexing time — before the chunk is embedded.
Quick answer
What is contextual retrieval? Contextual retrieval prepends a brief summary of the source document to each chunk using an LLM before embedding. The chunk then carries information about where it appears in the document and why it matters — context the raw chunk text alone does not contain.
The Anthropic team published results showing contextual retrieval reduced retrieval failure rates by 49 percent on their internal benchmark. The mechanism is simple: an LLM reads the full document and writes two to three sentences situating each chunk within that document. The augmented chunk is then embedded. At query time, the retriever has vectors that encode both the local meaning of the chunk and its global document context.
The tradeoff is indexing cost. You pay one LLM call per chunk at index build time. For a corpus of 50,000 chunks at roughly $0.0003 per call (using a small model like Claude Haiku or GPT-4o-mini), that is $15 at initial indexing. For most production corpora, that is noise. For very large corpora (10M+ chunks), apply contextual retrieval selectively — focus on the clusters with the highest query volume rather than the full corpus.
Chunk your documents with your base strategy
Apply the strategy that fits your document type from the decision matrix above. Contextual retrieval works with structure aware, semantic, sentence based, or recursive chunking — it is independent of the base approach.
Generate a context passage for each chunk
Prompt an LLM with the full document and the target chunk. Ask it to write 2 to 3 sentences situating that chunk within the document. Use the cheapest model that produces coherent output — this step does not need GPT-4.
Prepend the context before embedding
Concatenate the context sentences and the original chunk text, then embed the combined string. Store only the original chunk text in your vector store metadata for display to the user — the context is for the embedding only, not the generated answer.
Measure before and after with your evaluation suite
Use your existing RAG metrics (precision at K, NDCG, answer relevance) to confirm the gain. On narrative text corpora, expect a 20 to 50 percent improvement in retrieval precision. On already-structured corpora with clean structure aware chunking, the gain is typically 5 to 15 percent.
Section 05 · Common Mistakes
Mistakes that quietly degrade retrieval
Most chunking problems surface in evaluation metrics weeks after deployment, not during initial testing.
Using the tutorial default on a production corpus
512 token recursive chunking works well on tutorial datasets. Production corpora have heterogeneous document types. Run a three-way split (structure aware vs semantic vs recursive) on a representative 500-document sample, measure retrieval precision, and pick the winner. This single experiment takes two hours and is worth months of reranker tuning.
Chunking code files with a text splitter
A code search RAG that splits on double newlines will surface half a class or two unrelated functions for the same query. Add a language aware AST splitter for code file types. Most chunking libraries — LangChain, LlamaIndex, Chonkie — include one out of the box.
Setting overlap to zero on long documents
Zero overlap means an answer that straddles a chunk boundary is never retrieved whole. A 10 to 15 percent overlap window is low cost and prevents this failure mode. Do not skip it.
Feeding contextual retrieval context into the answer step
The context sentences generated during contextual retrieval are for embedding only. Do not pass them to the final LLM as retrieved context — they add meta-commentary the model has to work around and inflate the retrieved text length without adding answer-relevant information.
Never re-evaluating after corpus growth
A chunking strategy tuned on 10,000 documents may degrade as the corpus grows to 1 million and the topic distribution shifts. Build a chunking evaluation step into your quarterly RAG audit and re-tune when precision drifts more than five percentage points.
If you want to go deeper on measuring whether your chunking changes actually helped, the RAG evaluation metrics guide covers the precision at K, NDCG, and answer relevance metrics you need to run a credible before and after comparison.