What Is Chunking in RAG? Splitting Documents for Retrieval

Retrieval systems do not search documents; they search chunks. How documents are split before indexing determines what a query can match, what the model reads, and how much context fits. Chunks that are too small lose meaning; chunks that are too large bury the relevant sentence and waste tokens; boundaries that cut through tables or sections destroy information. This explainer covers why chunking matters, the strategies available, and how to choose, drawing on FISTA Solutions' AI enablement practice. The retrieval architecture is in what is rag and the system build in how to build a rag system.

What is chunking?

Chunking is the step in a retrieval pipeline that divides source documents into passages, each of which is embedded, indexed, and retrievable on its own. A chunk carries its text, metadata such as source, section, and date, and often a reference to its parent document. At query time, the most relevant chunks are retrieved and passed to the model as context. Everything downstream depends on chunks being findable and meaningful.

Why does chunk size matter?

Chunk size	Retrieval behavior	Context behavior	Risk
Very small	Precise matches on specific facts	Little surrounding context	Fragments that mislead without context
Small to medium	Good match precision, some context	Several fit in context	Ideas split across boundaries
Medium to large	Broader topical matches	Fewer fit; more tokens per answer	Relevant sentence buried in noise
Very large	Coarse matches	Few fit; high cost	Diluted relevance, wasted tokens

Context and token costs are in what is a context window and llm token cost explained.

What chunking strategies exist?

• Fixed-size windows: split by token or character count with optional overlap; simple, structure-blind.
• Structure-aware: split along headings, paragraphs, list items, and tables; keeps units intact.
• Semantic: split where topic shifts, detected by embedding similarity between sentences.
• Hierarchical or parent-child: index small chunks for matching but return larger parent sections for context.
• Document-type specific: contracts by clause, code by function, transcripts by speaker turn, tables as whole units with headers.
• Late or contextual chunking: embed chunks with document-level context included so isolated chunks retain meaning.

Search mechanics are in how to build a semantic search engine.

How do overlap and parent-child strategies preserve context?

Overlap repeats boundary text so split ideas appear whole in at least one chunk. Parent-child retrieval matches on small, precise chunks and returns the enclosing section, giving the model context without diluting retrieval. Both trade storage and duplication for fewer broken passages.

Why does metadata matter?

Source, section title, page, date, author, permissions, and document type attached to each chunk enable filtering before search, permission enforcement, recency weighting, and precise citations. Prepending section titles to chunk text also improves embedding quality. Permission-aware retrieval is in enterprise rag cost.

How does chunking interact with embeddings and search?

Embedding models have input limits and represent focused passages better than sprawling ones; keyword search benefits from chunks that keep terms together; reranking can rescue imperfect chunking by scoring candidates against the query. Chunking, embeddings, hybrid search, and reranking are tuned together. Hybrid retrieval is in what is hybrid search and reranking in what is a reranker.

How should chunking be evaluated?

Build a golden set of real questions with known relevant passages, measure retrieval recall and precision under each strategy, then measure end-to-end groundedness and answer quality, because retrieval and generation interact. Iterate on the strategy with data rather than intuition. Groundedness measurement is in what is groundedness in ai and evaluation practice in what is an eval in ai.

What are common chunking mistakes?

Fixed windows that cut tables and code, chunks without metadata or section context, one size for every document type, no overlap where ideas span boundaries, chunks too large for the embedding model, and never re-evaluating after documents or queries change. Retrieval improvements are in how to improve rag accuracy.

What does good chunking look like in practice?

A knowledge assistant over policy manuals, contracts, and support tickets uses structure-aware chunking per document type: policy sections with titles prepended, contract clauses as units, tickets by message with thread metadata, and tables kept whole with headers. Small chunks are indexed for matching with parent sections returned for context, metadata enables filtering by department and date, and a golden set confirms recall improves over fixed windows while tokens per answer fall. Vector infrastructure is in how to build a vector search service.

How FISTA Solutions approaches chunking

FISTA Solutions chunks by document type with structure awareness, attaches rich metadata, uses parent-child retrieval where context matters, tunes chunking with embeddings, hybrid search, and reranking together, and evaluates every strategy on client golden sets. The AI enablement practice delivers retrieval platforms, AI agents draw on well-chunked corpora, and forward deployed engineers embed with client knowledge teams. The record behind the approach is 150+ projects with 99.9% uptime.

To get retrieval right from the first cut, message FISTA on WhatsApp, or read what is hybrid search for the search layer that reads the chunks.