A document retrieval system rarely fails because nobody chose a fashionable chunk size. It fails because the retrieved passage does not contain enough evidence to answer the actual question, or because useful evidence is buried inside a large amount of unrelated material. Chunking is therefore a design choice to test against the work people need to do.
There is no single size that suits an employee handbook, a product table and a long technical report equally well. The aim is to preserve meaningful units of information while making them retrievable and usable. A good experiment compares several approaches using the same source documents and realistic questions.
Understand the documents before splitting them
Inspect a representative sample of the source collection. Note headings, tables, lists, footnotes, repeated navigation and scanned pages. Ask which structures carry meaning. A table row without its column headings can become misleading even when the extracted words are technically correct.
Check extraction quality separately from chunking. Broken reading order, missing characters or duplicated page headers can damage retrieval before a split strategy is applied. Changing the chunk size may hide a symptom without correcting the underlying extraction problem.
Preserve source identifiers, document versions and locations during processing. A retrieved passage should lead back to the relevant document and section. Without that connection, a reviewer cannot easily verify whether the answer uses the current procedure or a superseded copy.
Build questions that expose boundaries
Collect questions from real users and add deliberately difficult examples. Include a short factual lookup, a question whose answer spans adjacent paragraphs, a table lookup and a request requiring evidence from two separate documents. Also include questions the collection cannot answer.
For each question, identify the evidence required for a correct response. This does not have to be a polished model answer. It can be a set of source passages, conditions and caveats that a knowledgeable reviewer expects to see.
Pay particular attention to boundary cases. A procedure may state the normal rule in one paragraph and the exception in the next. If splitting separates them, retrieval can produce a confident but incomplete answer. Such cases make a stronger test than questions that quote an exact heading.
Compare a few clear alternatives
Begin with a small number of strategies rather than a large parameter sweep. One baseline might split by fixed length with overlap. Another might respect section boundaries. A third could treat tables and lists as special structures while splitting longer prose separately.
Microsoft's document chunking guidance discusses fixed-size, structural and semantic approaches, including overlap where context needs to cross boundaries. That range is a useful reminder that chunking is more than selecting one number. The appropriate choice depends on the source material and retrieval task.
Keep other variables stable during the first comparison. Use the same document versions, embedding configuration, query set and retrieval count. If several parts change together, you will struggle to explain whether a result improved because of chunking or because another setting moved.
Inspect retrieval before judging prose
Look first at the passages returned for each question. Does the result contain the required evidence? Are key conditions missing? Are duplicate chunks taking up most of the available space? A fluent generated answer can make poor retrieval harder to notice.
Record both missed evidence and distracting evidence. Very small chunks may return a relevant phrase without its qualification. Very large chunks may contain the answer but overwhelm the context with unrelated sections. Neither problem is captured well by checking only whether a keyword appears.
Use a simple review sheet linking each question to expected evidence, retrieved chunks and reviewer notes. Keep examples of failures rather than reducing everything immediately to a score. Those examples will guide the next change more effectively than a percentage without explanation.
Handle overlap and parent context deliberately
Overlap can preserve information near a split, but it also creates repeated content. Inspect how often overlapping passages occupy multiple retrieval slots. Repetition may crowd out a different source that the answer needs, particularly when the context budget is limited.
Another option is to retrieve a smaller passage and then include its surrounding section when appropriate. This can combine precise matching with enough context for interpretation. The surrounding material must still respect access permissions and size limits; parent context is not a reason to fetch an entire restricted document.
Keep a stable link between the small passage and its parent. If documents are reprocessed, make sure those relationships are rebuilt consistently. Otherwise the system may retrieve a current fragment and attach an older or unrelated section as supporting context.
Give tables and exceptions their own checks
For tabular content, test whether row labels, column headings, units and notes remain connected. A value without its unit or product variant is not reliable evidence. Consider preserving a table as a structured object or generating a carefully controlled textual representation for retrieval.
Test lists that depend on their introductory sentence. A numbered sequence may apply only to one equipment type or account category. Splitting off the list can remove the condition that tells the assistant when it is relevant.
Exceptions deserve targeted questions. Ask about edge cases, exclusions and effective dates, then inspect whether the retrieval strategy brings those qualifications into view. A system that answers common questions well but consistently misses exceptions may create a false sense of reliability.
Choose based on operational trade-offs
Compare evidence coverage, answer quality, retrieval latency, indexing effort and context consumption. Do not assume the most elaborate strategy is best. A simpler structure-aware split may be easier to maintain and perform adequately for the collection you actually have.
Document the chosen settings alongside the test set and source versions. Explain why the team accepted the trade-off, including known weak cases. This makes future changes reviewable when new document types arrive or users begin asking different questions.
Revisit the choice after meaningful changes, not on an arbitrary daily schedule. A new catalogue format, a different embedding model or a larger source collection can justify another comparison. Keep the earlier baseline so the team can tell whether the change genuinely helped.
For a RAG knowledge system, the useful deliverable is not a universal chunk-size recommendation. It is a tested relationship between documents, questions and evidence. Start with a small representative collection, inspect failures closely and choose the structure that makes the right information available when it is needed.