Retrieval-Augmented Generation is usually described as a retrieval problem. Better embeddings. Better rerankers. Better prompts. Better agents wrapped around the search step.
That framing hides the step that has already decided the answer.
Every chunk in your index is a fragment of reasoning carved out of a longer document. When the carving lines do not respect the document's own structure, your retriever can only hand the model halves of ideas — and there is no reranker, no agent, and no prompt that can reassemble an idea your index does not contain.
The ceiling on a RAG system's accuracy is set upstream of retrieval, at the step most teams treat as plumbing.
This post is about what changes when you stop treating chunking as plumbing and start treating it as the architectural problem it actually is.
1. The ceiling that doesn't respond to knobs
If you have shipped a RAG system into real use, you know the shape of the complaint. The answer is technically correct and factually incomplete. The information was in the corpus — you can find it yourself in under a minute — but the model's response only reached half of it. A reranker closes the gap on some queries and opens it on others. A larger embedder moves the problem around without solving it. Wrapping the whole thing in an agent helps, until the agent retrieves the same broken chunks more aggressively.
Every knob you try sits downstream of the actual problem. And because the failures look like retrieval failures, the instinct is to tune retrieval harder. The system keeps improving in every dimension except the one the user is actually measuring.
What changed for us, eventually, was accepting that most of the "hard" accuracy problems in RAG are not in retrieval at all.
2. Chunking is a global decision
The standard mental model treats chunking as a configuration problem. Pick a size. Pick an overlap. Pick a splitter. Move on. It is the first habit worth unlearning.
A chunk is not a window of tokens. It is a claim about where one idea ends and another begins. And that claim is global.
You cannot tell whether page 47 starts a new topic by looking only at page 47. You need to know what page 46 was doing, what page 50 is about, and what the document is trying to achieve overall. Any system that makes this decision with local signals — font size, whitespace, token count, even one model reading one page — will be confident and wrong on any document whose visual grammar does not match its assumptions.
Product documents style every page as important. Compliance documents bury section changes inside uniform prose. Financial reports lean on layout cues a text extractor will never see. Slide decks mix all three.
Fixing this at the page level is hopeless. The level itself is the problem.
3. "Using an LLM to chunk" is not agentic chunking
This is the distinction most writing on the topic skips.
A pipeline that calls a large model to propose boundaries on every page is an LLM pipeline. It is not an agentic one. A single very good model asked to do structure detection, text extraction, visual interpretation, and narrative authoring in the same call does all four worse than four specialists doing them separately.
Three reasons.
First, context dilution. A prompt that asks one model to do everything fills its attention with everything. The parts that matter most to the current decision have to compete with the parts that matter least.
Second, opaque reasoning. When one model decides a boundary, its justification is trapped inside a single generation. When boundaries are proposed by one role, verified by another, and reconciled by a third, why a boundary was chosen becomes an inspectable artefact at each step.
Third, wrong economics. The reasoning needed to spot a section break is not the reasoning needed to transcribe a dense table, and neither resembles the judgment needed to arbitrate between two draft values. Paying one top-tier model for all three is both more expensive and less accurate than paying each role on the tier that fits it.
Agentic chunking is the shift from one model, every task to a small team, each role optimal.
4. The specialist team
In our pipeline the structure decision is made by a small team of roles, each doing what it is best at.
A bulk extractor reads each page in isolation, in parallel, and produces a compact, verifiable digest: what is on the page, what kind of content it is, whether it appears to continue from the page before it.
A global reasoner never looks at pixels. It works from digests and short summaries, and it is responsible for thinking about the document as a whole.
A visual auditor answers plain-language questions about specific pages when the text-level evidence is ambiguous.
A reconciler has one job: resolve disagreements between passes into a single auditable plan.
None of them is asked to do the job of another. The quality improvement that follows does not come from a cleverer individual model. It comes from the division of labour itself.
5. Shape first, detail later
For short documents, the global reasoner can see the shape in one pass. For longer ones — hundreds of pages, thousands — no single model should be asked to reason over every page at once. Cost aside, focus degrades. The output gets vaguer, not sharper.
So we refuse to try.
The reasoner first reads a compressed representation of the entire document — a compact summary per page, drawn from the extractor's digests. From that view alone, it proposes tentative section boundaries across the whole document. This pass is fast, cheap, and honest about its uncertainty: it nominates, it does not commit.
A second pass then breaks the document into overlapping windows. Fresh reasoners work each window in parallel. Their job is not to redo the first pass — it is to audit the first pass's nominations inside their slice, shift the ones that look misplaced, and surface any boundaries the shape-level view missed. When the text-level evidence is ambiguous, they ask the visual auditor about specific pages.
A final pass reconciles every window's report with the original shape-level view into a single plan for the document.
The insight is architectural, not algorithmic. You do not need to give any one model the full document. You need to give each model the shape at the level it can reason about.
6. Text reasoners, vision workers
The part of this architecture that surprises people most is how much of the work is done by passing text between agents rather than pixels.
When the text-level reasoner is uncertain about a page, it asks a vision-specialist a plain question: "Does this page visually begin a new topic, or is it a continuation?" The vision specialist looks at the page image and answers in words. The reasoner never sees the pixels.
Two things fall out of this.
Context stays affordable. Long documents stay text-only at the reasoning layer. Visual verification happens out of band, at the resolution vision work actually needs, by a model chosen for exactly that.
Visual questions become arbitrarily cheap. The reasoner can ask dozens of targeted questions about specific pages without paying the cost of carrying images in its own context window.
This is not an optimization. It is the difference between asking one model to think like five specialists at once, and actually hiring five specialists.
7. Ground truth, per claim
Authoring a section's prose is a separate concern from deciding its boundaries, and it deserves its own ground-truth arrangement.
When the agent writing a section spots a probable disagreement between its reading of a page and the extractor's draft — a number, a date, a line of code, a financial figure — it does not guess. It flags the disagreement, keeps writing, and a focused arbitration pass resolves each flagged case against the original, native-resolution source. If even that cannot resolve cleanly, the system falls back to the extractor's value rather than the author's guess, on the principle that the extractor saw the page at full fidelity while the author did not.
This "escalate only the disputes, resolve them on ground truth" shape is a small fraction of overall cost and a disproportionate share of the accuracy gain. It replaces an all-or-nothing quality tradeoff with a per-claim one — and for the documents that matter most (financial filings, legal contracts, technical specifications), per-claim arbitration is the difference between "usually correct" and "answerable under audit."
8. Reconciliation is the work
Finding candidate boundaries is easy. Three different passes will happily give you three different opinions. The real engineering problem is what to do when they disagree.
The rule we landed on is specific enough to be auditable. When two windowed reviewers disagree about the same page, we prefer the reviewer whose window placed that page closest to its centre — the voice looking at the evidence most directly, rather than at its edge. When the shape-level nomination disagrees with every window, the windows win. When every source agrees, the page passes without argument.
This is not the glamorous part of the system. It is the part that turned the pipeline from "often correct" to "reliably correct." An architecture with explicit rules about whom to trust when is worth more than an architecture with cleverer components and no rules at all.
9. Determinism wraps the agents
Agentic systems have an earned reputation for being hard to reproduce and hard to trust. The fix is not to make them less agentic. It is to wrap their outputs in deterministic guarantees.
Every expensive artefact in the pipeline — per-page extractions, tentative boundaries, window reports, section drafts — is persisted atomically and resumable. A crash on page 740 of an 800-page document does not cost you pages 1 through 739. A deterministic post-check runs before any document is declared finished, and enforces invariants no model is allowed to relax: every page belongs to exactly one section, no page appears in two, no section references a page that does not exist, every output carries a complete, machine-readable header.
Agents propose. Deterministic checks dispose.
That boundary is where most of the trust in the system lives.
10. One pipeline, every format
Documents do not arrive in one format. Real ingestion has to accept PDFs, presentations, spreadsheets, word-processor files, CSVs — and it has to do so without shipping a different pipeline per extension.
Our approach is to collapse every input to one internal contract as early as possible. Structured data with no visual component short-circuits the model stack entirely — there is nothing a reasoner can add to a clean table that deterministic parsing cannot. Everything with meaningful layout — slides, long-form documents, spreadsheets with embedded imagery — converges to a single rendered form on the way in, and the entire chunker downstream is unaware of the original file type.
The operational win is obvious: one code path, one set of tests, one observability surface. The architectural win is bigger. The chunker never accumulates special cases, and every improvement to it lifts every format at once.
11. Why this compounds downstream
Chunking's effects compound through everything that comes after it.
Embeddings are only as good as the units you embed. A coherent section produces a coherent vector. A half-idea produces a vector that drifts in whichever direction the half fell, and every retrieval from that point on is slightly off-axis.
Routing sharpens. When every chunk carries keywords and a summary written for that section — not a page's worth of miscellany — the lightweight model picking which document to search stops guessing and starts deciding.
Attribution becomes honest. When the retrieved chunk is a real section, the answer can cite it as one. When it is an arbitrary token window, attribution devolves into "multiple sources suggest."
Latency follows. Fewer wasted neighbours per query. Fewer documents searched per collection. Smaller, more confident result sets reaching the final model.
You notice the improvement in retrieval latency, in routing accuracy, and above all in the complaints you stop hearing. The questions that used to come back incomplete start coming back complete.
From the bench
Argument is one thing; measurement is another. We tested the architecture above against a flat dual-page chunker on a 25-page security assessment — same embedder, same retriever, same generation model, same top-k. Only the chunking step differed.
Thirty-five hand-labelled questions covered the full mix: factual lookups, table cells, multi-hop synthesis across pages, boundary cases that straddle section breaks. A second, harder pass added twenty-two pure table-extraction questions designed to stress how each strategy survives tabular structure.
The pattern that came back is the one the architecture predicts.
The MRR lift (+0.144) is visible at every k. By rank 5, agentic finds the gold page on every question; dual-page is at 89%. Neither system fabricates: faithfulness is essentially perfect for both. The difference shows up in what each retrieves alongside the gold page — agentic averages 3.9 on-topic chunks per query, dual-page 2.5. Half of dual-page's top-5 is noise the model has to read past.
Where the gap matters most is exactly where the upstream argument predicted. Three question types tie cleanly: multi-hop, boundary, and easy roll-ups — once anything relevant lands in the top-5, the model stitches the answer. Two types do not tie.
Factual single-page lookup jumps from 27% to 55% R@1 — dual-page's two-page window pairs the wrong sibling page often enough to demote the gold page out of rank 1. Table extraction jumps from 70% to 100% — agentic's section frontmatter, with its keywords and short summary, gives the embedder the query-shaped surface area a flat token window has no way to manufacture. The 22-question tables-only follow-up reproduces the same shape: +22 points at R@1, perfect generation correctness against 95.5% for dual-page.
The compounding §11 anticipated is empirical, not rhetorical: better seams produce better embeddings produce better-ranked retrievals produce more on-topic context produce a model that has the whole idea, not half of it. The architecture's behaviour and the architecture's measurements describe the same phenomenon.
12. How to tell your chunker is the bottleneck
If you suspect chunking is the ceiling you have hit, the signals are specific.
- Your answers are technically correct and factually incomplete. The model's response is never wrong; it is just never whole.
- Your attribution sounds hedged. "According to multiple sources..." is a symptom of retrieving fragments that cannot individually stand behind a claim.
- Rerankers help less than you expect. A strong reranker reorders what retrieval returned; it cannot reassemble an idea that was split across non-adjacent chunks.
- Bigger embedders move the problem, not solve it. Dimensionality and benchmark quality both go up; the gap on your real queries does not close.
- Retrieval latency grows with the corpus in ways you did not predict. Low-signal chunks force larger neighbour searches to compensate.
- Section counts don't match the document. A 300-page manual becoming 280 sections or 4 sections is the same bug wearing different sizes.
- Users ask follow-ups that should not have been needed. The answer to question A gave them half of what they needed, so they ask B to cover the other half.
Any one of these, alone, is ambiguous. Three of them together is a chunking problem.
13. What this buys you
We built what we built because real answers were being lost in the seams between pages. The work is not finished — no chunking system ever is — but the ceiling has moved, and the shape of the problem no longer hides.
Agentic chunking is not "using an LLM to chunk." It is a design pattern with a specific grammar: global reasoning over compressed representations, local verification in overlapping windows, visual work delegated to visual specialists, disputes arbitrated on ground truth, reconciliation made explicit, determinism wrapping the whole thing.
What it buys is not better embeddings. It is a document you can reason over.