| # Chunking and retrieval data |
|
|
| ## Retrieval-model data |
|
|
| Daecore fine-tunes retrieval models on passages prepared for its document |
| workflow. A passage is the exact chunk the model reads: a section, excerpt, |
| table fragment or code block. It can supply useful partial evidence without |
| answering the whole question. |
|
|
| **Source documents → structural chunks → graded query–passage pairs → retrieval training** |
|
|
| | Ingredient | What it contributes | |
| |---|---| |
| | Mostly synthetic workplace documentation | Notes, procedures, decisions and technical material across generated project scenarios | |
| | One person's project documents, much of them AI-written | Additional examples, not a representative sample of other users' workspaces | |
| | Frozen structural chunks | The retrieval units that Gemma embeds and Ettin ranks; some retain earlier parser outputs | |
| | Model-written questions and graded evidence | Several useful passages per question, with decisive and partial evidence distinguished from merely related text | |
| | Public evidence annotations used by Gemma v2 | HotpotQA paragraphs, MultiDoc2Dial grounding passages and FinQA text/table evidence | |
|
|
| Public retrieval data also comes in passages; it is not uniformly made of full |
| documents. The distinction is their source and preparation. Each model applies |
| its tokenizer and input limits after chunking, so a chunk boundary and a model's |
| truncation limit are separate constraints. |
|
|
| This is deliberate specialization of already capable upstream models. The |
| private comparisons measure gains on this workflow, while public benchmarks |
| show the general-retrieval cost. Daecore accepted that tradeoff for its intended |
| use; the cards report both sides. Generated questions and shared preparation |
| and judging processes do not establish gains for every real user or agent. |
|
|
| Training and evaluation use frozen passage snapshots. Current chunking fixes |
| do not retroactively change those texts or transfer their relevance labels to |
| different cuts. |
|
|
| ## Chunking methods |
|
|
| - **Heading-based splitting:** use the document's sections; oversized sections |
| can split further at their subheadings. This follows the author's structure. |
| - **Title and heading context:** add the document title and relevant heading |
| path to section chunks so their subject stays clear. Smaller searchable pieces |
| retain that metadata but do not automatically repeat it inside every piece. |
| - **Boundary-aware cuts and overlap:** prefer a nearby sentence end, paragraph |
| break or line break. Ordinary prose windows repeat up to 200 characters across |
| cuts to carry nearby context; oversized structural continuations have their |
| own framing rules. |
| - **Code-block and table handling:** keep blocks together when they fit. Split |
| oversized tables with repeated column headers and oversized code blocks with |
| reopened and closed code fences, so continuations remain readable. |
| - **Record-aware splitting:** preserve row, field or entry context for CSV, |
| JSON, JSONL and dated entries. Oversized records are divided with identifying |
| labels rather than losing their remaining content. |
| - **Paragraph packing for imported memory files:** group whole paragraphs into |
| bounded pieces so excerpts usually begin at a paragraph. A single oversized |
| paragraph falls back to structural splitting. |
|
|
| These methods work together: make supported text and record chunks fit first, |
| then create bounded searchable pieces for sections that are still too large. |
| Both operations use the same structural cutter. |
|
|
