Update embeddinggemma-300m-memory-ft-v2 documentation

README.md

@@ -22,7 +22,15 @@ tags:
 
 An English embedding model for finding the passages that answer a question in notes, procedures, decision records and technical documentation. It is a fine-tune of [Google's EmbeddingGemma-300M](https://huggingface.co/google/embeddinggemma-300m), packaged as one ONNX graph with pooling and normalization built in, so it runs locally without PyTorch.
 
-On Daecore's **970-query dense-retrieval panel**, v2 raises nDCG@10 from **0.4506 to 0.5748** and precision@10 from **47.40% to 61.18%** compared with upstream Gemma. Both models search the same 82,719 passages, with reviewed labels through rank 20. V2 also recovers more than half of the first Daecore fine-tune's measured loss on FiQA and SciFact.
+| Stronger private retrieval | Less forgetting | Local deployment |
+|---|---|---|
+| nDCG@10 **0.4363 → 0.5574**; precision@10 **47.40% → 61.18%** | Recovers **more than half** of the first fine-tune's loss on FiQA and SciFact | **300M parameters**, ONNX, CPU / CUDA / Vulkan |
+
+The private comparison covers **970 queries and 82,719 passages**, using a
+shared reference of **127,665 graded query–passage pairs** and reviewed results
+through rank 50. Daecore accepted some general-retrieval loss for stronger
+evidence retrieval on its document workflow. The tables below show both the gains
+and the remaining public-benchmark gaps.
 
 Embed queries and passages with the same revision, and re-embed stored passages when switching to v2.
 
@@ -80,6 +88,13 @@ chooses the returned prefix. Chunking and candidate coverage therefore affect
 the final results alongside embedding quality. The package also works as a
 standalone embedder with other retrieval systems.
 
+Coverage at **50 candidates** measures what is available to a reranker. Hit@50
+asks whether that pool contains any useful evidence; known-positive recall@50
+measures how much of the judged useful evidence it contains. A reranker can
+improve the order but cannot recover a passage outside its pool. Gemma's dense
+top 50 describes a dense-only pipeline. Daecore's actual ceiling depends on
+the 50 candidates admitted after Gemma and BM25 are fused and deduplicated.
+
 ## Evaluation
 
 ### Gemma alone: dense retrieval on Daecore data
@@ -89,22 +104,32 @@ exact dense retrieval, the same frozen query forms and matched 128/1,024-token
 input limits. BM25 and Ettin do not contribute to these results. Each cell
 reads upstream → **Daecore v2**.
 
+**Reference:** 127,665 graded query–passage pairs · shared top-50 reference
+(September 2026) · abstentions excluded. This pooled reference covers both
+models and is not an exhaustive labeling of the corpus.
+
 | Cutoff | Hit | Precision | Known-positive recall | nDCG |
 |---|---:|---:|---:|---:|
-| 1 | 53.40% → **64.74%** | 53.40% → **64.74%** | 1.94% → **2.46%** | 0.4695 → **0.5421** |
-| 3 | 72.99% → **79.38%** | 50.83% → **63.08%** | 5.43% → **6.95%** | 0.4516 → **0.5524** |
-| 5 | 79.48% → **84.74%** | 49.95% → **62.66%** | 8.53% → **11.09%** | 0.4457 → **0.5586** |
-| 10 | 84.85% → **88.14%** | 47.40% → **61.18%** | 15.77% → **20.82%** | 0.4506 → **0.5748** |
-| 20 | 88.76% → **90.52%** | 43.94% → **58.11%** | 28.08% → **38.65%** | 0.4736 → **0.6065** |
+| 1 | 53.40% → **64.74%** | 53.40% → **64.74%** | 1.44% → **1.79%** | 0.4592 → **0.5313** |
+| 3 | 72.99% → **79.38%** | 50.83% → **63.08%** | 3.94% → **5.03%** | 0.4414 → **0.5407** |
+| 5 | 79.48% → **84.74%** | 49.95% → **62.66%** | 6.17% → **8.06%** | 0.4346 → **0.5453** |
+| 10 | 84.85% → **88.14%** | 47.40% → **61.18%** | 11.18% → **15.07%** | 0.4363 → **0.5574** |
+| 20 | 88.76% → **90.52%** | 43.94% → **58.11%** | 19.61% → **27.56%** | 0.4526 → **0.5819** |
+| 50 | 93.20% → **94.43%** | 38.60% → **50.45%** | 41.12% → **57.92%** | 0.5107 → **0.6450** |
 
 ![Gemma v2 dense retrieval compared with upstream](figures/gemma-comparison.svg)
 
-Every top-20 position has a relevance grade or a declared judging abstention;
-there are no missing-label ranges. All 970 queries remain eligible at every
-cutoff. At rank 20, 56 upstream positions and 53 v2 positions are excluded,
-without pulling in deeper results. Precision pools retained positions;
-known-positive recall averages the 911 queries with judged useful evidence.
-The corpus is not exhaustively labeled.
+Every top-50 position is graded or explicitly excluded as ungradable, with no
+missing-label ranges. All 970 queries remain eligible at every cutoff. At rank
+50, 164 upstream positions and 167 v2 positions are excluded without backfill.
+Precision pools retained positions; recall averages the 925 queries with judged
+useful evidence. The corpus is not exhaustively labeled.
+
+At 50, v2 finds useful evidence for **94.43%** of queries and retrieves **57.92%**
+of known useful passages, versus 93.20% and 41.12% upstream. The actual hybrid
+pool below has lower coverage on this panel, despite stronger early precision
+after reranking. Adding BM25 and capping the fused list at 50 can displace dense
+candidates; the two pools should not be treated as interchangeable.
 
 Grades 2–3 count as useful. nDCG uses gains 0, 1, 3 and 7 against the same
 reviewed reference for both models, averaging all 970 queries. Rankings from
@@ -131,61 +156,80 @@ V2 is below upstream on all five datasets, most on FiQA (−0.0273). The first D
 
 The model was trained at 768, 512, 256 and 128 dimensions. To use a smaller
 width, keep the first dimensions and normalize the shortened vector again.
-Use the same width for queries and passages. The table measures **v2 alone**,
-re-ranking the five public corpora from its cached FP32 vectors; the 768-wide
-scores exactly reproduce the original per-query nDCG@10 results.
-
-| Dimensions | Bytes per FP32 vector | SciFact | FiQA | NFCorpus | SciDocs | ArguAna |
-|---|---:|---:|---:|---:|---:|---:|
-| 768 (default) | 3,072 | 0.7783 | 0.4468 | 0.3890 | 0.1852 | 0.6259 |
-| 512 | 2,048 | 0.7841 | 0.4374 | 0.3876 | 0.1793 | 0.6137 |
-| 256 | 1,024 | 0.7757 | 0.4091 | 0.3660 | 0.1708 | 0.5987 |
-| 128 | 512 | 0.7488 | 0.3668 | 0.3311 | 0.1484 | 0.5533 |
-
-At 512 dimensions, each raw FP32 vector uses one-third less storage; at 256,
-two-thirds less. These byte counts exclude index overhead and compression.
-Shortening vectors does not reduce the encoder's forward-pass cost. Lower
-widths lose quality on most of these panels; Daecore uses 768, and its private
-and full-pipeline scores in this card apply only to that width.
+Use the same width for queries and passages. Each quality column below is
+**v2 nDCG@10**, computed from exact rankings of cached FP32 vectors. The Daecore
+column covers the same 970-query dense panel with reviewed top-10 labels at
+each width, against the same **127,665-pair reference**. The 768-wide results
+reproduce the corresponding full-width scores.
+
+| Dimensions | Bytes per FP32 vector | Daecore | SciFact | FiQA | NFCorpus | SciDocs | ArguAna |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| 768 (default) | 3,072 | 0.5574 | 0.7783 | 0.4468 | 0.3890 | 0.1852 | 0.6259 |
+| 512 | 2,048 | 0.5605 | 0.7841 | 0.4374 | 0.3876 | 0.1793 | 0.6137 |
+| 256 | 1,024 | 0.5292 | 0.7757 | 0.4091 | 0.3660 | 0.1708 | 0.5987 |
+| 128 | 512 | 0.5035 | 0.7488 | 0.3668 | 0.3311 | 0.1484 | 0.5533 |
+
+At 512 dimensions, each raw FP32 vector uses one-third less storage, with
+slightly higher Daecore nDCG@10 in this measurement and lower scores on four
+of the five public panels. At 256, vector storage falls by two-thirds with a
+larger quality cost. These byte counts exclude index overhead and compression;
+shorter vectors do not reduce encoder inference cost. Daecore uses 768, and
+smaller widths have not been qualified through the full hybrid pipeline.
 
 ### Full pipeline: Gemma v2 + BM25 + Ettin
 
-This separate replay measures the complete retrieval path: semantic and lexical
-search, reciprocal-rank fusion, then Ettin reranking. It covers **all 970
-development queries over 82,719 passages**, including queries with no known
-useful evidence. The table reports fixed return depths on CUDA FP16 and is
-shared by the Gemma and Ettin cards; it is not either model's standalone score.
+This replay measures semantic and lexical search, reciprocal-rank fusion, then
+Ettin reranking over **970 development queries and 82,719 passages**. It includes
+queries with no known useful evidence. Both model cards share this CUDA FP16
+table; it is not either model's standalone score.
+
+**Reference:** 970 queries · 82,719 passages · **127,665 graded query–passage
+pairs** in the shared top-50 reference (September 2026). Abstentions are excluded.
 
 | Return depth | Hit | Precision | Known-positive recall | nDCG |
 |---|---:|---:|---:|---:|
-| 1 | 72.33% | 72.33% | 3.46% | 0.6646 |
-| 3 | 85.36% | 71.58% | 9.52% | 0.6577 |
-| 5 | 88.45% | 70.28% | 15.38% | 0.6555 |
-| 10 | 89.90% | 67.18% | 28.03% | 0.6611 |
-| 20 | 91.03% | 61.79% | 48.85% | 0.6841 |
-
-Grades 2–3 count as useful. Depth 1 uses the same 965 queries for both
-providers: five are omitted because at least one first result was ungradable.
-At depths 3–20, Hit and nDCG average all 970 queries. Precision pools retained
-positions. Recall averages known-positive queries (890 at depth 1; 895 at
-depths 3–20) and measures judged passages, not every useful passage in the corpus. A shared judging-exclusion set removes 52 positions
-from the top 20 without backfilling. Daecore's selector chooses a prefix of
-3–20; these fixed-depth scores do not evaluate that choice. The
-[evaluation companion](evaluation/README.md) provides the
-records, exclusions and CUDA/Vulkan comparison; the
-[pipeline overview](https://huggingface.co/Daecore) explains the retrieval path.
+| 1 | 72.33% | 72.33% | 2.12% | 0.6458 |
+| 3 | 85.36% | 71.58% | 5.89% | 0.6374 |
+| 5 | 88.45% | 70.28% | 9.35% | 0.6326 |
+| 10 | 89.90% | 67.18% | 17.04% | 0.6315 |
+| 20 | 91.03% | 61.79% | 29.72% | 0.6417 |
+| 50 | 93.09% | 40.43% | 45.98% | 0.6051 |
+
+**The hybrid candidate ceiling is 93.09% Hit@50 and 45.98% known-positive
+recall@50.** These are coverage measures for the actual 50-candidate pool;
+reranking cannot change them. The row at 50 exposes that pool, while Daecore
+returns a selected prefix of 3–20. Fixed-depth scores do not evaluate the
+selector's choice.
+
+Grades 2–3 count as useful. At depth 1, both providers use the same 965 queries;
+five are omitted because a first result was ungradable. Depths 3–50 retain all
+970. Precision pools judged positions; recall averages the 925 queries with
+known useful evidence (920 at depth 1). Abstentions remove 52 positions at
+rank 20 and 93 at rank 50 per provider, without drawing in deeper passages.
+The corpus is not exhaustively labeled.
+
+Dense retrieval and this replay use the same expanded relevance reference.
+Earlier Hit and precision are unchanged; recall and nDCG were recomputed as
+more relevant passages became known. The [evaluation companion](evaluation/README.md)
+provides the records, exclusions and CUDA/Vulkan comparison; the
+[pipeline overview](https://huggingface.co/Daecore) explains retrieval and selection.
 
 ## Training data and objective
 
-Private supervision teaches passage relevance for agent retrieval. A question can have several useful passages; decisive evidence is graded above useful but incomplete evidence, and similar but unhelpful passages are explicit negatives. The documents are generated organizational material plus one person's project documentation, much of it AI-written. Four models from three provider families wrote the queries without seeing a designated answer, and model judges graded relevance under a common rubric. Construction, chunking, query writing and judging follow shared processes, so the data does not stand in for other users' workspaces.
+Private supervision adapts Gemma to **evidence retrieval from document chunks**.
+Several passages can be useful for one question, including partial evidence.
+
+| Training ingredient | Purpose |
+|---|---|
+| Mostly synthetic organizational documents, plus one person's largely AI-written project docs | Exercise project notes, procedures, decisions and technical material |
+| Frozen structural chunks with section context and table/code handling | Train on the passage units the retrieval workflow consumes |
+| Queries written by four models from three provider families, without a designated answer | Ask for evidence without reducing every question to one target passage |
+| Model-judged relevance grades | Distinguish decisive, partial, related and irrelevant evidence |
 
-Private training passages were prepared with frozen versions of Daecore's
-structural chunker: headings and section context, cuts near paragraph or sentence
-boundaries, and handling for tables and code blocks. The corpus includes retained
-chunks from earlier parser versions; it was not regenerated for this release.
-Gemma applies its own tokenizer and input limits after passage preparation.
-Changing boundaries can change whether evidence stays together and how much the
-model sees, so these results are tied to the evaluated passages.
+The [chunking guide](evaluation/chunking.md#retrieval-model-data) explains
+the source data and methods. The corpus retains earlier parser outputs;
+Gemma applies its own tokenizer and limits after chunking. These shared
+generation and judging processes limit generalization beyond the tested data.
 
 Public supervision uses existing evidence annotations from [HotpotQA](https://hotpotqa.github.io/), [MultiDoc2Dial](https://github.com/doc2dial/multidoc2dial) and [FinQA](https://github.com/czyssrs/FinQA): multi-document questions, document-grounded dialogue, and textual or table evidence for numerical questions. The model learns to retrieve that evidence, not to perform FinQA's calculations; FinQA is unrelated to the FiQA evaluation above.
 
@@ -229,7 +273,7 @@ Vulkan runs through the native ONNX Runtime WebGPU plugin, [`onnxruntime-ep-webg
 
 Checks used Windows x64 with an NVIDIA RTX 3060 Ti (8 GB); other GPUs and Linux are untested, even where Vulkan is available.
 
-The task-specific data is mostly generated and model-judged, with no human reference panel, so it describes this development distribution rather than other users' documents. The dense and full-pipeline panels use the same queries but different judgment pools; compare models within each table. Some questions omit the project or task context needed for an unambiguous judgment. The 970-query pipeline panel uses natural-language questions and reports no separate terse-query or paraphrase robustness scores. Passage relevance does not measure distinct-fact coverage or downstream agent success.
+The task-specific data is mostly generated and model-judged, with no human reference panel, so it describes this development distribution rather than other users' documents. The dense and full-pipeline panels share queries and a pooled relevance reference, but evaluate different retrieval paths. Some questions omit the project or task context needed for an unambiguous judgment. The 970-query pipeline panel uses natural-language questions and reports no separate terse-query or paraphrase robustness scores. Passage relevance does not measure distinct-fact coverage or downstream agent success.
 
 ## License
 

evaluation/README.md

@@ -2,10 +2,14 @@
 
 These records support the [model card's](../README.md) results: Gemma's dense retrieval on private and public data, its smaller embedding widths, and the full retrieval pipeline. They contain scores and anonymized relevance grades, without private queries or document text.
 
+The [chunking guide](chunking.md#retrieval-model-data) explains the synthetic
+source documents, passage preparation and relevance supervision behind the
+private evaluations.
+
 | File | Contents |
 |---|---|
-| `retriever.json` | Upstream and v2 reviewed top-20 dense rankings for all 970 queries over 82,719 passages |
-| `dimensions.json` | V2 per-query public nDCG@10 at 768, 512, 256 and 128 dimensions |
+| `retriever.json` | Upstream and v2 reviewed top-50 dense rankings for all 970 queries over 82,719 passages |
+| `dimensions.json` | V2 public nDCG@10 and private top-10 graded rankings at four embedding widths |
 | `promotion.json` | Ranked grades for the predecessor-to-v2 pipeline comparison and public nDCG@10 for upstream and v2 |
 | `serving.json` | The 970-query Gemma v2 + BM25 + Ettin replay used by both model cards, including CUDA/Vulkan results |
 | `*-summary.json` | Summaries that `metrics.py` reproduces |
@@ -32,59 +36,81 @@ prefixes and 128-query/1,024-passage token limits, and 768 dimensions. Exact
 cosine rankings take the top 50 for each frozen query form, then merge them
 with reciprocal-rank fusion (constant 60). Identical text is deduplicated;
 there is no parent-document filter, BM25 or reranker. The record includes
-model identities, source hashes, top-20 grades and reference grade counts.
+model identities, source hashes, top-50 grades and reference grade counts.
 This is a reused development panel, not a fresh holdout.
 
-Completing the comparison required 11,480 new pair decisions. Two independent
-GPT-6 Sol judges graded each pair, and a fresh blind judgment resolved 3,207
-ordinal disagreements. The primary assistant read 179 complete query–passage
-pairs: all 18 final quote flags, all 72 abstentions, and samples spanning every
-primary grade combination. The 18 quote repairs changed no grades. The result
-adds 11,408 grades to a shared reference containing 87,434 known pairs. No
-person reviewed these judgments, and the corpus is not exhaustively labeled.
-
-Every top-20 position is graded or explicitly excluded as ungradable; no
-missing-label bounds remain. Cutoffs 1, 3, 5, 10 and 20 retain all 970 queries.
-At rank 20, the upstream lists exclude 56 positions and v2 excludes 53.
-Cut first, drop abstentions second, and never backfill. A wholly abstained
-prefix would omit that query from both models at that cutoff; none occurs in
-this comparison. Precision pools retained positions. Recall averages the 911
-queries with known useful passages. nDCG uses gains 0, 1, 3 and 7 against the
-same reference for both models, reindexes retained positions, and averages
-all 970 queries; an ideal gain of zero contributes zero. The dense reference
-is larger than the pipeline reference below, so compare models within each
-table rather than treating their nDCG values as one common scale.
+The shared reference contains **127,665 graded query–passage pairs**. The top-50
+extension added 40,231 grades and 222 abstentions from 40,453 previously
+unjudged pairs, covering both dense models, the hybrid pool and smaller-width
+top-10 results. Two independent GPT-6.1 Sol judges graded each pair; a fresh
+blind adjudicator resolved 7,765 ordinal disagreements. A semantic abstention
+from either primary judge remained excluded. The primary assistant read all
+349 selected pairs: every abstention and final quotation flag, plus a
+stratified sample. Sixteen quotation repairs changed no grades. No person
+reviewed the labels.
+
+Every top-50 position is graded or explicitly excluded. All 970 queries remain
+eligible at 1, 3, 5, 10, 20 and 50. Upstream excludes 164 positions at rank 50;
+v2 excludes 167. Cut the original prefix first, drop abstentions second, and
+never backfill. A wholly abstained prefix would omit the query from both arms
+at that depth. Precision pools retained positions. Recall averages the 925
+queries with known useful passages; these are not exhaustive corpus labels.
+nDCG uses gains 0, 1, 3 and 7, reindexes retained positions and averages all
+970 queries; an ideal gain of zero contributes zero.
+
+**Reference expansion, not model change.** The existing top-20 grades and
+rankings are unchanged, so Hit and precision are unchanged. New relevant
+passages enlarge recall's denominator and can strengthen the ideal nDCG
+ranking. Dense nDCG@10 is now 0.4363 upstream and 0.5574 for v2, versus
+0.4506 and 0.5748 on the earlier 87,434-pair reference. Both models are recomputed together.
+Dense, smaller-width and current pipeline results now share this reference;
+the historical predecessor comparison below keeps its original reference.
 
 **Public data.**
 The model card compares v2 with upstream EmbeddingGemma on SciFact, FiQA, NFCorpus, SciDocs and ArguAna. `promotion.json` includes per-query nDCG@10 for all five datasets, using full corpora, matched preprocessing and official relevance judgments. V2 remains below upstream on every panel. The previous fine-tune was measured only on FiQA (0.4009) and SciFact (0.7679); its other three results are absent, not estimated.
 
 These datasets supplied no training examples, but informed development. They are not untouched tests.
 
-**Matryoshka widths.** `dimensions.json` contains per-query scores from new exact
-cosine rankings computed from the selected checkpoint's cached FP32 public vectors. For each
-smaller width, keep the first dimensions and L2-normalize query and passage
-vectors again. Self-ID matches are excluded; ties preserve corpus order.
-Official judgments use linear-gain nDCG, as in the original public scorer.
-All 1,406 ArguAna queries remain, including the five whose positives are absent
-from its corpus. The 768-wide per-query scores reproduce the original results
-exactly. Smaller widths were not evaluated on the private or full-pipeline
-panels and do not shorten the encoder's forward pass.
+**Matryoshka widths.** `dimensions.json` contains public per-query nDCG@10
+and a `private` record with graded top-10 rankings at 768, 512, 256 and 128
+dimensions. Vectors come from the selected checkpoint's cached FP32 outputs;
+smaller widths retain the first dimensions and are L2-normalized again.
+
+Public scoring uses exact cosine rankings, self-ID exclusion, stable corpus
+order for ties and official linear-gain nDCG. All 1,406 ArguAna queries remain,
+including five whose positives are absent from the corpus. These public
+scores are unchanged. Private scoring uses the same frozen query forms,
+dense-only fusion and expanded relevance reference as the 768-wide comparison.
+All 970 queries remain eligible at top 10; excluded positions are 22, 26, 16
+and 37 respectively. The 768-wide results reproduce both references exactly.
+Smaller widths have not been qualified through the full pipeline and do not
+shorten the encoder's forward pass.
 
 ## Effect in the retrieval pipeline
 
-The identical full-pipeline table on both model cards comes from `serving.json`:
-Gemma v2 + BM25 + fusion + unchanged Ettin on all 970 queries, using CUDA FP16
-for reranking. At depths 3–20, Hit and nDCG average all queries, and
-known-positive recall averages the 895 with judged useful evidence. Depth 1
-uses 965 shared judged queries, including 890 with known positives; five
-queries are omitted from both providers because a first result was ungradable.
-Precision pools retained positions. The serving reference adds 32 grades to the earlier promotion
-reference. Its shared judging-exclusion set removes 52 top-20 positions per
-provider without backfilling. Fixed cutoffs measure ranking independently of
-the selector's choice of a 3–20-item prefix. Vulkan matches CUDA at Hit@5,
-Hit@10 and Hit@20; Hit@3 differs by one query and nDCG@10 by 0.0004.
-
-The separate predecessor comparison below comes from `promotion.json`.
+The identical table on both cards comes from `serving.json`: Gemma v2 +
+BM25 + fusion + unchanged Ettin, on all 970 queries. CUDA FP16 and Vulkan FP32
+use the same 50 candidates. At 50, Hit is 93.09% and known-positive recall is
+45.98%; those candidate-coverage measures cannot change through reranking.
+Dense v2 reaches 94.43% and 57.92% on this panel. Fusion with a fixed 50-slot
+budget can displace dense candidates even while reranking improves early
+precision.
+
+At depths 3–50, Hit and nDCG average all 970 queries; recall averages the 925
+with known useful evidence. Depth 1 uses 965 shared judged queries, including
+920 with known positives; five are omitted because a first result was
+ungradable. Precision pools retained positions. Each provider excludes 52
+positions at rank 20 and 93 at rank 50 without backfill. Fixed cutoffs are
+separate from the selector's choice of a 3–20-item prefix. Vulkan matches
+CUDA at Hit@5, Hit@10, Hit@20 and Hit@50; Hit@3 differs by one query and
+nDCG@10 by 0.0003.
+
+### Historical predecessor comparison
+
+This table comes from `promotion.json`, with its **original 75,488-pair relevance
+reference**. Its nDCG is not directly comparable with the expanded-reference
+table on the main card.
+
 Only the embedding model changed in this comparison. BM25, reciprocal-rank fusion and the [Ettin reranker](https://huggingface.co/Daecore/ettin-150m-memory-reranker-ft-v1) stayed fixed. The same score mapping selected returned prefixes in both arms. These results measure the whole pipeline on 970 reused queries over 82,719 passage texts, rather than Gemma alone.
 
 | Metric | Previous Gemma | Gemma v2 |
@@ -117,4 +143,4 @@ retrieval method, not the query format.
 
 `serving-qualification.json` records this model's CPU, CUDA and Vulkan checks. The 74-vector panel covers token limits, mixed lengths and concurrent query and passage calls. It is a bounded serving check, separate from retrieval quality.
 
-The private documents and labels are mostly model-generated. Project-document sources come from one person's workspaces and are often AI-written. The panel was reused during development, training was run once, and there is no human reference panel. These results do not establish generalization across users, unique-fact coverage or downstream agent success.
+The private documents and labels are mostly model-generated. Project-document sources come from one person's workspaces and are often AI-written. Some queries omit the project or subject needed for a clear relevance judgment. The panel was reused during development, training was run once, and there is no human reference panel. These results do not establish generalization across users, unique-fact coverage or downstream agent success.

evaluation/chunking.md

@@ -0,0 +1,58 @@
+# Chunking and retrieval data
+
+## Retrieval-model data
+
+Daecore fine-tunes retrieval models on passages prepared for its document
+workflow. A passage is the exact chunk the model reads: a section, excerpt,
+table fragment or code block. It can supply useful partial evidence without
+answering the whole question.
+
+**Source documents → structural chunks → graded query–passage pairs → retrieval training**
+
+| Ingredient | What it contributes |
+|---|---|
+| Mostly synthetic workplace documentation | Notes, procedures, decisions and technical material across generated project scenarios |
+| One person's project documents, much of them AI-written | Additional examples, not a representative sample of other users' workspaces |
+| Frozen structural chunks | The retrieval units that Gemma embeds and Ettin ranks; some retain earlier parser outputs |
+| Model-written questions and graded evidence | Several useful passages per question, with decisive and partial evidence distinguished from merely related text |
+| Public evidence annotations used by Gemma v2 | HotpotQA paragraphs, MultiDoc2Dial grounding passages and FinQA text/table evidence |
+
+Public retrieval data also comes in passages; it is not uniformly made of full
+documents. The distinction is their source and preparation. Each model applies
+its tokenizer and input limits after chunking, so a chunk boundary and a model's
+truncation limit are separate constraints.
+
+This is deliberate specialization of already capable upstream models. The
+private comparisons measure gains on this workflow, while public benchmarks
+show the general-retrieval cost. Daecore accepted that tradeoff for its intended
+use; the cards report both sides. Generated questions and shared preparation
+and judging processes do not establish gains for every real user or agent.
+
+Training and evaluation use frozen passage snapshots. Current chunking fixes
+do not retroactively change those texts or transfer their relevance labels to
+different cuts.
+
+## Chunking methods
+
+- **Heading-based splitting:** use the document's sections; oversized sections
+  can split further at their subheadings. This follows the author's structure.
+- **Title and heading context:** add the document title and relevant heading
+  path to section chunks so their subject stays clear. Smaller searchable pieces
+  retain that metadata but do not automatically repeat it inside every piece.
+- **Boundary-aware cuts and overlap:** prefer a nearby sentence end, paragraph
+  break or line break. Ordinary prose windows repeat up to 200 characters across
+  cuts to carry nearby context; oversized structural continuations have their
+  own framing rules.
+- **Code-block and table handling:** keep blocks together when they fit. Split
+  oversized tables with repeated column headers and oversized code blocks with
+  reopened and closed code fences, so continuations remain readable.
+- **Record-aware splitting:** preserve row, field or entry context for CSV,
+  JSON, JSONL and dated entries. Oversized records are divided with identifying
+  labels rather than losing their remaining content.
+- **Paragraph packing for imported memory files:** group whole paragraphs into
+  bounded pieces so excerpts usually begin at a paragraph. A single oversized
+  paragraph falls back to structural splitting.
+
+These methods work together: make supported text and record chunks fit first,
+then create bounded searchable pieces for sections that are still too large.
+Both operations use the same structural cutter.

evaluation/dimensions-summary.json

@@ -1,56 +1,310 @@
-{
-  "queries": 3677,
-  "vector_bytes_fp32": {
-    "768": 3072,
-    "512": 2048,
-    "256": 1024,
-    "128": 512
-  },
-  "public": {
-    "arguana": {
-      "queries": 1406,
-      "ndcg@10": {
-        "768": 0.6258634411077135,
-        "512": 0.613697054466976,
-        "256": 0.5986725670753965,
-        "128": 0.5533073210679951
-      }
-    },
-    "fiqa": {
-      "queries": 648,
-      "ndcg@10": {
-        "768": 0.446838124778398,
-        "512": 0.43740349056283273,
-        "256": 0.40912280098290843,
-        "128": 0.3668286877036632
-      }
-    },
-    "nfcorpus": {
-      "queries": 323,
-      "ndcg@10": {
-        "768": 0.3889646280478944,
-        "512": 0.3875786151617704,
-        "256": 0.3660320811396263,
-        "128": 0.3310697612326329
-      }
-    },
-    "scidocs": {
-      "queries": 1000,
-      "ndcg@10": {
-        "768": 0.18521860884727676,
-        "512": 0.179327587641947,
-        "256": 0.17081239437849563,
-        "128": 0.1484461830477022
-      }
-    },
-    "scifact": {
-      "queries": 300,
-      "ndcg@10": {
-        "768": 0.7782845713792622,
-        "512": 0.7840710444235555,
-        "256": 0.7756812272647337,
-        "128": 0.7488095898261483
-      }
-    }
-  }
-}
+{
+  "queries": 3677,
+  "vector_bytes_fp32": {
+    "768": 3072,
+    "512": 2048,
+    "256": 1024,
+    "128": 512
+  },
+  "public": {
+    "arguana": {
+      "queries": 1406,
+      "ndcg@10": {
+        "768": 0.6258634411077135,
+        "512": 0.613697054466976,
+        "256": 0.5986725670753965,
+        "128": 0.5533073210679951
+      }
+    },
+    "fiqa": {
+      "queries": 648,
+      "ndcg@10": {
+        "768": 0.446838124778398,
+        "512": 0.43740349056283273,
+        "256": 0.40912280098290843,
+        "128": 0.3668286877036632
+      }
+    },
+    "nfcorpus": {
+      "queries": 323,
+      "ndcg@10": {
+        "768": 0.3889646280478944,
+        "512": 0.3875786151617704,
+        "256": 0.3660320811396263,
+        "128": 0.3310697612326329
+      }
+    },
+    "scidocs": {
+      "queries": 1000,
+      "ndcg@10": {
+        "768": 0.18521860884727676,
+        "512": 0.179327587641947,
+        "256": 0.17081239437849563,
+        "128": 0.1484461830477022
+      }
+    },
+    "scifact": {
+      "queries": 300,
+      "ndcg@10": {
+        "768": 0.7782845713792622,
+        "512": 0.7840710444235555,
+        "256": 0.7756812272647337,
+        "128": 0.7488095898261483
+      }
+    }
+  },
+  "private": {
+    "queries": 970,
+    "models": {
+      "768": {
+        "1": {
+          "scored_queries": 964,
+          "excluded_queries": 6,
+          "hit": 0.6452282157676349,
+          "precision": 0.6452282157676349,
+          "macro_precision": 0.6452282157676349,
+          "ndcg": 0.5301323849041691,
+          "known_positive_recall": 0.01788016629109334,
+          "recall_queries": 919,
+          "useful": 622,
+          "retained": 964,
+          "excluded_positions": 0,
+          "mean_useful": 0.6452282157676349,
+          "mean_retained": 1.0
+        },
+        "3": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.7938144329896907,
+          "precision": 0.6308009625300791,
+          "macro_precision": 0.6309278350515464,
+          "ndcg": 0.5406982958852574,
+          "known_positive_recall": 0.05034823759106709,
+          "recall_queries": 925,
+          "useful": 1835,
+          "retained": 2909,
+          "excluded_positions": 1,
+          "mean_useful": 1.8917525773195876,
+          "mean_retained": 2.9989690721649485
+        },
+        "5": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8474226804123711,
+          "precision": 0.6266005782734407,
+          "macro_precision": 0.6269759450171821,
+          "ndcg": 0.5453498298950709,
+          "known_positive_recall": 0.08055790386567772,
+          "recall_queries": 925,
+          "useful": 3034,
+          "retained": 4842,
+          "excluded_positions": 8,
+          "mean_useful": 3.1278350515463917,
+          "mean_retained": 4.991752577319588
+        },
+        "10": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8814432989690721,
+          "precision": 0.6117999586691465,
+          "macro_precision": 0.6123367697594502,
+          "ndcg": 0.5574064002423099,
+          "known_positive_recall": 0.15074270958466607,
+          "recall_queries": 925,
+          "useful": 5921,
+          "retained": 9678,
+          "excluded_positions": 22,
+          "mean_useful": 6.104123711340206,
+          "mean_retained": 9.977319587628866
+        }
+      },
+      "512": {
+        "1": {
+          "scored_queries": 964,
+          "excluded_queries": 6,
+          "hit": 0.6462655601659751,
+          "precision": 0.6462655601659751,
+          "macro_precision": 0.6462655601659751,
+          "ndcg": 0.5478166370282552,
+          "known_positive_recall": 0.01775688496570127,
+          "recall_queries": 919,
+          "useful": 623,
+          "retained": 964,
+          "excluded_positions": 0,
+          "mean_useful": 0.6462655601659751,
+          "mean_retained": 1.0
+        },
+        "3": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.7969072164948454,
+          "precision": 0.635237439779766,
+          "macro_precision": 0.6355670103092783,
+          "ndcg": 0.5429708859076539,
+          "known_positive_recall": 0.05057307077004017,
+          "recall_queries": 925,
+          "useful": 1846,
+          "retained": 2906,
+          "excluded_positions": 4,
+          "mean_useful": 1.9030927835051545,
+          "mean_retained": 2.995876288659794
+        },
+        "5": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8422680412371134,
+          "precision": 0.6308613922743235,
+          "macro_precision": 0.63106529209622,
+          "ndcg": 0.5504066702414289,
+          "known_positive_recall": 0.08121602068063129,
+          "recall_queries": 925,
+          "useful": 3054,
+          "retained": 4841,
+          "excluded_positions": 9,
+          "mean_useful": 3.1484536082474226,
+          "mean_retained": 4.990721649484536
+        },
+        "10": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8824742268041237,
+          "precision": 0.6132933636551582,
+          "macro_precision": 0.613993618065783,
+          "ndcg": 0.5604826042172213,
+          "known_positive_recall": 0.15004173661513098,
+          "recall_queries": 925,
+          "useful": 5933,
+          "retained": 9674,
+          "excluded_positions": 26,
+          "mean_useful": 6.116494845360824,
+          "mean_retained": 9.97319587628866
+        }
+      },
+      "256": {
+        "1": {
+          "scored_queries": 964,
+          "excluded_queries": 6,
+          "hit": 0.6037344398340249,
+          "precision": 0.6037344398340249,
+          "macro_precision": 0.6037344398340249,
+          "ndcg": 0.5127445168938944,
+          "known_positive_recall": 0.015976956236435802,
+          "recall_queries": 919,
+          "useful": 582,
+          "retained": 964,
+          "excluded_positions": 0,
+          "mean_useful": 0.6037344398340249,
+          "mean_retained": 1.0
+        },
+        "3": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.7608247422680412,
+          "precision": 0.5955249569707401,
+          "macro_precision": 0.5962199312714777,
+          "ndcg": 0.5111829833045054,
+          "known_positive_recall": 0.046233615834183193,
+          "recall_queries": 925,
+          "useful": 1730,
+          "retained": 2905,
+          "excluded_positions": 5,
+          "mean_useful": 1.7835051546391754,
+          "mean_retained": 2.9948453608247423
+        },
+        "5": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8216494845360824,
+          "precision": 0.5925696594427244,
+          "macro_precision": 0.5929381443298969,
+          "ndcg": 0.5174515106874455,
+          "known_positive_recall": 0.0744111972114428,
+          "recall_queries": 925,
+          "useful": 2871,
+          "retained": 4845,
+          "excluded_positions": 5,
+          "mean_useful": 2.95979381443299,
+          "mean_retained": 4.994845360824742
+        },
+        "10": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8731958762886598,
+          "precision": 0.5818876497315159,
+          "macro_precision": 0.5823911798396334,
+          "ndcg": 0.5291576970298347,
+          "known_positive_recall": 0.1419199356602385,
+          "recall_queries": 925,
+          "useful": 5635,
+          "retained": 9684,
+          "excluded_positions": 16,
+          "mean_useful": 5.809278350515464,
+          "mean_retained": 9.983505154639175
+        }
+      },
+      "128": {
+        "1": {
+          "scored_queries": 964,
+          "excluded_queries": 6,
+          "hit": 0.5726141078838174,
+          "precision": 0.5726141078838174,
+          "macro_precision": 0.5726141078838174,
+          "ndcg": 0.47288085358624776,
+          "known_positive_recall": 0.015019269226427342,
+          "recall_queries": 919,
+          "useful": 552,
+          "retained": 964,
+          "excluded_positions": 0,
+          "mean_useful": 0.5726141078838174,
+          "mean_retained": 1.0
+        },
+        "3": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.7402061855670103,
+          "precision": 0.5779310344827586,
+          "macro_precision": 0.5780068728522336,
+          "ndcg": 0.4818931326910878,
+          "known_positive_recall": 0.044531751738185646,
+          "recall_queries": 925,
+          "useful": 1676,
+          "retained": 2900,
+          "excluded_positions": 10,
+          "mean_useful": 1.7278350515463918,
+          "mean_retained": 2.9896907216494846
+        },
+        "5": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8041237113402062,
+          "precision": 0.574591351127664,
+          "macro_precision": 0.5746048109965636,
+          "ndcg": 0.48751134015621145,
+          "known_positive_recall": 0.07159737689223461,
+          "recall_queries": 925,
+          "useful": 2777,
+          "retained": 4833,
+          "excluded_positions": 17,
+          "mean_useful": 2.8628865979381444,
+          "mean_retained": 4.982474226804124
+        },
+        "10": {
+          "scored_queries": 970,
+          "excluded_queries": 0,
+          "hit": 0.8649484536082475,
+          "precision": 0.5659733002173238,
+          "macro_precision": 0.5665320733104239,
+          "ndcg": 0.5035498696905354,
+          "known_positive_recall": 0.13623796718702827,
+          "recall_queries": 925,
+          "useful": 5469,
+          "retained": 9663,
+          "excluded_positions": 37,
+          "mean_useful": 5.638144329896908,
+          "mean_retained": 9.961855670103093
+        }
+      }
+    },
+    "corpus_passages": 82719
+  }
+}

evaluation/figures.py

@@ -63,7 +63,7 @@ def reranker(summary: dict) -> str:
 
 
 def retriever(summary: dict) -> str:
-    cutoffs = [1, 3, 5, 10, 20]
+    cutoffs = sorted(map(int, summary['models']['upstream']))
     x = list(range(len(cutoffs)))
     panels = []
     legend = [('Upstream Gemma', UPSTREAM, None), ('Daecore v2', FIT, None)]

evaluation/metrics.py

@@ -143,7 +143,13 @@ def summarize_dimensions(data: dict) -> dict:
         panels[name] = {'queries': len(panel), 'ndcg@10': {
             str(w): mean(row['ndcg@10'][str(w)] for row in panel) for w in widths
         }}
-    return {'queries': len(rows), 'vector_bytes_fp32': {str(w): w * 4 for w in widths}, 'public': panels}
+    result = {'queries': len(rows), 'vector_bytes_fp32': {str(w): w * 4 for w in widths}, 'public': panels}
+    if 'private' in data:
+        private = data['private']
+        if private['model_order'] != [str(w) for w in widths]:
+            raise ValueError('Private width rankings must cover the same ordered widths')
+        result['private'] = summarize_record('retriever', private)
+    return result
 
 
 def _summarize_ranked_grades(data: dict, *, include_selected: bool) -> dict:
@@ -155,6 +161,12 @@ def _summarize_ranked_grades(data: dict, *, include_selected: bool) -> dict:
     rows = data['rows']
     if not rows or len({row['id'] for row in rows}) != len(rows):
         raise ValueError('Ranked rows require unique nonempty query identities')
+    # Published promotion and serving v1 records contain twenty positions.
+    # Extended dense/serving records declare their measured depth explicitly.
+    depth_limit = data.get('ranking_depth', 20)
+    allowed_depths = (20, 50) if include_selected else (10, 20, 50)
+    if type(depth_limit) is not int or depth_limit not in allowed_depths:
+        raise ValueError('Unsupported measured ranking depth')
     # Validate before excluding fully abstained prefixes. An omitted query
     # must never conceal a malformed grade, model map or selected depth.
     for row in rows:
@@ -163,9 +175,9 @@ def _summarize_ranked_grades(data: dict, *, include_selected: bool) -> dict:
             raise ValueError('Reference grade counts must cover grades zero through three')
         for model in data['model_order']:
             ranked, excluded = row['ranked_grades'][model], row['excluded'][model]
-            if (len(ranked) != 20 or len(excluded) != 20
+            if (len(ranked) != depth_limit or len(excluded) != depth_limit
                     or any(type(x) is not bool for x in excluded)):
-                raise ValueError('Ranked rows require a bounded original top twenty')
+                raise ValueError('Ranked rows must cover exactly the declared ranking depth')
             if include_selected:
                 depth = row['selected_depth'][model]
                 if type(depth) is not int or not 3 <= depth <= 20:
@@ -174,7 +186,9 @@ def _summarize_ranked_grades(data: dict, *, include_selected: bool) -> dict:
                    for grade, drop in zip(ranked, excluded, strict=True)):
                 raise ValueError('Only declared abstentions may lack grades')
     output = {'queries': len(rows), 'models': {}}
-    requested_cutoffs = (1, 3, 5, 10, 20, 'selected') if include_selected else (1, 3, 5, 10, 20)
+    requested_cutoffs = tuple(k for k in CUTOFFS if k <= depth_limit)
+    if include_selected:
+        requested_cutoffs += ('selected',)
     for model in data['model_order']:
         cutoffs = {}
         for cutoff in requested_cutoffs:

evaluation/retriever-summary.json

@@ -8,9 +8,9 @@
         "hit": 0.534020618556701,
         "precision": 0.534020618556701,
         "macro_precision": 0.534020618556701,
-        "ndcg": 0.4695139911634757,
-        "known_positive_recall": 0.019407636059998304,
-        "recall_queries": 911,
+        "ndcg": 0.4592047128129602,
+        "known_positive_recall": 0.014444554022554491,
+        "recall_queries": 925,
         "useful": 518,
         "retained": 970,
         "excluded_positions": 0,
@@ -23,9 +23,9 @@
         "hit": 0.7298969072164948,
         "precision": 0.5082530949105915,
         "macro_precision": 0.5082474226804123,
-        "ndcg": 0.4515657513806275,
-        "known_positive_recall": 0.05434321461039996,
-        "recall_queries": 911,
+        "ndcg": 0.4413745306026501,
+        "known_positive_recall": 0.03938697535513605,
+        "recall_queries": 925,
         "useful": 1478,
         "retained": 2908,
         "excluded_positions": 2,
@@ -38,9 +38,9 @@
         "hit": 0.7948453608247422,
         "precision": 0.49948379103861246,
         "macro_precision": 0.4996219931271478,
-        "ndcg": 0.4457312507652407,
-        "known_positive_recall": 0.08530117331466223,
-        "recall_queries": 911,
+        "ndcg": 0.43462885647387695,
+        "known_positive_recall": 0.06172226483192968,
+        "recall_queries": 925,
         "useful": 2419,
         "retained": 4843,
         "excluded_positions": 7,
@@ -53,9 +53,9 @@
         "hit": 0.8484536082474227,
         "precision": 0.4739561802397685,
         "macro_precision": 0.47407543773523153,
-        "ndcg": 0.45062269194348514,
-        "known_positive_recall": 0.15770613185252844,
-        "recall_queries": 911,
+        "ndcg": 0.43629894440575245,
+        "known_positive_recall": 0.11180689389376004,
+        "recall_queries": 925,
         "useful": 4586,
         "retained": 9676,
         "excluded_positions": 24,
@@ -68,14 +68,29 @@
         "hit": 0.8876288659793814,
         "precision": 0.4393610421836228,
         "macro_precision": 0.44005992535723476,
-        "ndcg": 0.4736202687789316,
-        "known_positive_recall": 0.2807541608701895,
-        "recall_queries": 911,
+        "ndcg": 0.4525661889832219,
+        "known_positive_recall": 0.19611550928151497,
+        "recall_queries": 925,
         "useful": 8499,
         "retained": 19344,
         "excluded_positions": 56,
         "mean_useful": 8.761855670103094,
         "mean_retained": 19.942268041237114
+      },
+      "50": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.931958762886598,
+        "precision": 0.38598560079443894,
+        "macro_precision": 0.3867178488702632,
+        "ndcg": 0.5107049932805754,
+        "known_positive_recall": 0.41118693689165037,
+        "recall_queries": 925,
+        "useful": 18657,
+        "retained": 48336,
+        "excluded_positions": 164,
+        "mean_useful": 19.234020618556702,
+        "mean_retained": 49.83092783505155
       }
     },
     "finetuned": {
@@ -85,9 +100,9 @@
         "hit": 0.6474226804123712,
         "precision": 0.6474226804123712,
         "macro_precision": 0.6474226804123712,
-        "ndcg": 0.5420716740304369,
-        "known_positive_recall": 0.02462514980632024,
-        "recall_queries": 911,
+        "ndcg": 0.5312714776632302,
+        "known_positive_recall": 0.017869431761799028,
+        "recall_queries": 925,
         "useful": 628,
         "retained": 970,
         "excluded_positions": 0,
@@ -100,9 +115,9 @@
         "hit": 0.7938144329896907,
         "precision": 0.6308009625300791,
         "macro_precision": 0.6309278350515464,
-        "ndcg": 0.5523629493718011,
-        "known_positive_recall": 0.06947139492907671,
-        "recall_queries": 911,
+        "ndcg": 0.5406982958852574,
+        "known_positive_recall": 0.05034823759106709,
+        "recall_queries": 925,
         "useful": 1835,
         "retained": 2909,
         "excluded_positions": 1,
@@ -115,9 +130,9 @@
         "hit": 0.8474226804123711,
         "precision": 0.6266005782734407,
         "macro_precision": 0.6269759450171821,
-        "ndcg": 0.558603868737523,
-        "known_positive_recall": 0.11088922136775149,
-        "recall_queries": 911,
+        "ndcg": 0.5453498298950709,
+        "known_positive_recall": 0.08055790386567772,
+        "recall_queries": 925,
         "useful": 3034,
         "retained": 4842,
         "excluded_positions": 8,
@@ -130,9 +145,9 @@
         "hit": 0.8814432989690721,
         "precision": 0.6117999586691465,
         "macro_precision": 0.6123367697594502,
-        "ndcg": 0.5748078775849167,
-        "known_positive_recall": 0.2082023156474653,
-        "recall_queries": 911,
+        "ndcg": 0.5574064002423099,
+        "known_positive_recall": 0.15074270958466607,
+        "recall_queries": 925,
         "useful": 5921,
         "retained": 9678,
         "excluded_positions": 22,
@@ -145,14 +160,29 @@
         "hit": 0.9051546391752577,
         "precision": 0.5810720008270016,
         "macro_precision": 0.5815832221977094,
-        "ndcg": 0.6064961581206146,
-        "known_positive_recall": 0.38649540366398033,
-        "recall_queries": 911,
+        "ndcg": 0.5818739813983459,
+        "known_positive_recall": 0.27556154002694083,
+        "recall_queries": 925,
         "useful": 11242,
         "retained": 19347,
         "excluded_positions": 53,
         "mean_useful": 11.589690721649484,
         "mean_retained": 19.945360824742266
+      },
+      "50": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9443298969072165,
+        "precision": 0.5045207208325575,
+        "macro_precision": 0.5051647365119746,
+        "ndcg": 0.6449716439245425,
+        "known_positive_recall": 0.5791516627126307,
+        "recall_queries": 925,
+        "useful": 24385,
+        "retained": 48333,
+        "excluded_positions": 167,
+        "mean_useful": 25.13917525773196,
+        "mean_retained": 49.827835051546394
       }
     }
   },

evaluation/serving-summary.json

@@ -8,9 +8,9 @@
         "hit": 0.7233160621761658,
         "precision": 0.7233160621761658,
         "macro_precision": 0.7233160621761658,
-        "ndcg": 0.664594127806563,
-        "known_positive_recall": 0.0346020393358114,
-        "recall_queries": 890,
+        "ndcg": 0.6458425857389588,
+        "known_positive_recall": 0.021209009540392988,
+        "recall_queries": 920,
         "useful": 698,
         "retained": 965,
         "excluded_positions": 0,
@@ -23,9 +23,9 @@
         "hit": 0.8536082474226804,
         "precision": 0.7157640565712314,
         "macro_precision": 0.7166666666666667,
-        "ndcg": 0.6576832474683237,
-        "known_positive_recall": 0.09521315062968948,
-        "recall_queries": 895,
+        "ndcg": 0.6374467700901159,
+        "known_positive_recall": 0.0589083117861917,
+        "recall_queries": 925,
         "useful": 2075,
         "retained": 2899,
         "excluded_positions": 11,
@@ -38,9 +38,9 @@
         "hit": 0.8845360824742268,
         "precision": 0.7028311634635255,
         "macro_precision": 0.7031786941580757,
-        "ndcg": 0.6554718901879515,
-        "known_positive_recall": 0.15383335671735354,
-        "recall_queries": 895,
+        "ndcg": 0.632593817782834,
+        "known_positive_recall": 0.09351524011264825,
+        "recall_queries": 925,
         "useful": 3401,
         "retained": 4839,
         "excluded_positions": 11,
@@ -53,9 +53,9 @@
         "hit": 0.8989690721649485,
         "precision": 0.67180070291503,
         "macro_precision": 0.6722631320569465,
-        "ndcg": 0.6610872123112499,
-        "known_positive_recall": 0.2802986435290336,
-        "recall_queries": 895,
+        "ndcg": 0.6315462046841925,
+        "known_positive_recall": 0.17039667688880922,
+        "recall_queries": 925,
         "useful": 6499,
         "retained": 9674,
         "excluded_positions": 26,
@@ -68,24 +68,39 @@
         "hit": 0.9103092783505154,
         "precision": 0.61794500723589,
         "macro_precision": 0.6184956360634603,
-        "ndcg": 0.6840908385828889,
-        "known_positive_recall": 0.4884616340614924,
-        "recall_queries": 895,
+        "ndcg": 0.6416731640651591,
+        "known_positive_recall": 0.29717756138114526,
+        "recall_queries": 925,
         "useful": 11956,
         "retained": 19348,
         "excluded_positions": 52,
         "mean_useful": 12.32577319587629,
         "mean_retained": 19.94639175257732
       },
+      "50": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9309278350515464,
+        "precision": 0.4043216890119198,
+        "macro_precision": 0.4044939351610387,
+        "ndcg": 0.6051209510262134,
+        "known_positive_recall": 0.4598077844640693,
+        "recall_queries": 925,
+        "useful": 19572,
+        "retained": 48407,
+        "excluded_positions": 93,
+        "mean_useful": 20.177319587628865,
+        "mean_retained": 49.904123711340205
+      },
       "selected": {
         "scored_queries": 970,
         "excluded_queries": 0,
         "hit": 0.8618556701030928,
         "precision": 0.8033770583310076,
         "macro_precision": 0.7040325511660611,
-        "ndcg": 0.6598955189009045,
-        "known_positive_recall": 0.2079138769097556,
-        "recall_queries": 895,
+        "ndcg": 0.6359990281117442,
+        "known_positive_recall": 0.128026594631235,
+        "recall_queries": 925,
         "useful": 5757,
         "retained": 7166,
         "excluded_positions": 17,
@@ -100,9 +115,9 @@
         "hit": 0.7243523316062176,
         "precision": 0.7243523316062176,
         "macro_precision": 0.7243523316062176,
-        "ndcg": 0.6648902047865778,
-        "known_positive_recall": 0.03463608768446649,
-        "recall_queries": 890,
+        "ndcg": 0.6461386627189736,
+        "known_positive_recall": 0.021228078953055077,
+        "recall_queries": 920,
         "useful": 699,
         "retained": 965,
         "excluded_positions": 0,
@@ -115,9 +130,9 @@
         "hit": 0.8525773195876288,
         "precision": 0.7157640565712314,
         "macro_precision": 0.7166666666666667,
-        "ndcg": 0.6577263968717306,
-        "known_positive_recall": 0.09519545071125639,
-        "recall_queries": 895,
+        "ndcg": 0.6373247400469291,
+        "known_positive_recall": 0.05893952330659241,
+        "recall_queries": 925,
         "useful": 2075,
         "retained": 2899,
         "excluded_positions": 11,
@@ -130,9 +145,9 @@
         "hit": 0.8845360824742268,
         "precision": 0.7023563455973543,
         "macro_precision": 0.702766323024055,
-        "ndcg": 0.6550340124265787,
-        "known_positive_recall": 0.15374094654114448,
-        "recall_queries": 895,
+        "ndcg": 0.6321994825191731,
+        "known_positive_recall": 0.09344587015679974,
+        "recall_queries": 925,
         "useful": 3398,
         "retained": 4838,
         "excluded_positions": 12,
@@ -145,9 +160,9 @@
         "hit": 0.8989690721649485,
         "precision": 0.6723514211886304,
         "macro_precision": 0.6727589592538046,
-        "ndcg": 0.6615310405020679,
-        "known_positive_recall": 0.2805978824757964,
-        "recall_queries": 895,
+        "ndcg": 0.631891212140629,
+        "known_positive_recall": 0.17053838380216288,
+        "recall_queries": 925,
         "useful": 6505,
         "retained": 9675,
         "excluded_positions": 25,
@@ -160,24 +175,39 @@
         "hit": 0.9103092783505154,
         "precision": 0.6178416373785404,
         "macro_precision": 0.6183925432799551,
-        "ndcg": 0.6840389893720067,
-        "known_positive_recall": 0.4883516349357555,
-        "recall_queries": 895,
+        "ndcg": 0.6416203428445276,
+        "known_positive_recall": 0.2971249330393808,
+        "recall_queries": 925,
         "useful": 11954,
         "retained": 19348,
         "excluded_positions": 52,
         "mean_useful": 12.323711340206186,
         "mean_retained": 19.94639175257732
       },
+      "50": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9309278350515464,
+        "precision": 0.4043216890119198,
+        "macro_precision": 0.4044939351610387,
+        "ndcg": 0.6050905860403181,
+        "known_positive_recall": 0.4598077844640693,
+        "recall_queries": 925,
+        "useful": 19572,
+        "retained": 48407,
+        "excluded_positions": 93,
+        "mean_useful": 20.177319587628865,
+        "mean_retained": 49.904123711340205
+      },
       "selected": {
         "scored_queries": 970,
         "excluded_queries": 0,
         "hit": 0.8608247422680413,
         "precision": 0.8027855153203343,
         "macro_precision": 0.7038533836490732,
-        "ndcg": 0.6599601279630594,
-        "known_positive_recall": 0.2082001919384574,
-        "recall_queries": 895,
+        "ndcg": 0.6358997417721292,
+        "known_positive_recall": 0.12825382711378852,
+        "recall_queries": 925,
         "useful": 5764,
         "retained": 7180,
         "excluded_positions": 17,

publication-manifest.json

@@ -1,6 +1,6 @@
 {
   "schema": "daecore.retrieval-publication-manifest",
-  "tool_sha256": "3ea615c62866883f911899f1e131d73ca8cc9701eea3540a3a016345ac5650f8",
+  "tool_sha256": "46dcb4b17750d5a4ba3d44b1236f33e9e5657673b648ae6cad48fe4ac4f147f8",
   "model_id": "embeddinggemma-300m-memory-ft-v2",
   "public_repo": "Daecore/embeddinggemma-300m-memory-ft-v2",
   "role": "dense-retriever",
@@ -12,7 +12,7 @@
   "export_receipt_sha256": "e24e1692f7e03c9af8040bb3f6e310a061d305f485861007f1f37d36808b20d8",
   "serving_sha256": "8c088b1635875d5406e148782b303cfc5a961138600783fb4842b413b98a98c3",
   "source_model_tree_sha256": "30a7b9f1c42b1fcee29a6729742685d5b744e8fa87d13cefab2fd2d49ea9150e",
-  "staged_at": "2026-09-29T16:47:59+00:00",
+  "staged_at": "2026-09-30T01:53:48+00:00",
   "files": {
     "added_tokens.json": {
       "sha256": "50b2f405ba56a26d4913fd772089992252d7f942123cc0a034d96424221ba946",
@@ -80,8 +80,8 @@
       "binding": "packaging record"
     },
     "evaluation/README.md": {
-      "sha256": "f7059b76c4e9956e899b5f8fb855a58262db42ffef301404c874c8ee9b1c19dc",
-      "size": 8475,
+      "sha256": "390b5e5d525ae6c0b2c3370050af1cc14b4574127bb258db911fb0afed670494",
+      "size": 9703,
       "binding": "packaging record"
     },
     "evaluation/serving-qualification.json": {
@@ -90,18 +90,23 @@
       "binding": "packaging record"
     },
     "evaluation/metrics.py": {
-      "sha256": "72e7452766b491c247514df8ad1c8f52c8023e1e305d779655a9e1f3159ce52d",
-      "size": 17556,
+      "sha256": "bbccf29f34db24aefc5630adcaf9630e5f214010902dbe1df0032618aa329a45",
+      "size": 18329,
+      "binding": "packaging record"
+    },
+    "evaluation/chunking.md": {
+      "sha256": "cda82fca3a467f02e06625a607fc8a5bfc9b16d48f4fe225d7f2282330d4955e",
+      "size": 3509,
       "binding": "packaging record"
     },
     "evaluation/retriever-summary.json": {
-      "sha256": "6d87cfa07056cb0316e21ed70a940994d1ad91e9a297161878482814811f736b",
-      "size": 5063,
+      "sha256": "b320430acd9938d08e31eb00693dd9f775e9890b17427f86577c9bbd91dd1bba",
+      "size": 6071,
       "binding": "packaging record"
     },
     "evaluation/dimensions-summary.json": {
-      "sha256": "d42adc93268a44675bcda3321a431f8cd92e6c9178efdb44e0314372468a7c6e",
-      "size": 1253,
+      "sha256": "3b160deb874b98e835b92b15070afe36e548b1ce692a7ef817af77026e5064f8",
+      "size": 9753,
       "binding": "packaging record"
     },
     "evaluation/promotion-summary.json": {
@@ -110,18 +115,18 @@
       "binding": "packaging record"
     },
     "evaluation/serving-summary.json": {
-      "sha256": "fc60b4e94fe01e0470dcced382046c4c30c329a87b5caa2b631b369015548198",
-      "size": 6029,
+      "sha256": "cd87757b53482e2ec9450a3c64f31a32afad2710900fbcbd7885828929aa9535",
+      "size": 7035,
       "binding": "packaging record"
     },
     "evaluation/retriever.json": {
-      "sha256": "fcfbe8155cd30319ee38745f92376a45635ecaed08b86962ee8604680ce910b7",
-      "size": 573086,
+      "sha256": "c209c96fd17144a6f7d2c7174bc27db4b346288910d085e2b614286b80ae4559",
+      "size": 1156315,
       "binding": "packaging record"
     },
     "evaluation/dimensions.json": {
-      "sha256": "a6f999513a4b1e044901af4cb4a9797cb6f3d0ae6d1124e68bf406983a02f3ac",
-      "size": 762884,
+      "sha256": "3d9b49d6bda19fe9f9b7f8859ce550defa230757fef85ee2bc1ff175f6f4fe4b",
+      "size": 1063026,
       "binding": "packaging record"
     },
     "evaluation/promotion.json": {
@@ -130,13 +135,13 @@
       "binding": "packaging record"
     },
     "evaluation/serving.json": {
-      "sha256": "0c017dbd0e66734c465d617eef24b2affb2670a96f7fd5b2c001514c3cc1381a",
-      "size": 500118,
+      "sha256": "f4870a5118400152d4a78c7fd7c96df4df73901bfe0c590fdcd00c2af477a4be",
+      "size": 1184937,
       "binding": "packaging record"
     },
     "evaluation/figures.py": {
-      "sha256": "53819eb947c3c5c25f6331b0e384e6e27f7b5da750c84f8bdf6fd6e7790c9d94",
-      "size": 5378,
+      "sha256": "6bc89020363df7a0cfcdebaa4e8be5b37385ab97b2b7bbf3f60cb9b46b2ac63e",
+      "size": 5408,
       "binding": "packaging record"
     },
     "evaluation/svg_figures.py": {
@@ -145,15 +150,15 @@
       "binding": "packaging record"
     },
     "figures/gemma-comparison.svg": {
-      "sha256": "f4bf74ea9e2e044e124893d5ca8bd028c6f2f79ef53babfb7ee718f8b6786be2",
-      "size": 13377,
+      "sha256": "4393db8afc6de32877a66b863708e396a1503d753932680dc7e4e590c4ccd513",
+      "size": 14507,
       "binding": "packaging record"
     },
     "README.md": {
-      "sha256": "e7d88fad6c2c12613c8e8a8108a38dc7fbb181c005ffadac172f1120f84c95f0",
-      "size": 14331,
+      "sha256": "b5fe1ab8b6d3821cf5ee42e9ffbb2d88d8509233d27834382ce63a385f5d60a4",
+      "size": 16388,
       "binding": "model card with upload-relative links",
-      "source_sha256": "5ea17ec12b2673539eba89fc6e3f85c9ed8ba7d879d9ce94dbeebcb1392a8afa"
+      "source_sha256": "58c1418eb860b7e8d2db5b84085780bb665ffbd92d06224a59cff700cebe0f02"
     }
   }
 }