Update embeddinggemma-300m-memory-ft-v2 documentation

README.md

@@ -22,7 +22,7 @@ tags:
 
 An English embedding model for finding the passages that answer a question in notes, procedures, decision records and technical documentation. It is a fine-tune of [Google's EmbeddingGemma-300M](https://huggingface.co/google/embeddinggemma-300m), packaged as one ONNX graph with pooling and normalization built in, so it runs locally without PyTorch.
 
-On Daecore's dense-retrieval panel, v2 finds a useful passage in the first ten results for **88.30%** of queries, compared with **74.20–76.60%** for upstream Gemma. The comparison uses 376 known-answerable queries over 66,240 passages; the upstream range preserves missing labels. V2 also recovers more than half of the first Daecore fine-tune's measured loss on FiQA and SciFact, while retaining its task-specific retrieval gains.
+On Daecore's **970-query dense-retrieval panel**, v2 raises nDCG@10 from **0.4506 to 0.5748** and precision@10 from **47.40% to 61.18%** compared with upstream Gemma. Both models search the same 82,719 passages, with reviewed labels through rank 20. V2 also recovers more than half of the first Daecore fine-tune's measured loss on FiQA and SciFact.
 
 Embed queries and passages with the same revision, and re-embed stored passages when switching to v2.
 
@@ -84,31 +84,34 @@ standalone embedder with other retrieval systems.
 
 ### Gemma alone: dense retrieval on Daecore data
 
-Both models search the same **66,240 passages for 376 known-answerable
-queries**, using exact dense retrieval, frozen query variants and matched
-128/1,024-token input limits. BM25 and Ettin do not contribute to this table.
-Each Hit and precision cell reads upstream → **Daecore v2**.
+Both models search the same **82,719 passages for all 970 queries**, using FP32
+exact dense retrieval, the same frozen query forms and matched 128/1,024-token
+input limits. BM25 and Ettin do not contribute to these results. Each cell
+reads upstream → **Daecore v2**.
 
-| Cutoff | Hit | Precision | V2 nDCG |
-|---|---:|---:|---:|
-| 3 | 64.63–65.16% → **75.53%** | 46.63–47.16% → **60.37%** | **0.5393** |
-| 5 | 69.15–70.21% → **82.71%** | 44.73–45.53% → **59.73%** | **0.5389** |
-| 10 | 74.20–76.60% → **88.30%** | 42.63–43.86% → **57.31%** | **0.5452** |
-| 20 | 79.79–87.77% → **91.76%** | 38.46–42.77% → **53.01%** | **0.5635** |
+| Cutoff | Hit | Precision | Known-positive recall | nDCG |
+|---|---:|---:|---:|---:|
+| 1 | 53.40% → **64.74%** | 53.40% → **64.74%** | 1.94% → **2.46%** | 0.4695 → **0.5421** |
+| 3 | 72.99% → **79.38%** | 50.83% → **63.08%** | 5.43% → **6.95%** | 0.4516 → **0.5524** |
+| 5 | 79.48% → **84.74%** | 49.95% → **62.66%** | 8.53% → **11.09%** | 0.4457 → **0.5586** |
+| 10 | 84.85% → **88.14%** | 47.40% → **61.18%** | 15.77% → **20.82%** | 0.4506 → **0.5748** |
+| 20 | 88.76% → **90.52%** | 43.94% → **58.11%** | 28.08% → **38.65%** | 0.4736 → **0.6065** |
 
 ![Gemma v2 dense retrieval compared with upstream](figures/gemma-comparison.svg)
 
-V2's top 20 is fully judged. Upstream has 324 unjudged positions across the
-top-20 lists, so its ranges assign unknown passages first to not useful, then
-to useful. These are missing-label bounds, not confidence intervals. Upstream
-nDCG is withheld until those labels are resolved. Grades 2–3 count as useful;
-nDCG uses graded gains 0, 1, 3 and 7 against the shared pool of known judgments.
+Every top-20 position has a relevance grade or a declared judging abstention;
+there are no missing-label ranges. All 970 queries remain eligible at every
+cutoff. At rank 20, 56 upstream positions and 53 v2 positions are excluded,
+without pulling in deeper results. Precision pools retained positions;
+known-positive recall averages the 911 queries with judged useful evidence.
+The corpus is not exhaustively labeled.
 
-This reused development panel combines query variants and permits at most one
-passage per parent document. It measures the embedding component under that
-recorded procedure; the current full pipeline below deduplicates identical
-text instead. The [evaluation companion](evaluation/README.md)
-includes the anonymous grades, exact model identities and metric code.
+Grades 2–3 count as useful. nDCG uses gains 0, 1, 3 and 7 against the same
+reviewed reference for both models, averaging all 970 queries. Rankings from
+query forms are combined with reciprocal-rank fusion, and identical passage
+text is deduplicated. The [evaluation companion](evaluation/README.md)
+includes anonymized grades, model identities, label coverage and metric code.
+This is a reused development panel, not an untouched test of new workspaces.
 
 ### Gemma alone: public dense retrieval
 
@@ -155,15 +158,17 @@ shared by the Gemma and Ettin cards; it is not either model's standalone score.
 
 | Return depth | Hit | Precision | Known-positive recall | nDCG |
 |---|---:|---:|---:|---:|
+| 1 | 72.33% | 72.33% | 3.46% | 0.6646 |
 | 3 | 85.36% | 71.58% | 9.52% | 0.6577 |
 | 5 | 88.45% | 70.28% | 15.38% | 0.6555 |
 | 10 | 89.90% | 67.18% | 28.03% | 0.6611 |
 | 20 | 91.03% | 61.79% | 48.85% | 0.6841 |
 
-Grades 2–3 count as useful. Hit and nDCG average all 970 queries; precision
-pools the retained positions. Recall averages the 895 queries with known
-positives and measures coverage of judged passages, not of every useful
-passage in the corpus. A shared judging-exclusion set removes 52 positions
+Grades 2–3 count as useful. Depth 1 uses the same 965 queries for both
+providers: five are omitted because at least one first result was ungradable.
+At depths 3–20, Hit and nDCG average all 970 queries. Precision pools retained
+positions. Recall averages known-positive queries (890 at depth 1; 895 at
+depths 3–20) and measures judged passages, not every useful passage in the corpus. A shared judging-exclusion set removes 52 positions
 from the top 20 without backfilling. Daecore's selector chooses a prefix of
 3–20; these fixed-depth scores do not evaluate that choice. The
 [evaluation companion](evaluation/README.md) provides the
@@ -224,7 +229,7 @@ Vulkan runs through the native ONNX Runtime WebGPU plugin, [`onnxruntime-ep-webg
 
 Checks used Windows x64 with an NVIDIA RTX 3060 Ti (8 GB); other GPUs and Linux are untested, even where Vulkan is available.
 
-The task-specific data is mostly generated and model-judged, with no human reference panel, so it describes this development distribution rather than other users' documents. The dense component panel is smaller than the full-pipeline panel, and the two scores should not be compared directly. The 970-query pipeline panel uses natural-language questions and reports no separate terse-query or paraphrase robustness scores. Passage relevance does not measure distinct-fact coverage or downstream agent success.
+The task-specific data is mostly generated and model-judged, with no human reference panel, so it describes this development distribution rather than other users' documents. The dense and full-pipeline panels use the same queries but different judgment pools; compare models within each table. Some questions omit the project or task context needed for an unambiguous judgment. The 970-query pipeline panel uses natural-language questions and reports no separate terse-query or paraphrase robustness scores. Passage relevance does not measure distinct-fact coverage or downstream agent success.
 
 ## License
 

evaluation/README.md

@@ -1,50 +1,63 @@
-# EmbeddingGemma v2 evaluation
-
+# EmbeddingGemma v2 evaluation
+
 These records support the [model card's](../README.md) results: Gemma's dense retrieval on private and public data, its smaller embedding widths, and the full retrieval pipeline. They contain scores and anonymized relevance grades, without private queries or document text.
-
-| File | Contents |
-|---|---|
-| `retriever.json` | Upstream and v2 top-20 grades for 376 known-answerable dense-retrieval queries over 66,240 passages |
+
+| File | Contents |
+|---|---|
+| `retriever.json` | Upstream and v2 reviewed top-20 dense rankings for all 970 queries over 82,719 passages |
 | `dimensions.json` | V2 per-query public nDCG@10 at 768, 512, 256 and 128 dimensions |
 | `promotion.json` | Ranked grades for the predecessor-to-v2 pipeline comparison and public nDCG@10 for upstream and v2 |
 | `serving.json` | The 970-query Gemma v2 + BM25 + Ettin replay used by both model cards, including CUDA/Vulkan results |
 | `*-summary.json` | Summaries that `metrics.py` reproduces |
-| `serving-qualification.json` | CPU, CUDA and Vulkan serving checks for this model |
+| `serving-qualification.json` | CPU, CUDA and Vulkan serving checks for this model |
 | `metrics.py` | Metric code |
 | `figures.py`, `svg_figures.py` | Code to regenerate the Gemma comparison figure |
-
-## Reproduce the tables
-
-From this directory, run with Python 3.11 or later:
-
-```sh
+
+## Reproduce the tables
+
+From this directory, run with Python 3.11 or later:
+
+```sh
 python metrics.py --directory .
 python figures.py --output ../figures
-```
-
+```
+
 No packages or network access are needed. Each result matches its corresponding summary. This recomputes metrics from saved results; running private retrieval again would require the private corpus.
-
+
 ## Dense retrieval
 
-**Private data.** `retriever.json` compares upstream Gemma with v2 on 376
-known-answerable queries over 66,240 passages. Both use FP32 inference, the
-same prefixes and 128-query/1,024-passage token limits, and 768 dimensions.
-Exact dense rankings from frozen query variants are combined into a list
-with at most one passage per parent document. BM25 and Ettin are absent.
-The model identities, source hashes, top-20 grades and reference grade
-counts are included. This is a reused development comparison, not a fresh
-holdout or a dense-only evaluation of the larger 970-query pipeline panel.
-
-V2's top 20 has no missing labels. The upstream rankings contain 324 unjudged
-top-20 positions. Hit and precision therefore retain missing-label bounds;
-they are not confidence intervals. The metric code withholds nDCG for any
-model/cutoff that has unjudged positions rather than treating them as wrong
-or silently dropping their queries. V2 nDCG uses gains 0, 1, 3 and 7 against
-the shared set of known judgments. The corpus is not exhaustively labeled.
+**Private data.** `retriever.json` compares upstream Gemma with v2 on all 970
+queries over the same 82,719 passage texts. Both use FP32 inference, the same
+prefixes and 128-query/1,024-passage token limits, and 768 dimensions. Exact
+cosine rankings take the top 50 for each frozen query form, then merge them
+with reciprocal-rank fusion (constant 60). Identical text is deduplicated;
+there is no parent-document filter, BM25 or reranker. The record includes
+model identities, source hashes, top-20 grades and reference grade counts.
+This is a reused development panel, not a fresh holdout.
+
+Completing the comparison required 11,480 new pair decisions. Two independent
+GPT-6 Sol judges graded each pair, and a fresh blind judgment resolved 3,207
+ordinal disagreements. The primary assistant read 179 complete query–passage
+pairs: all 18 final quote flags, all 72 abstentions, and samples spanning every
+primary grade combination. The 18 quote repairs changed no grades. The result
+adds 11,408 grades to a shared reference containing 87,434 known pairs. No
+person reviewed these judgments, and the corpus is not exhaustively labeled.
+
+Every top-20 position is graded or explicitly excluded as ungradable; no
+missing-label bounds remain. Cutoffs 1, 3, 5, 10 and 20 retain all 970 queries.
+At rank 20, the upstream lists exclude 56 positions and v2 excludes 53.
+Cut first, drop abstentions second, and never backfill. A wholly abstained
+prefix would omit that query from both models at that cutoff; none occurs in
+this comparison. Precision pools retained positions. Recall averages the 911
+queries with known useful passages. nDCG uses gains 0, 1, 3 and 7 against the
+same reference for both models, reindexes retained positions, and averages
+all 970 queries; an ideal gain of zero contributes zero. The dense reference
+is larger than the pipeline reference below, so compare models within each
+table rather than treating their nDCG values as one common scale.
 
 **Public data.**
 The model card compares v2 with upstream EmbeddingGemma on SciFact, FiQA, NFCorpus, SciDocs and ArguAna. `promotion.json` includes per-query nDCG@10 for all five datasets, using full corpora, matched preprocessing and official relevance judgments. V2 remains below upstream on every panel. The previous fine-tune was measured only on FiQA (0.4009) and SciFact (0.7679); its other three results are absent, not estimated.
-
+
 These datasets supplied no training examples, but informed development. They are not untouched tests.
 
 **Matryoshka widths.** `dimensions.json` contains per-query scores from new exact
@@ -61,9 +74,11 @@ panels and do not shorten the encoder's forward pass.
 
 The identical full-pipeline table on both model cards comes from `serving.json`:
 Gemma v2 + BM25 + fusion + unchanged Ettin on all 970 queries, using CUDA FP16
-for reranking. Hit and nDCG average all queries; precision pools retained
-positions. Known-positive recall averages the 895 queries with a judged useful
-passage. The serving reference adds 32 grades to the earlier promotion
+for reranking. At depths 3–20, Hit and nDCG average all queries, and
+known-positive recall averages the 895 with judged useful evidence. Depth 1
+uses 965 shared judged queries, including 890 with known positives; five
+queries are omitted from both providers because a first result was ungradable.
+Precision pools retained positions. The serving reference adds 32 grades to the earlier promotion
 reference. Its shared judging-exclusion set removes 52 top-20 positions per
 provider without backfilling. Fixed cutoffs measure ranking independently of
 the selector's choice of a 3–20-item prefix. Vulkan matches CUDA at Hit@5,
@@ -71,34 +86,35 @@ Hit@10 and Hit@20; Hit@3 differs by one query and nDCG@10 by 0.0004.
 
 The separate predecessor comparison below comes from `promotion.json`.
 Only the embedding model changed in this comparison. BM25, reciprocal-rank fusion and the [Ettin reranker](https://huggingface.co/Daecore/ettin-150m-memory-reranker-ft-v1) stayed fixed. The same score mapping selected returned prefixes in both arms. These results measure the whole pipeline on 970 reused queries over 82,719 passage texts, rather than Gemma alone.
-
-| Metric | Previous Gemma | Gemma v2 |
-|---|---:|---:|
-| Hit@3 | 83.92% | 85.36% |
-| Hit@5 | 85.88% | 88.45% |
-| Hit@10 | 87.84% | 89.90% |
-| Hit@20 | 90.10% | 91.03% |
-| nDCG@10 | 0.6764 | 0.6611 |
-| Selected-prefix precision | 80.41% | 80.34% |
-
-V2 found useful evidence for more queries, while the previous model placed higher-grade passages earlier. Seven reviewed grade corrections apply to both arms. All 970 queries remain. A shared set contains 65 unresolved query–passage judgments. Members of that set are excluded wherever they occur within an original cutoff, without filling their positions with lower-ranked passages. The record preserves those positions as `null` with an `excluded` flag.
-
-Grades 2 and 3 count as useful. Hit@k is the fraction of queries with useful evidence in the first k positions. Precision pools useful passages over retained positions. nDCG uses gain `2**grade - 1` and logarithmic rank discount; it reindexes retained passages and draws the ideal ranking from `reference_grade_counts`. Recall counts known useful passages, not distinct facts. Ties preserve candidate order.
-
-## Query coverage
-
-The shared 970-query panel contains 620 generated-source questions, 200
-questions written with potentially absent evidence, and 150 project-document
-questions. These are natural-language questions. In the pipeline replay, 483
-use the original query alone; 487 multipart queries also use two deterministic
-subquestions. This exercises hybrid retrieval but does not provide a separately
+
+| Metric | Previous Gemma | Gemma v2 |
+|---|---:|---:|
+| Hit@1 (964 queries) | 72.61% | 72.30% |
+| Hit@3 | 83.92% | 85.36% |
+| Hit@5 | 85.88% | 88.45% |
+| Hit@10 | 87.84% | 89.90% |
+| Hit@20 | 90.10% | 91.03% |
+| nDCG@10 | 0.6764 | 0.6611 |
+| Selected-prefix precision | 80.41% | 80.34% |
+
+V2 found useful evidence for more queries, while the previous model placed higher-grade passages earlier. Seven reviewed grade corrections apply to both arms. Depths 3–20 retain all 970 queries; depth 1 omits the same six queries from both models. A shared set contains 65 unresolved query–passage judgments. Members of that set are excluded wherever they occur within an original cutoff, without filling their positions with lower-ranked passages. The record preserves those positions as `null` with an `excluded` flag.
+
+Grades 2 and 3 count as useful. Hit@k is the fraction of queries with useful evidence in the first k positions. Precision pools useful passages over retained positions. nDCG uses gain `2**grade - 1` and logarithmic rank discount; it reindexes retained passages and draws the ideal ranking from `reference_grade_counts`. Recall counts known useful passages, not distinct facts. Ties preserve candidate order.
+
+## Query coverage
+
+The shared 970-query panel contains 620 generated-source questions, 200
+questions written with potentially absent evidence, and 150 project-document
+questions. These are natural-language questions. In the pipeline replay, 483
+use the original query alone; 487 multipart queries also use two deterministic
+subquestions. This exercises hybrid retrieval but does not provide a separately
 balanced terse-query or paraphrase benchmark. Training-query diversity is a
 separate claim from measured robustness on each query style. Dense retrieval
 can also serve keyword and identifier queries; “dense-only” identifies the
 retrieval method, not the query format.
-
-## Serving checks and limits
-
-`serving-qualification.json` records this model's CPU, CUDA and Vulkan checks. The 74-vector panel covers token limits, mixed lengths and concurrent query and passage calls. It is a bounded serving check, separate from retrieval quality.
-
-The private documents and labels are mostly model-generated. Project-document sources come from one person's workspaces and are often AI-written. The panel was reused during development, training was run once, and there is no human reference panel. These results do not establish generalization across users, unique-fact coverage or downstream agent success.
+
+## Serving checks and limits
+
+`serving-qualification.json` records this model's CPU, CUDA and Vulkan checks. The 74-vector panel covers token limits, mixed lengths and concurrent query and passage calls. It is a bounded serving check, separate from retrieval quality.
+
+The private documents and labels are mostly model-generated. Project-document sources come from one person's workspaces and are often AI-written. The panel was reused during development, training was run once, and there is no human reference panel. These results do not establish generalization across users, unique-fact coverage or downstream agent success.

evaluation/figures.py

@@ -1,4 +1,4 @@
-"""Regenerate model comparison SVGs from public labels and scores.
+"""Regenerate model comparison SVGs from the published summaries.
 
 Run with Python 3.11+: python figures.py --output ../figures
 The repository shares its renderer from eval/lib; HF packages include that
@@ -27,8 +27,7 @@ spec.loader.exec_module(svg)
 UPSTREAM, FIT, REFERENCE = '#0072B2', '#D55E00', '#777777'
 
 
-def classifier(data: dict) -> str:
-    summary = metrics.summarize_classifier(data)
+def classifier(summary: dict) -> str:
     facets = [*metrics.FACETS, 'macro']
     def values(model):
         result = summary['models'][model]
@@ -45,9 +44,8 @@ def classifier(data: dict) -> str:
     )
 
 
-def reranker(data: dict) -> str:
-    summary = metrics.summarize_reranker(data)
-    cutoffs = [k for k in metrics.CUTOFFS if k >= 3]
+def reranker(summary: dict) -> str:
+    cutoffs = list(metrics.CUTOFFS)
     definitions = [('hit', 'At least one useful passage'), ('precision', 'Useful passages / returned passages'), ('ndcg', 'Graded ranking quality')]
     panels = []
     legend = [('Upstream', UPSTREAM, None), ('Daecore fine-tune', FIT, None), ('Random pool order', REFERENCE, '5 3')]
@@ -64,29 +62,23 @@ def reranker(data: dict) -> str:
                            legend=legend, panel_width=300, panel_height=270)
 
 
-def retriever(data: dict) -> str:
-    summary = metrics.summarize_retriever(data)
-    cutoffs = [3, 5, 10, 20]
+def retriever(summary: dict) -> str:
+    cutoffs = [1, 3, 5, 10, 20]
     x = list(range(len(cutoffs)))
     panels = []
+    legend = [('Upstream Gemma', UPSTREAM, None), ('Daecore v2', FIT, None)]
     for field, title in [('hit', 'Find at least one useful passage'),
-                         ('precision', 'Fill early results with useful passages')]:
-        original, fitted = (summary['models'][name] for name in ('upstream', 'finetuned'))
-        low = [original[str(k)][f'{field}_lower'] for k in cutoffs]
-        high = [original[str(k)][f'{field}_upper'] for k in cutoffs]
-        panels.append(svg.Panel(title, [
-            svg.Series('Upstream bound', x, high, UPSTREAM, dash='5 3'),
-            svg.Series('Daecore v2', x, [fitted[str(k)][f'{field}_lower'] for k in cutoffs], FIT),
-        ], bands=[svg.Band(x, low, high, UPSTREAM)], xlabel='rank cutoff',
-            ylabel='Hit@k' if field == 'hit' else 'Precision@k',
-            xlim=(0, 3), ylim=(0, 1.02), xticks=list(enumerate(map(str, cutoffs))),
-            notes=['Shading spans unknown-label outcomes, not a confidence interval.'] if field == 'hit' else []))
-    return svg.render_grid(panels, columns=2,
-                           title='Gemma v2: stronger retrieval on Daecore data',
-                           subtitle='376 known-answerable queries · 66,240 passages · dense retrieval only',
-                           legend=[('Upstream: missing-label range', UPSTREAM, '5 3'),
-                                   ('Daecore v2: top 20 fully judged', FIT, None)],
-                           panel_width=430, panel_height=300)
+                         ('precision', 'Useful passages / retained passages'),
+                         ('ndcg', 'Graded ranking quality')]:
+        series = [svg.Series(label, x, [summary['models'][name][str(k)][field] for k in cutoffs], color)
+                  for name, (label, color, _) in zip(('upstream', 'finetuned'), legend, strict=True)]
+        panels.append(svg.Panel(title, series, xlabel='rank cutoff',
+            ylabel={'hit': 'Hit@k', 'precision': 'Precision@k', 'ndcg': 'nDCG@k'}[field],
+            xlim=(0, len(cutoffs) - 1), ylim=(0, 1.02), xticks=list(enumerate(map(str, cutoffs)))))
+    return svg.render_grid(panels, columns=3,
+                           title='Gemma v2: matched dense retrieval on Daecore data',
+                           subtitle=f"{summary['queries']:,} queries · {summary['corpus_passages']:,} passages · reviewed abstentions excluded",
+                           legend=legend, panel_width=300, panel_height=270)
 
 
 def main() -> None:
@@ -101,13 +93,13 @@ def main() -> None:
                    'retriever': ('retriever', 'gemma-comparison.svg')}
     names = [args.model] if args.model else [
         name for name, (record, _) in definitions.items()
-        if (args.directory / f'{record}.json').is_file()
+        if (args.directory / f'{record}-summary.json').is_file()
     ]
     if not names:
-        parser.error('No model comparison records found in the selected directory')
+        parser.error('No model comparison summaries found in the selected directory')
     for name in names:
         record, filename = definitions[name]
-        data = json.loads((args.directory / f'{record}.json').read_text(encoding='utf-8'))
+        data = json.loads((args.directory / f'{record}-summary.json').read_text(encoding='utf-8'))
         (args.output / filename).write_text(globals()[name](data), encoding='utf-8', newline='\n')
 
 

evaluation/metrics.py

@@ -117,43 +117,11 @@ def summarize_reranker(data: dict) -> dict:
 
 
 def summarize_retriever(data: dict) -> dict:
-    """Dense rankings: retain unknown-label bounds and withhold incomplete nDCG."""
-    rows = data['rows']
-    if not rows or len({row['id'] for row in rows}) != len(rows):
-        raise ValueError('Dense comparison requires distinct query identities')
-    output = {'queries': len(rows), 'corpus_passages': data['corpus_passages'], 'models': {}}
-    for model in data['model_order']:
-        cutoffs = {}
-        for k in (3, 5, 10, 20):
-            values = []
-            for row in rows:
-                counts = row['reference_grade_counts']
-                if set(counts) != {'0', '1', '2', '3'} or any(type(n) is not int or n < 0 for n in counts.values()):
-                    raise ValueError('Reference grade counts must cover grades zero through three')
-                grades = row['ranked_grades'][model][:k]
-                if len(grades) != k or any(g is not None and (type(g) is not int or g not in range(4)) for g in grades):
-                    raise ValueError('Dense rows must contain each ranked grade or null')
-                if counts['2'] + counts['3'] == 0:
-                    raise ValueError('Dense comparison contains only known-answerable queries')
-                ideal = [g for g in (3, 2, 1, 0) for _ in range(min(counts[str(g)], k))][:k]
-                useful, unknown = sum(g is not None and g >= 2 for g in grades), grades.count(None)
-                values.append({
-                    'hit_lower': float(useful > 0), 'hit_upper': float(useful + unknown > 0),
-                    'precision_lower': useful / k, 'precision_upper': (useful + unknown) / k,
-                    'unjudged_positions': unknown,
-                    'ndcg': None if unknown else dcg(grades, k) / dcg(ideal, k),
-                })
-            cutoffs[str(k)] = {
-                field: mean(row[field] for row in values)
-                for field in ('hit_lower', 'hit_upper', 'precision_lower', 'precision_upper')
-            }
-            cutoffs[str(k)]['unjudged_positions'] = sum(row['unjudged_positions'] for row in values)
-            cutoffs[str(k)]['ndcg'] = (
-                None if any(row['ndcg'] is None for row in values)
-                else mean(row['ndcg'] for row in values)
-            )
-        output['models'][model] = cutoffs
-    return output
+    """Fully reviewed dense rankings, including queries without known positives."""
+    if type(data['corpus_passages']) is not int or data['corpus_passages'] <= 0:
+        raise ValueError('Dense comparison requires a positive corpus size')
+    return {**_summarize_ranked_grades(data, include_selected=False),
+            'corpus_passages': data['corpus_passages']}
 
 
 def summarize_dimensions(data: dict) -> dict:
@@ -178,36 +146,52 @@ def summarize_dimensions(data: dict) -> dict:
     return {'queries': len(rows), 'vector_bytes_fp32': {str(w): w * 4 for w in widths}, 'public': panels}
 
 
-def summarize_promotion(data: dict) -> dict:
-    """Hybrid search with reviewed exclusions inside each original prefix.
+def _summarize_ranked_grades(data: dict, *, include_selected: bool) -> dict:
+    """Reviewed exclusions inside each original dense or hybrid prefix.
 
     A null is permitted only for an explicitly excluded judging abstention.
     Cut first, remove exclusions second, and never backfill from a deeper rank.
     """
     rows = data['rows']
     if not rows or len({row['id'] for row in rows}) != len(rows):
-        raise ValueError('Promotion rows require unique nonempty query identities')
-    output = {'queries': len(rows), 'models': {}, 'public': {}}
+        raise ValueError('Ranked rows require unique nonempty query identities')
+    # Validate before excluding fully abstained prefixes. An omitted query
+    # must never conceal a malformed grade, model map or selected depth.
+    for row in rows:
+        counts = row['reference_grade_counts']
+        if set(counts) != {'0', '1', '2', '3'} or any(type(n) is not int or n < 0 for n in counts.values()):
+            raise ValueError('Reference grade counts must cover grades zero through three')
+        for model in data['model_order']:
+            ranked, excluded = row['ranked_grades'][model], row['excluded'][model]
+            if (len(ranked) != 20 or len(excluded) != 20
+                    or any(type(x) is not bool for x in excluded)):
+                raise ValueError('Ranked rows require a bounded original top twenty')
+            if include_selected:
+                depth = row['selected_depth'][model]
+                if type(depth) is not int or not 3 <= depth <= 20:
+                    raise ValueError('Selected depth must be between three and twenty')
+            if any((grade is not None if drop else type(grade) is not int or grade not in range(4))
+                   for grade, drop in zip(ranked, excluded, strict=True)):
+                raise ValueError('Only declared abstentions may lack grades')
+    output = {'queries': len(rows), 'models': {}}
+    requested_cutoffs = (1, 3, 5, 10, 20, 'selected') if include_selected else (1, 3, 5, 10, 20)
     for model in data['model_order']:
         cutoffs = {}
-        for cutoff in (3, 5, 10, 20, 'selected'):
+        for cutoff in requested_cutoffs:
+            eligible = [row for row in rows if all(
+                any(not drop for drop in row['excluded'][arm][:(
+                    row['selected_depth'][arm] if cutoff == 'selected' else cutoff)])
+                for arm in data['model_order']
+            )]
+            if not eligible:
+                raise ValueError('No shared judged queries at this cutoff')
             per_query = []
-            for row in rows:
+            for row in eligible:
                 counts = row['reference_grade_counts']
-                if set(counts) != {'0', '1', '2', '3'} or any(type(n) is not int or n < 0 for n in counts.values()):
-                    raise ValueError('Reference grade counts must cover grades zero through three')
                 ranked = row['ranked_grades'][model]
                 excluded = row['excluded'][model]
                 depth = row['selected_depth'][model] if cutoff == 'selected' else cutoff
-                if (len(ranked) != 20 or len(excluded) != 20 or type(depth) is not int
-                        or not 3 <= depth <= 20 or any(type(x) is not bool for x in excluded)):
-                    raise ValueError('Promotion rows require a bounded original top twenty')
-                if any((grade is not None if drop else type(grade) is not int or grade not in range(4))
-                       for grade, drop in zip(ranked, excluded, strict=True)):
-                    raise ValueError('Only declared abstentions may lack grades')
                 kept = [grade for grade, drop in zip(ranked[:depth], excluded[:depth], strict=True) if not drop]
-                if not kept:
-                    raise ValueError('Every scored prefix must retain judged passages')
                 ideal_grades = [g for g in (3, 2, 1, 0) for _ in range(min(counts[str(g)], len(kept)))][:len(kept)]
                 useful = sum(g >= 2 for g in kept)
                 positives = counts['2'] + counts['3']
@@ -222,15 +206,22 @@ def summarize_promotion(data: dict) -> dict:
             retained = sum(row['retained'] for row in per_query)
             recalls = [row['known_recall'] for row in per_query if row['known_recall'] is not None]
             cutoffs[str(cutoff)] = {
+                'scored_queries': len(eligible), 'excluded_queries': len(rows) - len(eligible),
                 'hit': mean(row['hit'] for row in per_query), 'precision': useful / retained,
                 'macro_precision': mean(row['precision'] for row in per_query),
                 'ndcg': mean(row['ndcg'] for row in per_query),
                 'known_positive_recall': mean(recalls) if recalls else None,
                 'recall_queries': len(recalls), 'useful': useful, 'retained': retained,
                 'excluded_positions': sum(row['excluded'] for row in per_query),
-                'mean_useful': useful / len(rows), 'mean_retained': retained / len(rows),
+                'mean_useful': useful / len(eligible), 'mean_retained': retained / len(eligible),
             }
         output['models'][model] = cutoffs
+    return output
+
+
+def summarize_promotion(data: dict) -> dict:
+    """Hybrid rankings and the separately measured public dense panels."""
+    output = {**_summarize_ranked_grades(data, include_selected=True), 'public': {}}
     for dataset, panel in data['public'].items():
         if not panel['rows'] or len({row['id'] for row in panel['rows']}) != len(panel['rows']):
             raise ValueError('Public panel requires distinct query identities')
@@ -249,28 +240,86 @@ def summarize_promotion(data: dict) -> dict:
     return output
 
 
+SUMMARIZERS = {
+    'classifier': summarize_classifier, 'reranker': summarize_reranker,
+    'retriever': summarize_retriever, 'dimensions': summarize_dimensions,
+    'promotion': summarize_promotion, 'serving': summarize_promotion,
+}
+
+
+def summarize_record(name: str, data: dict) -> dict:
+    """Check anonymous publication records before recomputing their summary."""
+    fields = {
+        'classifier': {'id', 'group', 'labels', 'scores'},
+        'reranker': {'id', 'group', 'grades', 'scores'},
+        'retriever': {'id', 'group', 'reference_grade_counts', 'ranked_grades', 'excluded'},
+        'dimensions': {'id', 'dataset', 'ndcg@10'},
+        'promotion': {'id', 'group', 'reference_grade_counts', 'ranked_grades', 'excluded', 'selected_depth'},
+        'serving': {'id', 'group', 'reference_grade_counts', 'ranked_grades', 'excluded', 'selected_depth'},
+    }[name]
+    rows = data['rows']
+    if not rows or any(set(row) != fields for row in rows):
+        raise ValueError(f'Unexpected evaluation row fields in {name}')
+    ids = [row['id'] for row in rows]
+    if len(set(ids)) != len(ids):
+        raise ValueError(f'Repeated query/passage identity in {name}')
+    for row in rows:
+        identity = row['id']
+        if not isinstance(identity, str) or identity[:1] not in ('p', 'q') or not identity[1:].isdigit():
+            raise ValueError(f'Non-anonymous row identity in {name}')
+        if 'group' in row and (not isinstance(row['group'], str) or not row['group'].startswith('g') or not row['group'][1:].isdigit()):
+            raise ValueError(f'Non-anonymous group identity in {name}')
+    if name != 'dimensions' and set(data['model_order']) != set(data['models']):
+        raise ValueError(f'Model identities do not match in {name}')
+    models = set(data.get('model_order', ()))
+    for row in rows:
+        for field in ('scores', 'ranked_grades', 'excluded', 'selected_depth'):
+            if field in row and set(row[field]) != models:
+                raise ValueError(f'Unexpected model fields in {name}.{field}')
+        if name == 'classifier':
+            required = {facet for facet, label in row['labels'].items() if label is not None}
+            if set(row['labels']) != set(FACETS) or any(not required <= set(scores) <= set(FACETS) for scores in row['scores'].values()):
+                raise ValueError('Unexpected classifier facet fields')
+            if any(type(value) not in (int, float) or not math.isfinite(value)
+                   for scores in row['scores'].values() for value in scores.values()):
+                raise ValueError('Classifier scores must be finite numbers, including unresolved facets')
+    for panel in data.get('public', {}).values():
+        if set(panel) != {'model_order', 'rows'} or not panel['rows']:
+            raise ValueError('Unexpected public-panel fields')
+        panel_models = set(panel['model_order'])
+        panel_ids = []
+        for row in panel['rows']:
+            if set(row) != {'id', 'ndcg@10'} or set(row['ndcg@10']) != panel_models:
+                raise ValueError('Unexpected public-panel row fields')
+            identity = row['id']
+            if not isinstance(identity, str) or identity[:1] != 'q' or not identity[1:].isdigit():
+                raise ValueError('Non-anonymous public-panel row identity')
+            if any(type(value) not in (int, float) or not math.isfinite(value) or not 0 <= value <= 1 for value in row['ndcg@10'].values()):
+                raise ValueError('Invalid public-panel score')
+            panel_ids.append(identity)
+        if len(panel_ids) != len(set(panel_ids)):
+            raise ValueError('Repeated public-panel query identity')
+    text = json.dumps(data).lower()
+    if any(value in text for value in ('c:\\', 'eval/private', 'qrel_target_id', 'query_text', 'passage_text')):
+        raise ValueError(f'Private payload in {name}')
+    return SUMMARIZERS[name](data)
+
+
 def main() -> None:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--directory", type=Path, default=Path(__file__).resolve().parent)
     parser.add_argument("--output", type=Path)
     parser.add_argument("--model", choices=("classifier", "reranker", "retriever", "dimensions", "promotion", "serving"), help="Recompute one comparison; default: every record in this package")
     args = parser.parse_args()
-    summarizers = {"classifier": summarize_classifier, "reranker": summarize_reranker,
-                   "retriever": summarize_retriever, "dimensions": summarize_dimensions,
-                   "promotion": summarize_promotion, "serving": summarize_promotion}
     names = [args.model] if args.model else [
-        name for name in summarizers if (args.directory / f"{name}.json").is_file()
+        name for name in SUMMARIZERS if (args.directory / f"{name}.json").is_file()
     ]
     if not names:
         parser.error("No evaluation records found in the selected directory")
     result = {}
     for name in names:
-        summarize = summarizers[name]
         data = json.loads((args.directory / f"{name}.json").read_text(encoding="utf-8"))
-        ids = [r["id"] for r in data["rows"]]
-        if len(set(ids)) != len(ids):
-            raise ValueError(f"Repeated query/passage identity in {name}")
-        result[name] = summarize(data)
+        result[name] = summarize_record(name, data)
     text = json.dumps(result, indent=2, allow_nan=False) + "\n"
     if args.output:
         args.output.write_text(text, encoding="utf-8")

evaluation/promotion-summary.json

@@ -1 +1,228 @@
-{"queries":970,"models":{"production":{"3":{"hit":0.8391752577319588,"precision":0.7108890420399724,"macro_precision":0.7109965635738832,"ndcg":0.6737359241548024,"known_positive_recall":0.09151322333990437,"recall_queries":895,"useful":2063,"retained":2902,"excluded_positions":8,"mean_useful":2.12680412371134,"mean_retained":2.9917525773195877},"5":{"hit":0.8587628865979381,"precision":0.6968194960760017,"macro_precision":0.6970103092783505,"ndcg":0.6720934096988599,"known_positive_recall":0.14456571515697228,"recall_queries":895,"useful":3374,"retained":4842,"excluded_positions":8,"mean_useful":3.4783505154639176,"mean_retained":4.991752577319588},"10":{"hit":0.8783505154639175,"precision":0.6640181611804767,"macro_precision":0.6641809851088202,"ndcg":0.6764288375969912,"known_positive_recall":0.2647168884118442,"recall_queries":895,"useful":6435,"retained":9691,"excluded_positions":9,"mean_useful":6.634020618556701,"mean_retained":9.990721649484536},"20":{"hit":0.9010309278350516,"precision":0.6041172221648953,"macro_precision":0.604256841821554,"ndcg":0.6943443750382963,"known_positive_recall":0.4574631459961227,"recall_queries":895,"useful":11709,"retained":19382,"excluded_positions":18,"mean_useful":12.071134020618556,"mean_retained":19.981443298969072},"selected":{"hit":0.8525773195876288,"precision":0.8040927303949628,"macro_precision":0.7015133181852246,"ndcg":0.67618405088445,"known_positive_recall":0.19797868631055085,"recall_queries":895,"useful":5619,"retained":6988,"excluded_positions":8,"mean_useful":5.792783505154639,"mean_retained":7.204123711340206}},"candidate":{"3":{"hit":0.8536082474226804,"precision":0.7157640565712314,"macro_precision":0.7166666666666667,"ndcg":0.6576832474683237,"known_positive_recall":0.09526530468586906,"recall_queries":895,"useful":2075,"retained":2899,"excluded_positions":11,"mean_useful":2.1391752577319587,"mean_retained":2.988659793814433},"5":{"hit":0.8845360824742268,"precision":0.7028311634635255,"macro_precision":0.7031786941580757,"ndcg":0.6554718901879515,"known_positive_recall":0.15391852985862914,"recall_queries":895,"useful":3401,"retained":4839,"excluded_positions":11,"mean_useful":3.5061855670103093,"mean_retained":4.988659793814433},"10":{"hit":0.8989690721649485,"precision":0.67180070291503,"macro_precision":0.6722631320569465,"ndcg":0.6610872123112499,"known_positive_recall":0.28045885967323153,"recall_queries":895,"useful":6499,"retained":9674,"excluded_positions":26,"mean_useful":6.7,"mean_retained":9.97319587628866},"20":{"hit":0.9103092783505154,"precision":0.61794500723589,"macro_precision":0.6184956360634603,"ndcg":0.684223528795109,"known_positive_recall":0.4887591886594314,"recall_queries":895,"useful":11956,"retained":19348,"excluded_positions":52,"mean_useful":12.32577319587629,"mean_retained":19.94639175257732},"selected":{"hit":0.8618556701030928,"precision":0.8033770583310076,"macro_precision":0.7040325511660611,"ndcg":0.6599128936120324,"known_positive_recall":0.20805438995183476,"recall_queries":895,"useful":5757,"retained":7166,"excluded_positions":17,"mean_useful":5.935051546391753,"mean_retained":7.387628865979382}}},"public":{"scifact":{"queries":300,"ndcg@10":{"upstream":0.7875641458078616,"production":0.7678535660256963,"candidate":0.7782845713792622}},"fiqa":{"queries":648,"ndcg@10":{"upstream":0.474145026242454,"production":0.4008741437969072,"candidate":0.446838124778398}},"nfcorpus":{"queries":323,"ndcg@10":{"upstream":0.3932480325545885,"candidate":0.3889646280478944}},"scidocs":{"queries":1000,"ndcg@10":{"upstream":0.19447454865159183,"candidate":0.18521860884727676}},"arguana":{"queries":1406,"ndcg@10":{"upstream":0.6431541443325537,"candidate":0.6258634411077135}}}}
+{
+  "queries": 970,
+  "models": {
+    "production": {
+      "1": {
+        "scored_queries": 964,
+        "excluded_queries": 6,
+        "hit": 0.7261410788381742,
+        "precision": 0.7261410788381742,
+        "macro_precision": 0.7261410788381742,
+        "ndcg": 0.6822762299940723,
+        "known_positive_recall": 0.03374565629667812,
+        "recall_queries": 889,
+        "useful": 700,
+        "retained": 964,
+        "excluded_positions": 0,
+        "mean_useful": 0.7261410788381742,
+        "mean_retained": 1.0
+      },
+      "3": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8391752577319588,
+        "precision": 0.7108890420399724,
+        "macro_precision": 0.7109965635738832,
+        "ndcg": 0.6737359241548024,
+        "known_positive_recall": 0.09151322333990437,
+        "recall_queries": 895,
+        "useful": 2063,
+        "retained": 2902,
+        "excluded_positions": 8,
+        "mean_useful": 2.12680412371134,
+        "mean_retained": 2.9917525773195877
+      },
+      "5": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8587628865979381,
+        "precision": 0.6968194960760017,
+        "macro_precision": 0.6970103092783505,
+        "ndcg": 0.6720934096988599,
+        "known_positive_recall": 0.14456571515697228,
+        "recall_queries": 895,
+        "useful": 3374,
+        "retained": 4842,
+        "excluded_positions": 8,
+        "mean_useful": 3.4783505154639176,
+        "mean_retained": 4.991752577319588
+      },
+      "10": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8783505154639175,
+        "precision": 0.6640181611804767,
+        "macro_precision": 0.6641809851088202,
+        "ndcg": 0.6764288375969912,
+        "known_positive_recall": 0.2647168884118442,
+        "recall_queries": 895,
+        "useful": 6435,
+        "retained": 9691,
+        "excluded_positions": 9,
+        "mean_useful": 6.634020618556701,
+        "mean_retained": 9.990721649484536
+      },
+      "20": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9010309278350516,
+        "precision": 0.6041172221648953,
+        "macro_precision": 0.604256841821554,
+        "ndcg": 0.6943443750382963,
+        "known_positive_recall": 0.4574631459961227,
+        "recall_queries": 895,
+        "useful": 11709,
+        "retained": 19382,
+        "excluded_positions": 18,
+        "mean_useful": 12.071134020618556,
+        "mean_retained": 19.981443298969072
+      },
+      "selected": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8525773195876288,
+        "precision": 0.8040927303949628,
+        "macro_precision": 0.7015133181852246,
+        "ndcg": 0.67618405088445,
+        "known_positive_recall": 0.19797868631055085,
+        "recall_queries": 895,
+        "useful": 5619,
+        "retained": 6988,
+        "excluded_positions": 8,
+        "mean_useful": 5.792783505154639,
+        "mean_retained": 7.204123711340206
+      }
+    },
+    "candidate": {
+      "1": {
+        "scored_queries": 964,
+        "excluded_queries": 6,
+        "hit": 0.7230290456431535,
+        "precision": 0.7230290456431535,
+        "macro_precision": 0.7230290456431535,
+        "ndcg": 0.6648389646314957,
+        "known_positive_recall": 0.03456403636248349,
+        "recall_queries": 889,
+        "useful": 697,
+        "retained": 964,
+        "excluded_positions": 0,
+        "mean_useful": 0.7230290456431535,
+        "mean_retained": 1.0
+      },
+      "3": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8536082474226804,
+        "precision": 0.7157640565712314,
+        "macro_precision": 0.7166666666666667,
+        "ndcg": 0.6576832474683237,
+        "known_positive_recall": 0.09526530468586906,
+        "recall_queries": 895,
+        "useful": 2075,
+        "retained": 2899,
+        "excluded_positions": 11,
+        "mean_useful": 2.1391752577319587,
+        "mean_retained": 2.988659793814433
+      },
+      "5": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8845360824742268,
+        "precision": 0.7028311634635255,
+        "macro_precision": 0.7031786941580757,
+        "ndcg": 0.6554718901879515,
+        "known_positive_recall": 0.15391852985862914,
+        "recall_queries": 895,
+        "useful": 3401,
+        "retained": 4839,
+        "excluded_positions": 11,
+        "mean_useful": 3.5061855670103093,
+        "mean_retained": 4.988659793814433
+      },
+      "10": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8989690721649485,
+        "precision": 0.67180070291503,
+        "macro_precision": 0.6722631320569465,
+        "ndcg": 0.6610872123112499,
+        "known_positive_recall": 0.28045885967323153,
+        "recall_queries": 895,
+        "useful": 6499,
+        "retained": 9674,
+        "excluded_positions": 26,
+        "mean_useful": 6.7,
+        "mean_retained": 9.97319587628866
+      },
+      "20": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9103092783505154,
+        "precision": 0.61794500723589,
+        "macro_precision": 0.6184956360634603,
+        "ndcg": 0.684223528795109,
+        "known_positive_recall": 0.4887591886594314,
+        "recall_queries": 895,
+        "useful": 11956,
+        "retained": 19348,
+        "excluded_positions": 52,
+        "mean_useful": 12.32577319587629,
+        "mean_retained": 19.94639175257732
+      },
+      "selected": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8618556701030928,
+        "precision": 0.8033770583310076,
+        "macro_precision": 0.7040325511660611,
+        "ndcg": 0.6599128936120324,
+        "known_positive_recall": 0.20805438995183476,
+        "recall_queries": 895,
+        "useful": 5757,
+        "retained": 7166,
+        "excluded_positions": 17,
+        "mean_useful": 5.935051546391753,
+        "mean_retained": 7.387628865979382
+      }
+    }
+  },
+  "public": {
+    "scifact": {
+      "queries": 300,
+      "ndcg@10": {
+        "upstream": 0.7875641458078616,
+        "production": 0.7678535660256963,
+        "candidate": 0.7782845713792622
+      }
+    },
+    "fiqa": {
+      "queries": 648,
+      "ndcg@10": {
+        "upstream": 0.474145026242454,
+        "production": 0.4008741437969072,
+        "candidate": 0.446838124778398
+      }
+    },
+    "nfcorpus": {
+      "queries": 323,
+      "ndcg@10": {
+        "upstream": 0.3932480325545885,
+        "candidate": 0.3889646280478944
+      }
+    },
+    "scidocs": {
+      "queries": 1000,
+      "ndcg@10": {
+        "upstream": 0.19447454865159183,
+        "candidate": 0.18521860884727676
+      }
+    },
+    "arguana": {
+      "queries": 1406,
+      "ndcg@10": {
+        "upstream": 0.6431541443325537,
+        "candidate": 0.6258634411077135
+      }
+    }
+  }
+}

evaluation/retriever-summary.json

@@ -1,74 +1,160 @@
-{
-  "queries": 376,
-  "corpus_passages": 66240,
-  "models": {
-    "upstream": {
-      "3": {
-        "hit_lower": 0.6462765957446809,
-        "hit_upper": 0.651595744680851,
-        "precision_lower": 0.46631205673758863,
-        "precision_upper": 0.4716312056737589,
-        "unjudged_positions": 6,
-        "ndcg": null
-      },
-      "5": {
-        "hit_lower": 0.6914893617021277,
-        "hit_upper": 0.7021276595744681,
-        "precision_lower": 0.4473404255319149,
-        "precision_upper": 0.4553191489361702,
-        "unjudged_positions": 15,
-        "ndcg": null
-      },
-      "10": {
-        "hit_lower": 0.7420212765957447,
-        "hit_upper": 0.7659574468085106,
-        "precision_lower": 0.4263297872340426,
-        "precision_upper": 0.43856382978723407,
-        "unjudged_positions": 46,
-        "ndcg": null
-      },
-      "20": {
-        "hit_lower": 0.7978723404255319,
-        "hit_upper": 0.8776595744680851,
-        "precision_lower": 0.38457446808510637,
-        "precision_upper": 0.4276595744680851,
-        "unjudged_positions": 324,
-        "ndcg": null
-      }
-    },
-    "finetuned": {
-      "3": {
-        "hit_lower": 0.7553191489361702,
-        "hit_upper": 0.7553191489361702,
-        "precision_lower": 0.6037234042553191,
-        "precision_upper": 0.6037234042553191,
-        "unjudged_positions": 0,
-        "ndcg": 0.5392937674390154
-      },
-      "5": {
-        "hit_lower": 0.8271276595744681,
-        "hit_upper": 0.8271276595744681,
-        "precision_lower": 0.5973404255319149,
-        "precision_upper": 0.5973404255319149,
-        "unjudged_positions": 0,
-        "ndcg": 0.538938189801325
-      },
-      "10": {
-        "hit_lower": 0.8829787234042553,
-        "hit_upper": 0.8829787234042553,
-        "precision_lower": 0.5731382978723404,
-        "precision_upper": 0.5731382978723404,
-        "unjudged_positions": 0,
-        "ndcg": 0.5452229890785482
-      },
-      "20": {
-        "hit_lower": 0.9175531914893617,
-        "hit_upper": 0.9175531914893617,
-        "precision_lower": 0.5300531914893617,
-        "precision_upper": 0.5300531914893617,
-        "unjudged_positions": 0,
-        "ndcg": 0.5635282364004299
-      }
-    }
-  }
-}
+{
+  "queries": 970,
+  "models": {
+    "upstream": {
+      "1": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.534020618556701,
+        "precision": 0.534020618556701,
+        "macro_precision": 0.534020618556701,
+        "ndcg": 0.4695139911634757,
+        "known_positive_recall": 0.019407636059998304,
+        "recall_queries": 911,
+        "useful": 518,
+        "retained": 970,
+        "excluded_positions": 0,
+        "mean_useful": 0.534020618556701,
+        "mean_retained": 1.0
+      },
+      "3": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.7298969072164948,
+        "precision": 0.5082530949105915,
+        "macro_precision": 0.5082474226804123,
+        "ndcg": 0.4515657513806275,
+        "known_positive_recall": 0.05434321461039996,
+        "recall_queries": 911,
+        "useful": 1478,
+        "retained": 2908,
+        "excluded_positions": 2,
+        "mean_useful": 1.5237113402061855,
+        "mean_retained": 2.997938144329897
+      },
+      "5": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.7948453608247422,
+        "precision": 0.49948379103861246,
+        "macro_precision": 0.4996219931271478,
+        "ndcg": 0.4457312507652407,
+        "known_positive_recall": 0.08530117331466223,
+        "recall_queries": 911,
+        "useful": 2419,
+        "retained": 4843,
+        "excluded_positions": 7,
+        "mean_useful": 2.4938144329896907,
+        "mean_retained": 4.992783505154639
+      },
+      "10": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8484536082474227,
+        "precision": 0.4739561802397685,
+        "macro_precision": 0.47407543773523153,
+        "ndcg": 0.45062269194348514,
+        "known_positive_recall": 0.15770613185252844,
+        "recall_queries": 911,
+        "useful": 4586,
+        "retained": 9676,
+        "excluded_positions": 24,
+        "mean_useful": 4.727835051546392,
+        "mean_retained": 9.975257731958763
+      },
+      "20": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8876288659793814,
+        "precision": 0.4393610421836228,
+        "macro_precision": 0.44005992535723476,
+        "ndcg": 0.4736202687789316,
+        "known_positive_recall": 0.2807541608701895,
+        "recall_queries": 911,
+        "useful": 8499,
+        "retained": 19344,
+        "excluded_positions": 56,
+        "mean_useful": 8.761855670103094,
+        "mean_retained": 19.942268041237114
+      }
+    },
+    "finetuned": {
+      "1": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.6474226804123712,
+        "precision": 0.6474226804123712,
+        "macro_precision": 0.6474226804123712,
+        "ndcg": 0.5420716740304369,
+        "known_positive_recall": 0.02462514980632024,
+        "recall_queries": 911,
+        "useful": 628,
+        "retained": 970,
+        "excluded_positions": 0,
+        "mean_useful": 0.6474226804123712,
+        "mean_retained": 1.0
+      },
+      "3": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.7938144329896907,
+        "precision": 0.6308009625300791,
+        "macro_precision": 0.6309278350515464,
+        "ndcg": 0.5523629493718011,
+        "known_positive_recall": 0.06947139492907671,
+        "recall_queries": 911,
+        "useful": 1835,
+        "retained": 2909,
+        "excluded_positions": 1,
+        "mean_useful": 1.8917525773195876,
+        "mean_retained": 2.9989690721649485
+      },
+      "5": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8474226804123711,
+        "precision": 0.6266005782734407,
+        "macro_precision": 0.6269759450171821,
+        "ndcg": 0.558603868737523,
+        "known_positive_recall": 0.11088922136775149,
+        "recall_queries": 911,
+        "useful": 3034,
+        "retained": 4842,
+        "excluded_positions": 8,
+        "mean_useful": 3.1278350515463917,
+        "mean_retained": 4.991752577319588
+      },
+      "10": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8814432989690721,
+        "precision": 0.6117999586691465,
+        "macro_precision": 0.6123367697594502,
+        "ndcg": 0.5748078775849167,
+        "known_positive_recall": 0.2082023156474653,
+        "recall_queries": 911,
+        "useful": 5921,
+        "retained": 9678,
+        "excluded_positions": 22,
+        "mean_useful": 6.104123711340206,
+        "mean_retained": 9.977319587628866
+      },
+      "20": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9051546391752577,
+        "precision": 0.5810720008270016,
+        "macro_precision": 0.5815832221977094,
+        "ndcg": 0.6064961581206146,
+        "known_positive_recall": 0.38649540366398033,
+        "recall_queries": 911,
+        "useful": 11242,
+        "retained": 19347,
+        "excluded_positions": 53,
+        "mean_useful": 11.589690721649484,
+        "mean_retained": 19.945360824742266
+      }
+    }
+  },
+  "corpus_passages": 82719
+}

evaluation/serving-summary.json

@@ -1 +1,190 @@
-{"queries":970,"models":{"cuda":{"3":{"hit":0.8536082474226804,"precision":0.7157640565712314,"macro_precision":0.7166666666666667,"ndcg":0.6576832474683237,"known_positive_recall":0.09521315062968948,"recall_queries":895,"useful":2075,"retained":2899,"excluded_positions":11,"mean_useful":2.1391752577319587,"mean_retained":2.988659793814433},"5":{"hit":0.8845360824742268,"precision":0.7028311634635255,"macro_precision":0.7031786941580757,"ndcg":0.6554718901879515,"known_positive_recall":0.15383335671735354,"recall_queries":895,"useful":3401,"retained":4839,"excluded_positions":11,"mean_useful":3.5061855670103093,"mean_retained":4.988659793814433},"10":{"hit":0.8989690721649485,"precision":0.67180070291503,"macro_precision":0.6722631320569465,"ndcg":0.6610872123112499,"known_positive_recall":0.2802986435290336,"recall_queries":895,"useful":6499,"retained":9674,"excluded_positions":26,"mean_useful":6.7,"mean_retained":9.97319587628866},"20":{"hit":0.9103092783505154,"precision":0.61794500723589,"macro_precision":0.6184956360634603,"ndcg":0.6840908385828889,"known_positive_recall":0.4884616340614924,"recall_queries":895,"useful":11956,"retained":19348,"excluded_positions":52,"mean_useful":12.32577319587629,"mean_retained":19.94639175257732},"selected":{"hit":0.8618556701030928,"precision":0.8033770583310076,"macro_precision":0.7040325511660611,"ndcg":0.6598955189009045,"known_positive_recall":0.2079138769097556,"recall_queries":895,"useful":5757,"retained":7166,"excluded_positions":17,"mean_useful":5.935051546391753,"mean_retained":7.387628865979382}},"vulkan":{"3":{"hit":0.8525773195876288,"precision":0.7157640565712314,"macro_precision":0.7166666666666667,"ndcg":0.6577263968717306,"known_positive_recall":0.09519545071125639,"recall_queries":895,"useful":2075,"retained":2899,"excluded_positions":11,"mean_useful":2.1391752577319587,"mean_retained":2.988659793814433},"5":{"hit":0.8845360824742268,"precision":0.7023563455973543,"macro_precision":0.702766323024055,"ndcg":0.6550340124265787,"known_positive_recall":0.15374094654114448,"recall_queries":895,"useful":3398,"retained":4838,"excluded_positions":12,"mean_useful":3.5030927835051546,"mean_retained":4.9876288659793815},"10":{"hit":0.8989690721649485,"precision":0.6723514211886304,"macro_precision":0.6727589592538046,"ndcg":0.6615310405020679,"known_positive_recall":0.2805978824757964,"recall_queries":895,"useful":6505,"retained":9675,"excluded_positions":25,"mean_useful":6.706185567010309,"mean_retained":9.974226804123711},"20":{"hit":0.9103092783505154,"precision":0.6178416373785404,"macro_precision":0.6183925432799551,"ndcg":0.6840389893720067,"known_positive_recall":0.4883516349357555,"recall_queries":895,"useful":11954,"retained":19348,"excluded_positions":52,"mean_useful":12.323711340206186,"mean_retained":19.94639175257732},"selected":{"hit":0.8608247422680413,"precision":0.8027855153203343,"macro_precision":0.7038533836490732,"ndcg":0.6599601279630594,"known_positive_recall":0.2082001919384574,"recall_queries":895,"useful":5764,"retained":7180,"excluded_positions":17,"mean_useful":5.942268041237114,"mean_retained":7.402061855670103}}},"public":{}}
+{
+  "queries": 970,
+  "models": {
+    "cuda": {
+      "1": {
+        "scored_queries": 965,
+        "excluded_queries": 5,
+        "hit": 0.7233160621761658,
+        "precision": 0.7233160621761658,
+        "macro_precision": 0.7233160621761658,
+        "ndcg": 0.664594127806563,
+        "known_positive_recall": 0.0346020393358114,
+        "recall_queries": 890,
+        "useful": 698,
+        "retained": 965,
+        "excluded_positions": 0,
+        "mean_useful": 0.7233160621761658,
+        "mean_retained": 1.0
+      },
+      "3": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8536082474226804,
+        "precision": 0.7157640565712314,
+        "macro_precision": 0.7166666666666667,
+        "ndcg": 0.6576832474683237,
+        "known_positive_recall": 0.09521315062968948,
+        "recall_queries": 895,
+        "useful": 2075,
+        "retained": 2899,
+        "excluded_positions": 11,
+        "mean_useful": 2.1391752577319587,
+        "mean_retained": 2.988659793814433
+      },
+      "5": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8845360824742268,
+        "precision": 0.7028311634635255,
+        "macro_precision": 0.7031786941580757,
+        "ndcg": 0.6554718901879515,
+        "known_positive_recall": 0.15383335671735354,
+        "recall_queries": 895,
+        "useful": 3401,
+        "retained": 4839,
+        "excluded_positions": 11,
+        "mean_useful": 3.5061855670103093,
+        "mean_retained": 4.988659793814433
+      },
+      "10": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8989690721649485,
+        "precision": 0.67180070291503,
+        "macro_precision": 0.6722631320569465,
+        "ndcg": 0.6610872123112499,
+        "known_positive_recall": 0.2802986435290336,
+        "recall_queries": 895,
+        "useful": 6499,
+        "retained": 9674,
+        "excluded_positions": 26,
+        "mean_useful": 6.7,
+        "mean_retained": 9.97319587628866
+      },
+      "20": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9103092783505154,
+        "precision": 0.61794500723589,
+        "macro_precision": 0.6184956360634603,
+        "ndcg": 0.6840908385828889,
+        "known_positive_recall": 0.4884616340614924,
+        "recall_queries": 895,
+        "useful": 11956,
+        "retained": 19348,
+        "excluded_positions": 52,
+        "mean_useful": 12.32577319587629,
+        "mean_retained": 19.94639175257732
+      },
+      "selected": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8618556701030928,
+        "precision": 0.8033770583310076,
+        "macro_precision": 0.7040325511660611,
+        "ndcg": 0.6598955189009045,
+        "known_positive_recall": 0.2079138769097556,
+        "recall_queries": 895,
+        "useful": 5757,
+        "retained": 7166,
+        "excluded_positions": 17,
+        "mean_useful": 5.935051546391753,
+        "mean_retained": 7.387628865979382
+      }
+    },
+    "vulkan": {
+      "1": {
+        "scored_queries": 965,
+        "excluded_queries": 5,
+        "hit": 0.7243523316062176,
+        "precision": 0.7243523316062176,
+        "macro_precision": 0.7243523316062176,
+        "ndcg": 0.6648902047865778,
+        "known_positive_recall": 0.03463608768446649,
+        "recall_queries": 890,
+        "useful": 699,
+        "retained": 965,
+        "excluded_positions": 0,
+        "mean_useful": 0.7243523316062176,
+        "mean_retained": 1.0
+      },
+      "3": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8525773195876288,
+        "precision": 0.7157640565712314,
+        "macro_precision": 0.7166666666666667,
+        "ndcg": 0.6577263968717306,
+        "known_positive_recall": 0.09519545071125639,
+        "recall_queries": 895,
+        "useful": 2075,
+        "retained": 2899,
+        "excluded_positions": 11,
+        "mean_useful": 2.1391752577319587,
+        "mean_retained": 2.988659793814433
+      },
+      "5": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8845360824742268,
+        "precision": 0.7023563455973543,
+        "macro_precision": 0.702766323024055,
+        "ndcg": 0.6550340124265787,
+        "known_positive_recall": 0.15374094654114448,
+        "recall_queries": 895,
+        "useful": 3398,
+        "retained": 4838,
+        "excluded_positions": 12,
+        "mean_useful": 3.5030927835051546,
+        "mean_retained": 4.9876288659793815
+      },
+      "10": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8989690721649485,
+        "precision": 0.6723514211886304,
+        "macro_precision": 0.6727589592538046,
+        "ndcg": 0.6615310405020679,
+        "known_positive_recall": 0.2805978824757964,
+        "recall_queries": 895,
+        "useful": 6505,
+        "retained": 9675,
+        "excluded_positions": 25,
+        "mean_useful": 6.706185567010309,
+        "mean_retained": 9.974226804123711
+      },
+      "20": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.9103092783505154,
+        "precision": 0.6178416373785404,
+        "macro_precision": 0.6183925432799551,
+        "ndcg": 0.6840389893720067,
+        "known_positive_recall": 0.4883516349357555,
+        "recall_queries": 895,
+        "useful": 11954,
+        "retained": 19348,
+        "excluded_positions": 52,
+        "mean_useful": 12.323711340206186,
+        "mean_retained": 19.94639175257732
+      },
+      "selected": {
+        "scored_queries": 970,
+        "excluded_queries": 0,
+        "hit": 0.8608247422680413,
+        "precision": 0.8027855153203343,
+        "macro_precision": 0.7038533836490732,
+        "ndcg": 0.6599601279630594,
+        "known_positive_recall": 0.2082001919384574,
+        "recall_queries": 895,
+        "useful": 5764,
+        "retained": 7180,
+        "excluded_positions": 17,
+        "mean_useful": 5.942268041237114,
+        "mean_retained": 7.402061855670103
+      }
+    }
+  },
+  "public": {}
+}

publication-manifest.json

@@ -1,6 +1,6 @@
 {
   "schema": "daecore.retrieval-publication-manifest",
-  "tool_sha256": "126414fafb5b9cfeb339a34060e2afadc20b00470a86b551fa09dc2097af3f40",
+  "tool_sha256": "3ea615c62866883f911899f1e131d73ca8cc9701eea3540a3a016345ac5650f8",
   "model_id": "embeddinggemma-300m-memory-ft-v2",
   "public_repo": "Daecore/embeddinggemma-300m-memory-ft-v2",
   "role": "dense-retriever",
@@ -12,7 +12,7 @@
   "export_receipt_sha256": "e24e1692f7e03c9af8040bb3f6e310a061d305f485861007f1f37d36808b20d8",
   "serving_sha256": "8c088b1635875d5406e148782b303cfc5a961138600783fb4842b413b98a98c3",
   "source_model_tree_sha256": "30a7b9f1c42b1fcee29a6729742685d5b744e8fa87d13cefab2fd2d49ea9150e",
-  "staged_at": "2026-09-29T11:56:39+00:00",
+  "staged_at": "2026-09-29T16:47:59+00:00",
   "files": {
     "added_tokens.json": {
       "sha256": "50b2f405ba56a26d4913fd772089992252d7f942123cc0a034d96424221ba946",
@@ -80,8 +80,8 @@
       "binding": "packaging record"
     },
     "evaluation/README.md": {
-      "sha256": "3750eb00fcca3ece452897c3266afe827f3bdc964f8615c11900fc8362f348d9",
-      "size": 7334,
+      "sha256": "f7059b76c4e9956e899b5f8fb855a58262db42ffef301404c874c8ee9b1c19dc",
+      "size": 8475,
       "binding": "packaging record"
     },
     "evaluation/serving-qualification.json": {
@@ -90,18 +90,33 @@
       "binding": "packaging record"
     },
     "evaluation/metrics.py": {
-      "sha256": "175cd34c626edbb502a6fb69f3a78cac8a023e5cbc478788a3a3ff7ebd9c7d5e",
-      "size": 14828,
+      "sha256": "72e7452766b491c247514df8ad1c8f52c8023e1e305d779655a9e1f3159ce52d",
+      "size": 17556,
       "binding": "packaging record"
     },
-    "evaluation/retriever.json": {
-      "sha256": "169722c802bee8a3aa240e82d1b49e909192d0e111a24a221a04fd7b8e132d57",
-      "size": 318331,
+    "evaluation/retriever-summary.json": {
+      "sha256": "6d87cfa07056cb0316e21ed70a940994d1ad91e9a297161878482814811f736b",
+      "size": 5063,
       "binding": "packaging record"
     },
-    "evaluation/retriever-summary.json": {
-      "sha256": "b3c61ebdb624ced6a0af65eef60fcd22a34d316a7bd5ab2fc17d487d3f45dcfc",
-      "size": 2271,
+    "evaluation/dimensions-summary.json": {
+      "sha256": "d42adc93268a44675bcda3321a431f8cd92e6c9178efdb44e0314372468a7c6e",
+      "size": 1253,
+      "binding": "packaging record"
+    },
+    "evaluation/promotion-summary.json": {
+      "sha256": "c12e58b9ec5f5dd76baa2dc36a602816c9895f2b57e39b95cb74ca167506626a",
+      "size": 6886,
+      "binding": "packaging record"
+    },
+    "evaluation/serving-summary.json": {
+      "sha256": "fc60b4e94fe01e0470dcced382046c4c30c329a87b5caa2b631b369015548198",
+      "size": 6029,
+      "binding": "packaging record"
+    },
+    "evaluation/retriever.json": {
+      "sha256": "fcfbe8155cd30319ee38745f92376a45635ecaed08b86962ee8604680ce910b7",
+      "size": 573086,
       "binding": "packaging record"
     },
     "evaluation/dimensions.json": {
@@ -109,34 +124,19 @@
       "size": 762884,
       "binding": "packaging record"
     },
-    "evaluation/dimensions-summary.json": {
-      "sha256": "d42adc93268a44675bcda3321a431f8cd92e6c9178efdb44e0314372468a7c6e",
-      "size": 1253,
-      "binding": "packaging record"
-    },
     "evaluation/promotion.json": {
       "sha256": "dca79b36c99e2f90c05bff12227634fe175f1370a2ac83368da084ccf7aeffc3",
       "size": 816500,
       "binding": "packaging record"
     },
-    "evaluation/promotion-summary.json": {
-      "sha256": "3dcd954b778e8955f90c7e160790227602b701eea0897d539ecc1f8ab265567a",
-      "size": 3724,
-      "binding": "packaging record"
-    },
     "evaluation/serving.json": {
       "sha256": "0c017dbd0e66734c465d617eef24b2affb2670a96f7fd5b2c001514c3cc1381a",
       "size": 500118,
       "binding": "packaging record"
     },
-    "evaluation/serving-summary.json": {
-      "sha256": "3a7c6a2c96495917e65a7dabde9c02c7c457b5416df51cbb58bb7267d5c74fda",
-      "size": 3162,
-      "binding": "packaging record"
-    },
     "evaluation/figures.py": {
-      "sha256": "7cebbf3111664de8f08c88fae1b9b40e9212549b30bcb28f45d1459a4dc50e53",
-      "size": 5805,
+      "sha256": "53819eb947c3c5c25f6331b0e384e6e27f7b5da750c84f8bdf6fd6e7790c9d94",
+      "size": 5378,
       "binding": "packaging record"
     },
     "evaluation/svg_figures.py": {
@@ -145,15 +145,15 @@
       "binding": "packaging record"
     },
     "figures/gemma-comparison.svg": {
-      "sha256": "4e9fa6536f3a08fe7cfff16d48f7218fff065ee9b77967b2a42ed5d5216895ab",
-      "size": 9252,
+      "sha256": "f4bf74ea9e2e044e124893d5ca8bd028c6f2f79ef53babfb7ee718f8b6786be2",
+      "size": 13377,
       "binding": "packaging record"
     },
     "README.md": {
-      "sha256": "d321faf5d4e422c051b71cdadec912f6aea9e646f826b75b5303b35be38e0352",
-      "size": 13845,
+      "sha256": "e7d88fad6c2c12613c8e8a8108a38dc7fbb181c005ffadac172f1120f84c95f0",
+      "size": 14331,
       "binding": "model card with upload-relative links",
-      "source_sha256": "8b3f1168f3bfcf7ee826167a2ca87fd33de5e2ed3d1477b1b32430cff033aa51"
+      "source_sha256": "5ea17ec12b2673539eba89fc6e3f85c9ed8ba7d879d9ce94dbeebcb1392a8afa"
     }
   }
 }