Publish ettin-150m-memory-reranker-ft-v1 (qualified ONNX export and complete attribution)

MODIFICATIONS.md

@@ -16,21 +16,24 @@ What changed:
   not human annotations.
 - **Export.** The merged model was exported to ONNX graphs taking
   `input_ids` and `attention_mask` and emitting one score per query-passage
-  pair. CUDA uses FP16. DirectML widens the same stored weights and floating
-  calculations to FP32, and uses compatible inferred reshapes. Both graphs
+  pair. CUDA uses FP16. Vulkan widens the same stored weights and floating
+  calculations to FP32, and uses compatible inferred reshapes. Bounded
+  integer/boolean attention-mask calculations are expressed exactly in FP32
+  before restoring their original output types. Both graphs
   share one external parameter file; the CUDA graph and weights are unchanged.
-- **Serving contracts.** `serving.json` and `serving.directml.json` use the
+- **Serving contracts.** `serving.json` and `serving.vulkan.json` use the
   same descriptor format for each graph: 1,153 total pair tokens and batch
-  ceilings of 16 on CUDA and 8 on DirectML. CPU handles permitted control
+  ceilings of 16 on CUDA and 8 on Vulkan. CPU handles permitted control
   operations, not a fallback execution of the whole reranker.
 
 Files modified or generated by Daecore:
 
 - `model.onnx`: the generated ONNX graph of the fine-tuned reranker;
-- `model.directml.onnx`: the FP32 calculation graph for DirectML;
+- `model.vulkan.onnx`: the derived FP32 calculation graph for Vulkan;
 - `model.onnx.data`: the fine-tuned parameters shared by both graphs;
 - `config.json`: the export configuration of the modified reranker;
-- `serving.json` and `serving.directml.json`: the runtime and sequence contracts.
+- `serving.json` and `serving.vulkan.json`: the runtime and sequence contracts;
+- `vulkan-derivation.json`: exact source and derived graph/weight identities.
 
 Unchanged: `tokenizer.json` and `tokenizer_config.json` preserve the upstream
 tokenizer contract.

README.md

@@ -1,219 +1,122 @@
----
-license: apache-2.0
-base_model: cross-encoder/ettin-reranker-150m-v1
-pipeline_tag: text-ranking
-tags:
-  - ettin
-  - modernbert
-  - cross-encoder
-  - reranker
-  - fine-tuned
-  - onnx
----
-
-# Ettin-150M memory reranker v1
-
-## Summary
-
-This model reorders the fixed 50-passage pool that a hybrid first stage
-produces for a memory query. It is a fine-tune of
-[`cross-encoder/ettin-reranker-150m-v1`](https://huggingface.co/cross-encoder/ettin-reranker-150m-v1)
-trained with a graded listwise objective on 144,100 unique query-passage pairs
-across 2,882 queries.
-
-On the 441 development queries whose fixed pool holds at least one useful
-passage, it raised nDCG@10 from 0.562 for the fused first stage to 0.762 and
-Hit@1 from 0.644 to 0.807. In the protected promotion read, on the same 865
-answerable pools, it scored nDCG@10 0.728 against 0.595 for the previous
-reranker, +0.133 with a one-sided 95% lower bound of +0.119.
-
-All relevance grades are frontier-model judgments under a frozen protocol,
-the corpus is mostly generated documents, and the per-query evidence behind
-the development and gate reads was lost after the aggregates were recorded.
-The limitations section spells out each.
-
-## Model details
-
-| Item | Value |
-|---|---|
-| Architecture | Ettin-150M (ModernBERT-family) cross-encoder with a trained scoring head; rank-16 LoRA+ adapter merged into the weights |
-| Input | one query and one passage, 1,153 tokens total |
-| Pool | a fixed 50-passage candidate set; the model reorders it and never adds to it |
-| Output | one relevance score per pair, used for ordering |
-| Serving precision | FP16 on CUDA; FP32 calculations on DirectML, sharing the same stored weights |
-| Deployment state | CUDA release plus an approved DirectML graph; installation is a separate update |
-
-## Intended use
-
-Final ordering inside a local memory-retrieval pipeline: the dense retriever
-and BM25 each retrieve, reciprocal-rank fusion merges them, and this model
-scores the top 50. It cannot recover a passage the first stage missed; on the
-full 553-query development panel the 112 queries with no useful passage in
-their pool stay at zero before and after reranking.
-
-## Training data
-
-| Item | Value |
-|---|---|
-| Queries | 2,882 answerable, plus 30 explicit no-positive rows kept out of the ranking loss |
-| Unique pairs | 144,100, every one seen; 184,448 list presentations |
-| Unique passages | 35,006 |
-| Grade counts 0 / 1 / 2 / 3 | 32,331 / 37,258 / 43,731 / 30,780 |
-| Query mix | 1,318 anchor-conditioned, 1,270 unanchored, 294 lexical-style |
-| Protected queries or families in gradients | 0 |
-
-Grades come from final consensus ledgers, so every grade-1 label is usable
-supervision rather than a masked unknown. The lexical-style queries (terse,
-exact, pasted, or lightly misspelled) are there so the model learns the
-lexical lane's cases as well as rich semantic questions.
-
-## Recipe
-
-Grade-gain ListNet over lists of 16 with an adjacent-grade RankNet auxiliary
-at weight 0.2; LoRA+ at rank 16 and alpha 32 on all linear layers, learning
-rate 5e-5 for the A matrices and the score head and 8e-4 for the B matrices;
-2,882 optimizer updates; FP16. Merge and reload changed scores by 0.
-
-## Evaluation
-
-**Development panel.** 553 queries with every one of the 27,650 pool
-passages graded; 441 have a useful passage in their pool.
-
-| Stage | nDCG@10 | Hit@1 | Hit@5 | Hit@10 | Hit@20 | Hit@50 |
-|---|---:|---:|---:|---:|---:|---:|
-| fused first stage, 441 answerable | 0.5617 | 0.6440 | 0.8662 | 0.9206 | 0.9569 | 1.0000 |
-| this model, 441 answerable | 0.7621 | 0.8073 | 0.9410 | 0.9615 | 0.9841 | 1.0000 |
-| fused first stage, all 553 | 0.4479 | 0.5136 | 0.6908 | 0.7342 | 0.7631 | 0.7975 |
-| this model, all 553 | 0.6078 | 0.6438 | 0.7505 | 0.7667 | 0.7848 | 0.7975 |
-
-Pairwise accuracy on the answerable pools is 0.820. The all-query rows are the
-honest end-to-end view; Hit@50 is unchanged by design.
-
-**How it was chosen.** Against the consensus hard-label baseline (0.680
-nDCG@10 on this panel), a Lambda-style objective lost, ListNet at width 8 tied,
-and width 16 without a higher learning rate lost. The adjacent-grade term at
-5e-5 reached 0.715. Restarting from the base over every paid pair gave 0.728
-with pure ListNet, 0.731 with the auxiliary and standard LoRA, and 0.732 with
-DoRA. LoRA+ at rank 16 reached 0.762 and was the only large optimization gain;
-rank 32 tied it (+0.0006) and a further top-10 continuation lost 0.005.
-
-**Protected gate, same-pool comparison.** On the 865 sealed queries whose
-candidate pool holds a useful passage, this model scored nDCG@10 0.7280
-against 0.5952 for the previous reranker, +0.1328 with a one-sided 95% lower
-bound of +0.1191 against a registered 0.02 margin. The judge-noise band for
-this comparison was 0.0036. The full pipeline reached Hit@1 0.7609 and nDCG@10
-0.6692 on the 895 union-answerable queries, against 0.6078 and 0.4988 before.
-
-## Runtime
-
-The FP16 ONNX export removes CPU-hosted sequence operations from all 22 MLP
-layers so the whole graph runs on the GPU. It is bounded-quality equivalent
-rather than bit-exact to the Torch model: as recorded at qualification,
-+0.0002 nDCG@10 and +0.0023 Hit@1 on the development pools, 4 of 441 top
-choices and 15 of 441 top-10 memberships changed, and a 95th-percentile score
-drift of 0.0098. A top-50 pool reranks in 0.664 s median and 1.032 s at the
-95th percentile on an RTX 3060 Ti; the one-second target is advisory.
-
-The export regenerated on 2026-09-04 reproduced the qualified graph byte for
-byte. A parity recheck the same day had to use the training partition (2,882
-queries, 144,100 pairs) because the qualification pools were lost, and it
-fails the frozen policy on two checks: Hit@1 moved by −0.0003, one query,
-and one passage moved 16 places against a limit of 4. All 10 top-1 changes
-and all 73 top-10 membership changes on that panel occur between passages
-whose FP16 scores are exactly tied in one run and one unit apart in the
-other, so the tie-break decides the order; the largest score change, 0.08,
-sits at rank 47. On that panel the export reranked a pool in 0.584 s median
-and 0.912 s at the 95th percentile, and the Torch FP16 model in 0.532 s and
-0.777 s.
-
-### DirectML comparison
-
-The DirectML package adds `model.directml.onnx` alongside the unchanged CUDA
-`model.onnx`. Both read the same `model.onnx.data`; no second parameter file
-or training framework is needed at runtime. DirectML uses full-precision
-calculations because the half-precision graph had larger score differences.
-CUDA keeps its faster half-precision graph. Each provider plans batches from
-available memory and pair length, with measured ceilings of 16 rows for CUDA
-and 8 for DirectML.
-
-On a current 970-query replay (48,500 fixed-pool pairs), DirectML completed
-without a failure: 2.231 seconds median, 3.247 seconds at the 95th percentile,
-and 4.286 seconds maximum on an RTX 3060 Ti. No call exceeded ten seconds.
-This panel retains its per-query evidence; 2,498 pairs are unjudged, and their
-grades remain unknown. It is separate from the lost historical gate evidence.
-
-Against the published CUDA graph, nDCG@10 changed by +0.000017. Eight first
-choices and 36 top-ten sets changed. Both versions returned known-useful
-evidence for 816 of 970 queries; DirectML returned 5,528 known-useful passages
-versus 5,524. The unchanged calibration was applied for comparison only; its
-DirectML registration is not implied by these results.
-
-Two strict cross-precision checks fail and were explicitly accepted:
-Hit@1 and Hit@5 each lose one query, and one passage moves five positions
-against a limit of four. The affected useful passages remain in the returned
-sets; the five-position move is rank 33 to 28, outside the returned three.
-On the 482 fully judged pools, no Hit metric decreases.
-
-Running the same FP32 graph on CUDA for all 165 queries with changed
-deliveries or relevant ranking differences isolates the provider error:
-8,250 pairs have a 95th-percentile score difference of 0.00000954 and a
-maximum of 0.000149. One pair exceeds the diagnostic 0.0001 bound; one full
-ordering changes and no first choice changes. The two Hit losses and the
-five-position move also occur on CUDA FP32, identifying precision rather
-than DirectML as their cause.
-
-On the supported sequential build policy, concurrent embedding and recall
-kept reranking below 3.2 seconds in three rounds. Eight queued recalls across
-four threads completed within 9.8 seconds. Forcing all three GPU models to
-remain resident on this 8 GB card exceeded the memory budget and failed the
-ten-second limit; those forced-overload timings are not a passing result.
-Worker-kill recovery completed for reranking, embedding, and classification.
-
-## Limitations, ranked
-
-1. **Model-judged relevance.** No human grades; session agreement 0.858 and
-   kappa 0.682 on a 419-query audit. Scores measure agreement with that
-   instrument.
-2. **Generated corpus.** Mostly fictional document worlds chosen for breadth;
-   behavior on other real corpora is unmeasured.
-3. **Lost per-query evidence.** The development ledger, gate judgments, pool
-   scores, and gate report were destroyed in a local storage incident after
-   these aggregates were recorded. The numbers stand as recorded and cannot be
-   re-audited; the recipe-search arms survive only as recorded results.
-4. **One training run per arm.** Several selection margins (rank 16 versus 32,
-   width 8 versus 16) are inside a plausible seed effect; compute set that limit.
-5. **First-stage ceiling.** 112 of 553 development queries and 95 of 990 gate
-   queries have no useful passage in any top-50 pool; ranks 51 to 200 were
-   never judged.
-6. **Provider limits.** CPU reranking is not qualified. DirectML has been
-   measured on an NVIDIA RTX 3060 Ti, not AMD or Intel hardware, and its
-   cross-precision policy exceptions above were accepted with disclosure.
-   Hosts without a qualified reranker serve fusion order with a typed notice.
-7. **The reproducible parity read fails the frozen policy.** The recorded
-   development-panel pass cannot be re-run, and the 2026-09-04 recheck on the
-   training partition fails two checks at exact FP16 ties. The operator
-   accepted the recorded qualification with this disclosure rather than
-   exporting an FP32 graph or amending the policy. The current FP32 provider
-   comparison is separate evidence and does not erase that historical result.
-
-## Reproducibility
-
-- Model tree SHA-256 `bd8028b5dbc6ab2cd2ab9a6019346de7b25536f50022617194bcb8503d4467ad`;
-  base revision `025501c4e0f9bbeb4c5b198318e0089ff061cc14`, tree
-  `e0edfb38cc495f23507de50233d0965f4e72ad0872e70e6f4e8fc889d394ea01`.
-- Training data manifest `286318992c584ae790a72a5a86b0e7d03d8860f8ae93c35f6cead4b08b4d395a`,
-  schedule `e98d3cec2533b6edcded7c0b5017cba432e4af0e73fd68fca84997f934fa8f63`.
-- Recipe id `final-listnet-adjacent-loraplus16-full-coverage-w16-lr5e5`;
-  registry manifest `../model_registry/ettin-150m-memory-reranker-ft-v1.json`;
-  experiment log `../history/README.md`.
-- CUDA graph SHA-256 `7df4a3b3cd89d216c016abd7d169f0fd215e9f79c06a2ffed57bac1abf556073`.
-- DirectML graph SHA-256 `05abdc03628b599712a993381919aef1e22fcc5ad6384608abc2e069c402e369`.
-- Shared weights SHA-256 `78ddab681c2f506d62bbcd475fec5c10c62d867ed60d97a0170828efaff5bdde`.
-- CUDA serving-contract SHA-256 `ce05a6a810007372c6de0f4c56dc94f652e1d28f2eeddf1878b64a170b4c48c2`.
-- Build stack: Transformers 5.2.0, Sentence Transformers 5.5.1, PyTorch
-  2.10.0; export opset 18.
-
-## License
-
-Apache 2.0, following the upstream model; the distribution includes the
-license, a notice, and a modification disclosure.
+---
+license: apache-2.0
+base_model: cross-encoder/ettin-reranker-150m-v1
+pipeline_tag: text-ranking
+language:
+  - en
+tags:
+  - ettin
+  - modernbert
+  - cross-encoder
+  - reranker
+  - fine-tuned
+  - onnx
+---
+
+# Ettin-150M memory reranker
+
+An English cross-encoder for ranking passages from notes, procedures, decisions and technical documentation. Give it a query and candidate passages; it returns one relevance score per pair. It is a fine-tune of [Ettin Reranker 150M](https://huggingface.co/cross-encoder/ettin-reranker-150m-v1) and runs locally through ONNX Runtime.
+
+This revision adds a Vulkan FP32 graph, `model.vulkan.onnx`; the learned weights and the CUDA FP16 graph are unchanged. It omits the DirectML graph, which remains in the previous revision.
+
+## Quick start
+
+Install `tokenizers`, `numpy` and `onnxruntime-gpu[cuda,cudnn]==1.24.4`. Keep `model.onnx`, `model.onnx.data` and the tokenizer from the same package together. This CUDA example assumes that package is in `downloaded-model` and scores two passages:
+
+```python
+import numpy as np
+import onnxruntime as ort
+from pathlib import Path
+from tokenizers import Tokenizer
+
+root = Path("downloaded-model")  # The exact downloaded serving package.
+tokenizer = Tokenizer.from_file(f"{root}/tokenizer.json")
+tokenizer.no_padding()
+tokenizer.no_truncation()
+ort.preload_dlls()
+session = ort.InferenceSession(
+    f"{root}/model.onnx", providers=[("CUDAExecutionProvider", {"use_tf32": "0"})]
+)
+if "CUDAExecutionProvider" not in session.get_providers():
+    raise RuntimeError("CUDA did not initialize; check the ONNX Runtime CUDA dependencies.")
+query = "What must happen before a database migration?"
+passages = [
+    "Take a verified backup before applying the migration.",
+    "The dashboard theme uses a blue background.",
+]
+query_tokens = tokenizer.encode(query, add_special_tokens=False)
+if len(query_tokens.ids) > 128:
+    raise ValueError("Query exceeds the qualified 128-token content limit.")
+scores = []
+for passage in passages:
+    passage_tokens = tokenizer.encode(passage, add_special_tokens=False)
+    passage_tokens.truncate(1022)
+    pair = tokenizer.post_process(query_tokens, passage_tokens, add_special_tokens=True)
+    inputs = {
+        "input_ids": np.array([pair.ids], dtype=np.int64),
+        "attention_mask": np.array([pair.attention_mask], dtype=np.int64),
+    }
+    scores.append(float(session.run(None, inputs)[0].reshape(-1)[0]))
+print(sorted(zip(passages, scores), key=lambda item: item[1], reverse=True))
+```
+
+Higher scores rank first; raw scores are not probabilities. The input envelope is 128 query-content tokens plus 1,022 passage-content tokens and three special tokens, a maximum of **1,153 tokens**. The example rejects an oversized query and truncates the passage, matching the tested input policy.
+
+## Evaluation
+
+Upstream and fine-tuned Ettin scored the same **970 pools of 50 candidates**, with all 48,500 pairs judged. The table uses the **873 answerable pools** that contain at least one useful passage, so it measures ordering after retrieval. “Upstream” is the exact starting checkpoint without Daecore fine-tuning.
+
+| Metric, answerable pools | Upstream Ettin | Daecore fine-tune |
+|---|---:|---:|
+| Hit@3 | 83.16% | 92.55% |
+| Precision@3 | 59.76% | 80.03% |
+| nDCG@10 | 0.5519 | 0.7407 |
+
+![Ettin ordering quality across return depths](figures/ettin-comparison.svg)
+
+Grades 2 and 3 count as useful. Hit asks whether at least one useful passage appears, precision measures the useful share of returned positions, and nDCG rewards higher grades near the top. Many pools contain several useful passages, so random ordering already reaches 97.30% Hit@20; early precision and graded ranking are more informative than deep Hit. The comparison runs both models in PyTorch FP16 with matched tokenization on reused development inputs. The [evaluation companion](evaluation/README.md) has every cutoff through 20, all-query results, model identities and metric code.
+
+On public data, both models rerank the same 50 candidates that upstream Gemma retrieved for each query, scored with official relevance judgments:
+
+| Dataset | Queries | Upstream Ettin | Daecore fine-tune |
+|---|---:|---:|---:|
+| FiQA | 648 | 0.4860 | 0.4527 |
+| SciFact | 300 | 0.7487 | 0.7544 |
+
+These are nDCG@10 scores for reranking fixed candidate pools, not dense Gemma scores. Fine-tuning lowered FiQA and slightly raised SciFact. Later continuation recipes did not recover FiQA while keeping Daecore ranking quality, so the original learned weights remain.
+
+## Training data and objective
+
+Training passages come from generated organizational documents and one person's project documentation, much of it AI-written. Queries include full questions, keywords, identifiers and pasted fragments. Model judges assigned grades from 0 to 3, and several passages can be useful for one query. ListNet learns ordering within each candidate list, and a pairwise term emphasizes distinctions between neighboring grades. Lists rotate through the full labeled pools, so useful, partly useful and unhelpful passages stay in the comparison.
+
+<details>
+<summary>Training recipe and exposure</summary>
+
+| Item | Value |
+|---|---|
+| Starting model | `cross-encoder/ettin-reranker-150m-v1` |
+| Training queries | 2,882 with ranking supervision; 30 no-positive queries excluded from the ranking loss |
+| Unique query–passage pairs | 144,100 |
+| Unique passage texts | 35,006 |
+| Supervision | Four relevance grades, 0–3; grades 2–3 count as useful |
+| Objective | ListNet with an adjacent-grade pairwise term weighted 0.2 |
+| Adapter | LoRA+, rank 16, alpha 32; merged for serving |
+| Learning rates | 5e-5 for LoRA A and the scoring head; 8e-4 for LoRA B |
+| Training steps | 2,882 scheduled; 2,881 applied updates, accumulating four lists of 16 per step |
+| Exposure | 11,528 lists; 184,448 candidate presentations |
+
+</details>
+
+## Runtime and limits
+
+`model.onnx` is the CUDA FP16 graph and `model.vulkan.onnx` the Vulkan FP32 graph; both read `model.onnx.data`. The Vulkan derivation rewrites bounded mask operations and keeps the learned weights. The FP32 graph's results were also checked for correctness on the CPU execution provider; no CPU latency guidance is given.
+
+Vulkan requires `onnxruntime==1.24.4` and the native WebGPU plugin [`onnxruntime-ep-webgpu==0.4.0`](https://pypi.org/project/onnxruntime-ep-webgpu/), registered explicitly. Add the plugin device to the session options with `dawnBackendType=Vulkan`, and create the session without a providers list; the plugin name alone does not select Vulkan. `serving.vulkan.json` records the tested provider options.
+
+On 970 Daecore candidate pools, CUDA and Vulkan produced the same Hit@5, Hit@10 and Hit@20; Hit@3 differed by one query and nDCG@10 by 0.0004. The paths are not bit-exact, so close scores can swap order. Checks used Windows x64 with an NVIDIA RTX 3060 Ti (8 GB); other GPUs and Linux are untested, even where Vulkan is available. Inside Daecore, Ettin reorders up to 50 deduplicated BM25 and Gemma candidates; the [retrieval pipeline overview](https://huggingface.co/Daecore) describes that path and result selection.
+
+The task-specific panel is mostly generated and model-labeled, with no human reference, and was reused during development; training-seed variation is unmeasured. A reranker cannot recover a passage missing from its candidates, and several relevant passages may repeat one fact. These results do not establish unique-evidence coverage or downstream agent success.
+
+## License
+
+Apache-2.0. The package includes upstream attribution and a modification notice.

evaluation/README.md

@@ -0,0 +1,83 @@
+# Model evaluation records
+
+These files let readers recompute the comparisons in the Daecore model cards without model weights. Task-specific records contain anonymized row and group IDs, labels, and scores or ranked grades; none contains query or document text, private source identifiers or workspace paths. Model identities are recorded inside each file.
+
+## Recompute
+
+Python 3.11 or later is enough; no packages or network access are needed. From the directory that holds the records:
+
+```sh
+python metrics.py --directory .
+```
+
+The command summarizes every record present and stops with an error if there is none. Each entry in its output equals the matching `*-summary.json`. The classifier and Ettin packages also include `figures.py`; `python figures.py --output <directory>` redraws their card chart. Gemma uses a table. Each model package contains only its own records and summaries; the source repository holds all four record types. Fractions are stored at full precision; cards round percentages to two decimals and ranking metrics to four.
+
+The records support metric recomputation. Re-running inference would need the private text and corpus, which are not included. They come from task-specific development evaluations, not a new untouched test.
+
+## Records
+
+| Record | Inputs and denominator | Comparison |
+|---|---|---|
+| `classifier.json` | 1,300 generated passages from three held-out source families; resolved labels only, per facet | Upstream GLiClass, a trained word-feature baseline and the fine-tune |
+| `reranker.json` | 970 fixed pools of 50 passages, all 48,500 pairs judged; 873 pools have useful evidence | Upstream Ettin and the fine-tune, plus the expected result of random ordering |
+| `promotion.json` | 970 hybrid-search queries over 82,719 passage texts, with reviewed labels and declared exclusions; five public dense-retrieval panels | Previous and updated Gemma with unchanged Ettin; the public panels also include upstream Gemma |
+| `serving.json` | The same 970 queries with updated-Gemma candidate pools; reference extended by 32 grades | Ettin through CUDA FP16 and Vulkan FP32, with identical weights and score mapping |
+
+`serving-qualification.json` summarizes provider and recovery checks by receipt hash and keeps the aggregate FiQA and SciFact results for upstream and fine-tuned Ettin.
+
+“Upstream” means no Daecore fine-tuning, not an untrained network; upstream Ettin is already a trained reranker. Interim training checkpoints are not included.
+
+## How each comparison was run
+
+**Classifier.** Evaluation families were excluded from training. Upstream uses the same five label definitions and 768-token input limit as the fine-tune. Its raw logits and the fine-tune's calibrated probabilities are used only to rank within each facet; their scales are not compared. All 1,300 saved predictions matched fresh CPU inference of the released graph within 5.62e-6. The baseline uses word unigram and bigram TF-IDF with one balanced logistic-regression model per facet, fitted on the same 59,886 training passages; it sees full passage text, while the neural models apply their 768-token limit.
+
+**Ettin, fixed pools.** Candidates and grades are held fixed. Both models run in PyTorch FP16 with the same 1,153-token pair construction, Transformers 5.2.0 and Sentence Transformers 5.5.1; upstream's metadata names a newer library version, but both use the same reference runtime here. The fine-tune's saved scores were checked against fresh inference on three complete pools. This evaluates the trained models, not ONNX serving or latency. The public FiQA and SciFact results rerank fixed 50-candidate pools from upstream Gemma.
+
+**Gemma update.** Only Gemma changes; BM25, fusion and Ettin's weights stay fixed. All 970 queries are retained, and seven reviewed grade corrections apply to both models. Within each original top-k or selected prefix, 65 unresolved query–passage abstentions are excluded without backfilling: `ranked_grades` keeps their positions as `null`, with the required `excluded` flag. Precision is pooled over retained positions; nDCG reindexes them and takes its ideal ordering at the retained depth from `reference_grade_counts`. Selected depths refer to the original score-selected prefixes.
+
+| Metric, 970 queries | Previous Gemma | Updated Gemma |
+|---|---:|---:|
+| Hit@3 | 83.92% | 85.36% |
+| Hit@5 | 85.88% | 88.45% |
+| Hit@10 | 87.84% | 89.90% |
+| Hit@20 | 90.10% | 91.03% |
+| nDCG@10 | 0.6764 | 0.6611 |
+| Selected-prefix precision | 80.41% | 80.34% |
+
+The same record holds per-query dense nDCG@10 for the five public panels. The previous fine-tune was measured only on FiQA (0.4009) and SciFact (0.7679); missing baselines stay absent. These sets supplied no training examples but informed development, and recomputing their means is different from rerunning retrieval on the public corpora.
+
+**Ettin providers.** Updated-Gemma candidate pools for the same 970 queries were scored through CUDA FP16 and Vulkan FP32 with unchanged weights and score mapping. The two paths' top-20 results included 32 query–passage pairs without a grade. Of these, 21 reuse grades from the hard-contrast Ettin evaluation. The other 11 received two independent GPT-6 Sol judgments plus a resolution step and were then reviewed against the full passage text by GPT-6 Astra; no person reviewed them. No existing grade changed, and the 65 abstentions remain excluded. The added grades slightly change nDCG's ideal ordering, so the promotion record keeps its original reference.
+
+| Metric, 970 queries | CUDA FP16 | Vulkan FP32 |
+|---|---:|---:|
+| Hit@3 | 85.36% | 85.26% |
+| Hit@5 | 88.45% | 88.45% |
+| Hit@10 | 89.90% | 89.90% |
+| Hit@20 | 91.03% | 91.03% |
+| nDCG@10 | 0.6611 | 0.6615 |
+| Selected-prefix precision | 80.34% | 80.28% |
+
+Small numerical differences between the paths can reorder close scores. On 20 matched pools replayed twice, second-pass reranking took 1.56 s median and 2.11 s at the 95th percentile with Vulkan, against 1.92 s and 2.70 s with the previous package's DirectML graph. All provider measurements come from one Windows x64 machine with an NVIDIA RTX 3060 Ti (8 GB) and do not transfer to other GPUs or platforms.
+
+## Metric definitions
+
+Relevance grades are 0–3, and grades **2 and 3** count as useful. An “answerable” query has at least one useful labeled passage in the specified pool; the absence of a useful judgment does not prove that no answer exists.
+
+- **Hit@k:** fraction of queries with at least one useful passage in the first k positions.
+- **Precision@k:** useful passages among the first k. Fixed-pool tables average it over queries; the promotion and serving records pool it over retained positions. Selected-prefix precision applies the same pooling to all returned passages.
+- **Recall@k:** useful passages retrieved divided by the query's known useful passages, averaged over queries. It counts labeled passages, not every fact an answer needs.
+- **nDCG@k:** gain `2**grade - 1`, discount `1/log2(rank + 1)`, divided by the ideal ordering at the same cutoff. Grade 1 adds a small gain although it is not useful for Hit, precision or recall. Each cutoff has its own ideal, so values need not change monotonically with k.
+- **Average precision (AP):** area under the stepwise precision–recall curve, with tied scores grouped at one threshold; macro AP weights the five facets equally.
+- **Precision at 90% or 95% recall:** the best measured precision at any threshold reaching that recall, taken from the evaluation curve rather than a threshold chosen in advance.
+
+Ties keep the original candidate order. Classifier fields marked `null` are excluded identically for every model. Conditional reranker nDCG and recall use the 873 answerable pools; all-query Hit and precision use all 970. The random-order reference is exact within each pool: with N candidates and R useful passages, expected precision is R/N and Hit@k is `1 - C(N-R, k)/C(N, k)`. Random ordering already reaches 97.30% Hit@20 on the answerable reranker pools.
+
+## Limits
+
+Task-specific labels are language-model judgments without a human-adjudicated reference. Much of the source material is generated. Project-document sources are narrow and often AI-written; they do not establish generalization across users. The panels were reused during development, and each released model was trained once. Several relevant passages can repeat one fact, so passage-level scores do not measure unique-fact coverage, answer completeness or downstream agent success.
+
+## Cards
+
+- [Five-facet classifier](https://huggingface.co/Daecore/gliclass-std-base-v3-5facet-qint8-v2)
+- [EmbeddingGemma retriever](https://huggingface.co/Daecore/embeddinggemma-300m-memory-ft-v1)
+- [Ettin reranker](https://huggingface.co/Daecore/ettin-150m-memory-reranker-ft-v1)

evaluation/figures.py

@@ -0,0 +1,89 @@
+"""Regenerate the classifier and reranker SVGs from the public labels and scores.
+
+Run with Python 3.11+: python figures.py --output ../figures
+The repository shares its renderer from eval/lib; HF packages include that
+same renderer next to this script.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import sys
+from pathlib import Path
+
+import metrics
+
+HERE = Path(__file__).resolve().parent
+renderer = HERE / 'svg_figures.py'
+if not renderer.is_file():
+    renderer = HERE.parents[1] / 'lib/svg_figures.py'
+spec = importlib.util.spec_from_file_location('model_card_svg', renderer)
+svg = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = svg
+spec.loader.exec_module(svg)
+
+UPSTREAM, FIT, REFERENCE = '#0072B2', '#D55E00', '#777777'
+
+
+def classifier(data: dict) -> str:
+    summary = metrics.summarize_classifier(data)
+    facets = [*metrics.FACETS, 'macro']
+    def values(model):
+        result = summary['models'][model]
+        return [result['facets'][f]['average_precision'] for f in metrics.FACETS] + [result['macro']['average_precision']]
+    return svg.render_bars(
+        facets, [('Upstream', values('upstream'), UPSTREAM),
+                 ('TF-IDF + logistic regression', values('linear'), REFERENCE),
+                 ('Daecore fine-tune', values('finetuned'), FIT)],
+        title='Classifier: matched before and after fine-tuning',
+        subtitle='1,300 generated passages · 3 held-out families · resolved labels only',
+        ylabel='average precision', ylim=(0, 1.05), separator_before=5,
+        width=840, height=360,
+        notes=['Model-generated labels; these results do not establish accuracy on unrelated real documents.'],
+    )
+
+
+def reranker(data: dict) -> str:
+    summary = metrics.summarize_reranker(data)
+    definitions = [('hit', 'At least one useful passage'), ('precision', 'Useful passages / returned passages'), ('ndcg', 'Graded ranking quality')]
+    panels = []
+    legend = [('Upstream', UPSTREAM, None), ('Daecore fine-tune', FIT, None), ('Random pool order', REFERENCE, '5 3')]
+    for field, title in definitions:
+        series = []
+        for model, (label, color, dash) in zip(('upstream', 'finetuned', 'random'), legend, strict=True):
+            values = summary['models'][model]['answerable']
+            series.append(svg.Series(label, list(range(5)), [values[str(k)][field] for k in metrics.CUTOFFS], color, dash=dash))
+        panels.append(svg.Panel(title, series, xlabel='rank cutoff',
+                                ylabel={'hit': 'Hit@k', 'precision': 'Precision@k', 'ndcg': 'nDCG@k'}[field],
+                                xlim=(0, 4), ylim=(0, 1.02), xticks=list(enumerate(map(str, metrics.CUTOFFS)))))
+    return svg.render_grid(panels, columns=3, title='Ettin: identical candidates, different ordering',
+                           subtitle='873 answerable pools of 50 · 970 queries in total · useful = grade 2 or 3',
+                           legend=legend, panel_width=300, panel_height=270)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--directory', type=Path, default=HERE)
+    parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--model', choices=('classifier', 'reranker'))
+    args = parser.parse_args()
+    args.output.mkdir(parents=True, exist_ok=True)
+    definitions = {'classifier': ('classifier', 'classifier-comparison.svg'),
+                   'reranker': ('reranker', 'ettin-comparison.svg')}
+    names = [args.model] if args.model else [
+        name for name, (record, _) in definitions.items()
+        if (args.directory / f'{record}.json').is_file()
+    ]
+    if not names:
+        parser.error('No model comparison records found in the selected directory')
+    for name in names:
+        record, filename = definitions[name]
+        data = json.loads((args.directory / f'{record}.json').read_text(encoding='utf-8'))
+        (args.output / filename).write_text(globals()[name](data), encoding='utf-8', newline='\n')
+
+
+
+if __name__ == '__main__':
+    main()

evaluation/metrics.py

@@ -0,0 +1,219 @@
+"""Recompute the published model comparisons from text-free evaluation records.
+
+Python 3.11+, standard library only. Run: python metrics.py --directory .
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+from pathlib import Path
+from statistics import mean
+
+CUTOFFS = (1, 3, 5, 10, 20)
+FACETS = ("trap", "decision", "constraint", "mechanism", "procedure")
+
+
+def classification_metrics(labels: list[int], scores: list[float]) -> dict[str, float]:
+    """Threshold-grouped AP and best precision at or above each recall target."""
+    if len(labels) != len(scores) or not labels or set(labels) - {0, 1}:
+        raise ValueError("Classification labels and scores must be aligned binary rows")
+    if not all(math.isfinite(s) for s in scores) or sum(labels) == 0:
+        raise ValueError("Classification scores must be finite with positive support")
+    order = sorted(range(len(labels)), key=lambda i: -scores[i])
+    positives = sum(labels)
+    true_positives = 0
+    previous_recall = 0.0
+    ap = 0.0
+    points = []
+    for rank, index in enumerate(order, 1):
+        true_positives += labels[index]
+        if rank < len(order) and scores[order[rank]] == scores[index]:
+            continue
+        precision = true_positives / rank
+        recall = true_positives / positives
+        ap += (recall - previous_recall) * precision
+        previous_recall = recall
+        points.append((precision, recall))
+    return {
+        "average_precision": ap,
+        "precision_at_recall_90": max(p for p, r in points if r >= 0.9),
+        "precision_at_recall_95": max(p for p, r in points if r >= 0.95),
+        "prevalence": positives / len(labels),
+    }
+
+
+def dcg(grades: list[int], k: int) -> float:
+    return sum((2**g - 1) / math.log2(i + 2) for i, g in enumerate(grades[:k]))
+
+
+def ranking_metrics(grades: list[int], scores: list[float], k: int) -> dict[str, float]:
+    if len(grades) != len(scores) or len(grades) < k or k < 1:
+        raise ValueError("Ranking inputs must be aligned and cover the cutoff")
+    if any(type(g) is not int or g not in range(4) for g in grades):
+        raise ValueError("Ranking requires fully judged integer grades 0 through 3")
+    if not all(math.isfinite(s) for s in scores):
+        raise ValueError("Ranking scores must be finite")
+    order = sorted(range(len(scores)), key=lambda i: (-scores[i], i))
+    ranked = [grades[i] for i in order]
+    useful = sum(g >= 2 for g in ranked[:k])
+    relevant = sum(g >= 2 for g in grades)
+    ideal = dcg(sorted(grades, reverse=True), k)
+    result = {"hit": float(useful > 0), "precision": useful / k, "useful": float(useful)}
+    if relevant:
+        result["recall"] = useful / relevant
+        result["ndcg"] = dcg(ranked, k) / ideal
+    return result
+
+
+def random_ranking_metrics(grades: list[int], k: int) -> dict[str, float]:
+    """Exact expectation under a uniform permutation of one fixed judged pool."""
+    n = len(grades)
+    if k < 1 or k > n or any(type(g) is not int or g not in range(4) for g in grades):
+        raise ValueError("Random reference requires judged grades and a valid cutoff")
+    relevant = sum(g >= 2 for g in grades)
+    misses = math.comb(n - relevant, k) if n - relevant >= k else 0
+    result = {"hit": 1 - misses / math.comb(n, k), "precision": relevant / n, "useful": k * relevant / n}
+    if relevant:
+        expected_dcg = mean(2**g - 1 for g in grades) * sum(1 / math.log2(i + 2) for i in range(k))
+        result["recall"] = k / n
+        result["ndcg"] = expected_dcg / dcg(sorted(grades, reverse=True), k)
+    return result
+
+
+def summarize_classifier(data: dict) -> dict:
+    rows = data["rows"]
+    result = {"rows": len(rows), "families": len({r["group"] for r in rows}), "models": {}}
+    for model in data["model_order"]:
+        by_facet = {}
+        for facet in FACETS:
+            resolved = [r for r in rows if r["labels"][facet] is not None]
+            if any(r["labels"][facet] not in (0, 1) for r in resolved):
+                raise ValueError("Invalid resolved classifier label")
+            by_facet[facet] = {"rows": len(resolved), **classification_metrics(
+                [r["labels"][facet] for r in resolved], [r["scores"][model][facet] for r in resolved]
+            )}
+        macro = {k: mean(v[k] for v in by_facet.values()) for k in ("average_precision", "precision_at_recall_90", "precision_at_recall_95")}
+        result["models"][model] = {"facets": by_facet, "macro": macro}
+    return result
+
+
+def summarize_reranker(data: dict) -> dict:
+    rows = data["rows"]
+    answerable = [r for r in rows if any(g >= 2 for g in r["grades"])]
+    output = {"queries": len(rows), "answerable": len(answerable), "models": {}}
+    for model in ["random", *data["model_order"]]:
+        scopes = {}
+        for scope, subset in [("answerable", answerable), ("all", rows)]:
+            cuts = {}
+            for k in CUTOFFS:
+                values = [random_ranking_metrics(r["grades"], k) if model == "random" else ranking_metrics(r["grades"], r["scores"][model], k) for r in subset]
+                fields = ("hit", "precision", "useful", "recall", "ndcg") if scope == "answerable" else ("hit", "precision", "useful")
+                cuts[str(k)] = {field: mean(v[field] for v in values) for field in fields}
+            scopes[scope] = cuts
+        output["models"][model] = scopes
+    return output
+
+
+def summarize_promotion(data: dict) -> dict:
+    """Hybrid search with reviewed exclusions inside each original prefix.
+
+    A null is permitted only for an explicitly excluded judging abstention.
+    Cut first, remove exclusions second, and never backfill from a deeper rank.
+    """
+    rows = data['rows']
+    if not rows or len({row['id'] for row in rows}) != len(rows):
+        raise ValueError('Promotion rows require unique nonempty query identities')
+    output = {'queries': len(rows), 'models': {}, 'public': {}}
+    for model in data['model_order']:
+        cutoffs = {}
+        for cutoff in (3, 5, 10, 20, 'selected'):
+            per_query = []
+            for row in rows:
+                counts = row['reference_grade_counts']
+                if set(counts) != {'0', '1', '2', '3'} or any(type(n) is not int or n < 0 for n in counts.values()):
+                    raise ValueError('Reference grade counts must cover grades zero through three')
+                ranked = row['ranked_grades'][model]
+                excluded = row['excluded'][model]
+                depth = row['selected_depth'][model] if cutoff == 'selected' else cutoff
+                if (len(ranked) != 20 or len(excluded) != 20 or type(depth) is not int
+                        or not 3 <= depth <= 20 or any(type(x) is not bool for x in excluded)):
+                    raise ValueError('Promotion rows require a bounded original top twenty')
+                if any((grade is not None if drop else type(grade) is not int or grade not in range(4))
+                       for grade, drop in zip(ranked, excluded, strict=True)):
+                    raise ValueError('Only declared abstentions may lack grades')
+                kept = [grade for grade, drop in zip(ranked[:depth], excluded[:depth], strict=True) if not drop]
+                if not kept:
+                    raise ValueError('Every scored prefix must retain judged passages')
+                ideal_grades = [g for g in (3, 2, 1, 0) for _ in range(min(counts[str(g)], len(kept)))][:len(kept)]
+                useful = sum(g >= 2 for g in kept)
+                positives = counts['2'] + counts['3']
+                ideal = dcg(ideal_grades, len(kept))
+                per_query.append({
+                    'hit': float(useful > 0), 'precision': useful / len(kept),
+                    'useful': useful, 'retained': len(kept), 'excluded': depth - len(kept),
+                    'ndcg': dcg(kept, len(kept)) / ideal if ideal else 0.0,
+                    'known_recall': useful / positives if positives else None,
+                })
+            useful = sum(row['useful'] for row in per_query)
+            retained = sum(row['retained'] for row in per_query)
+            recalls = [row['known_recall'] for row in per_query if row['known_recall'] is not None]
+            cutoffs[str(cutoff)] = {
+                'hit': mean(row['hit'] for row in per_query), 'precision': useful / retained,
+                'macro_precision': mean(row['precision'] for row in per_query),
+                'ndcg': mean(row['ndcg'] for row in per_query),
+                'known_positive_recall': mean(recalls) if recalls else None,
+                'recall_queries': len(recalls), 'useful': useful, 'retained': retained,
+                'excluded_positions': sum(row['excluded'] for row in per_query),
+                'mean_useful': useful / len(rows), 'mean_retained': retained / len(rows),
+            }
+        output['models'][model] = cutoffs
+    for dataset, panel in data['public'].items():
+        if not panel['rows'] or len({row['id'] for row in panel['rows']}) != len(panel['rows']):
+            raise ValueError('Public panel requires distinct query identities')
+        measured = panel['model_order']
+        for row in panel['rows']:
+            if set(row['ndcg@10']) != set(measured) or any(
+                isinstance(value, bool) or not isinstance(value, int | float)
+                or not math.isfinite(value) or not 0 <= value <= 1
+                for value in row['ndcg@10'].values()
+            ):
+                raise ValueError('Public nDCG values must be finite measured scores')
+        output['public'][dataset] = {
+            'queries': len(panel['rows']),
+            'ndcg@10': {model: mean(row['ndcg@10'][model] for row in panel['rows']) for model in measured},
+        }
+    return output
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--directory", type=Path, default=Path(__file__).resolve().parent)
+    parser.add_argument("--output", type=Path)
+    parser.add_argument("--model", choices=("classifier", "reranker", "promotion", "serving"), help="Recompute one comparison; default: every record in this package")
+    args = parser.parse_args()
+    summarizers = {"classifier": summarize_classifier, "reranker": summarize_reranker,
+                   "promotion": summarize_promotion, "serving": summarize_promotion}
+    names = [args.model] if args.model else [
+        name for name in summarizers if (args.directory / f"{name}.json").is_file()
+    ]
+    if not names:
+        parser.error("No evaluation records found in the selected directory")
+    result = {}
+    for name in names:
+        summarize = summarizers[name]
+        data = json.loads((args.directory / f"{name}.json").read_text(encoding="utf-8"))
+        ids = [r["id"] for r in data["rows"]]
+        if len(set(ids)) != len(ids):
+            raise ValueError(f"Repeated query/passage identity in {name}")
+        result[name] = summarize(data)
+    text = json.dumps(result, indent=2, allow_nan=False) + "\n"
+    if args.output:
+        args.output.write_text(text, encoding="utf-8")
+    else:
+        print(text, end="")
+
+
+if __name__ == "__main__":
+    main()

evaluation/reranker-summary.json

@@ -0,0 +1,204 @@
+{
+  "queries": 970,
+  "answerable": 873,
+  "models": {
+    "random": {
+      "answerable": {
+        "1": {
+          "hit": 0.4268041237113402,
+          "precision": 0.4268041237113402,
+          "useful": 0.4268041237113402,
+          "recall": 0.02,
+          "ndcg": 0.34859651993672613
+        },
+        "3": {
+          "hit": 0.726812305678285,
+          "precision": 0.4268041237113402,
+          "useful": 1.2804123711340205,
+          "recall": 0.06,
+          "ndcg": 0.3568668683201474
+        },
+        "5": {
+          "hit": 0.83263944209344,
+          "precision": 0.4268041237113402,
+          "useful": 2.134020618556701,
+          "recall": 0.1,
+          "ndcg": 0.3673965034684685
+        },
+        "10": {
+          "hit": 0.9245071219176823,
+          "precision": 0.4268041237113402,
+          "useful": 4.268041237113402,
+          "recall": 0.2,
+          "ndcg": 0.3973178040013556
+        },
+        "20": {
+          "hit": 0.9730113306184918,
+          "precision": 0.4268041237113402,
+          "useful": 8.536082474226804,
+          "recall": 0.4,
+          "ndcg": 0.46181890469238246
+        }
+      },
+      "all": {
+        "1": {
+          "hit": 0.3841237113402062,
+          "precision": 0.3841237113402062,
+          "useful": 0.3841237113402062
+        },
+        "3": {
+          "hit": 0.6541310751104565,
+          "precision": 0.3841237113402062,
+          "useful": 1.1523711340206186
+        },
+        "5": {
+          "hit": 0.749375497884096,
+          "precision": 0.3841237113402062,
+          "useful": 1.920618556701031
+        },
+        "10": {
+          "hit": 0.832056409725914,
+          "precision": 0.3841237113402062,
+          "useful": 3.841237113402062
+        },
+        "20": {
+          "hit": 0.8757101975566426,
+          "precision": 0.3841237113402062,
+          "useful": 7.682474226804124
+        }
+      }
+    },
+    "upstream": {
+      "answerable": {
+        "1": {
+          "hit": 0.6174112256586484,
+          "precision": 0.6174112256586484,
+          "useful": 0.6174112256586484,
+          "recall": 0.03681805740540042,
+          "ndcg": 0.5277368679430535
+        },
+        "3": {
+          "hit": 0.8316151202749141,
+          "precision": 0.5975563192058038,
+          "useful": 1.7926689576174113,
+          "recall": 0.10028426529744264,
+          "ndcg": 0.5180026543309457
+        },
+        "5": {
+          "hit": 0.8946162657502864,
+          "precision": 0.5857961053837343,
+          "useful": 2.928980526918671,
+          "recall": 0.15797988036502988,
+          "ndcg": 0.5264221881850474
+        },
+        "10": {
+          "hit": 0.9381443298969072,
+          "precision": 0.5673539518900343,
+          "useful": 5.673539518900344,
+          "recall": 0.2906998695460454,
+          "ndcg": 0.5518800541282721
+        },
+        "20": {
+          "hit": 0.981672394043528,
+          "precision": 0.5446162657502863,
+          "useful": 10.892325315005728,
+          "recall": 0.5378196846495221,
+          "ndcg": 0.6153644012290921
+        }
+      },
+      "all": {
+        "1": {
+          "hit": 0.5556701030927835,
+          "precision": 0.5556701030927835,
+          "useful": 0.5556701030927835
+        },
+        "3": {
+          "hit": 0.7484536082474227,
+          "precision": 0.5378006872852233,
+          "useful": 1.6134020618556701
+        },
+        "5": {
+          "hit": 0.8051546391752578,
+          "precision": 0.5272164948453608,
+          "useful": 2.636082474226804
+        },
+        "10": {
+          "hit": 0.8443298969072165,
+          "precision": 0.510618556701031,
+          "useful": 5.106185567010309
+        },
+        "20": {
+          "hit": 0.8835051546391752,
+          "precision": 0.49015463917525776,
+          "useful": 9.803092783505155
+        }
+      }
+    },
+    "finetuned": {
+      "answerable": {
+        "1": {
+          "hit": 0.8144329896907216,
+          "precision": 0.8144329896907216,
+          "useful": 0.8144329896907216,
+          "recall": 0.0523348139467294,
+          "ndcg": 0.718158511972945
+        },
+        "3": {
+          "hit": 0.9255441008018328,
+          "precision": 0.8003054600992745,
+          "useful": 2.4009163802978235,
+          "recall": 0.14284244957257694,
+          "ndcg": 0.7197739775683897
+        },
+        "5": {
+          "hit": 0.9404352806414662,
+          "precision": 0.7793814432989691,
+          "useful": 3.8969072164948453,
+          "recall": 0.22343247007861236,
+          "ndcg": 0.7212210726901884
+        },
+        "10": {
+          "hit": 0.9690721649484536,
+          "precision": 0.7418098510882016,
+          "useful": 7.418098510882016,
+          "recall": 0.39954728808322054,
+          "ndcg": 0.7406757160512017
+        },
+        "20": {
+          "hit": 0.9896907216494846,
+          "precision": 0.6734249713631157,
+          "useful": 13.468499427262314,
+          "recall": 0.6787919700186864,
+          "ndcg": 0.7868944732410001
+        }
+      },
+      "all": {
+        "1": {
+          "hit": 0.7329896907216494,
+          "precision": 0.7329896907216494,
+          "useful": 0.7329896907216494
+        },
+        "3": {
+          "hit": 0.8329896907216495,
+          "precision": 0.7202749140893471,
+          "useful": 2.160824742268041
+        },
+        "5": {
+          "hit": 0.8463917525773196,
+          "precision": 0.7014432989690722,
+          "useful": 3.507216494845361
+        },
+        "10": {
+          "hit": 0.8721649484536083,
+          "precision": 0.6676288659793814,
+          "useful": 6.676288659793815
+        },
+        "20": {
+          "hit": 0.8907216494845361,
+          "precision": 0.6060824742268042,
+          "useful": 12.121649484536082
+        }
+      }
+    }
+  }
+}

evaluation/serving-qualification.json

@@ -0,0 +1,134 @@
+{
+  "schema": "daecore.serving-preparation-summary.v1",
+  "date": "2026-09-28",
+  "status": "local-model-serving-checks-complete-release-transition-pending",
+  "hardware": "Windows x64, NVIDIA RTX 3060 Ti 8 GiB",
+  "unmeasured_targets": [
+    "AMD",
+    "Intel",
+    "Linux x64"
+  ],
+  "runtimes": {
+    "onnxruntime": "1.24.4",
+    "vulkan_plugin": "0.4.0",
+    "backend": "Vulkan",
+    "storage_buffer_cache": "lazyRelease"
+  },
+  "models": {
+    "gemma": {
+      "vectors": 74,
+      "providers": {
+        "cpu": {
+          "receipt_sha256": "31851791104fa95d6b7b25d35f950b25e239a3d6f8fa458addb2a44917c25cb4",
+          "maximum_vector_delta": 3.0174851417541504e-07,
+          "concurrent_delta": 0.0
+        },
+        "cuda": {
+          "receipt_sha256": "6977d0aa0ebf9c39ea0fd24078c47367d1d50e81184653225fcd10758961903e",
+          "maximum_vector_delta": 2.644956111907959e-07,
+          "concurrent_delta": 2.2351741790771484e-08
+        },
+        "vulkan": {
+          "receipt_sha256": "567fb89ea195e8b188203d2334f3eda82994ae8c7fc97f2126fac3e6f4766079",
+          "maximum_vector_delta": 5.438923835754395e-07,
+          "concurrent_delta": 0.0
+        }
+      },
+      "admission": {
+        "cases": 3841,
+        "semantic_fallbacks": 679,
+        "new_useful_exclusions": 0,
+        "additional_noise_retained": 2,
+        "receipt_sha256": "71550f606c95c1e1192dd5939488b280d5f2f2f091f99413f6b63f4408a5bc2c"
+      }
+    },
+    "classifier": {
+      "cpu": {
+        "rows": 1300,
+        "maximum_posterior_delta": 5.612167303103988e-06,
+        "changed_threshold_labels": {
+          "contract": 0,
+          "recall_leaning": 0
+        },
+        "receipt_sha256": "ea545fb0a30ff25bf3da178b60593e934e6025e789c2db67290ff5360e8669a4"
+      },
+      "cuda": {
+        "rows": 1300,
+        "maximum_posterior_delta": 5.612167303103988e-06,
+        "changed_threshold_labels": {
+          "contract": 0,
+          "recall_leaning": 0
+        },
+        "receipt_sha256": "5616891e9dfdc3b6615951901cb24aa31189d6a81ed84b0080f461b85c045065"
+      },
+      "vulkan": {
+        "rows": 1300,
+        "maximum_posterior_delta": 5.612167303103988e-06,
+        "changed_threshold_labels": {
+          "contract": 0,
+          "recall_leaning": 0
+        },
+        "receipt_sha256": "c7ec971e020d9771adf0756316f468629cc05e300d7edf9abb52612901adf6b0"
+      }
+    },
+    "ettin": {
+      "queries": 970,
+      "unjudged_top20": 0,
+      "quality_evidence": "serving.json",
+      "quality_receipt_sha256": "805a1b0acd7baaf248e182908bc23cd52171e40aa8f1281799a20d15b58c6905",
+      "full_replay_seconds_p50_p95": [
+        1.7189999999827705,
+        2.610000000044238
+      ],
+      "matched_20_pool_seconds_p50_p95": {
+        "vulkan": [
+          1.5565476999909151,
+          2.1063475799834124
+        ],
+        "predecessor_directml": [
+          1.9197023500164505,
+          2.701977299965802
+        ]
+      },
+      "pressure": "One preflight refusal after 322 queries with lazy release; resumed all remaining rows with zero further retries. An earlier default-cache run stopped after 424. No native OOM observed.",
+      "precision": "FP32 Vulkan versus FP16 CUDA; rankings not bit-exact",
+      "selector": "Unchanged CUDA mapping transferred for measurement; new package/provider binding pending"
+    }
+  },
+  "operator_runtime_changed": false,
+  "published": false,
+  "remaining": [
+    "immutable-publication-revisions",
+    "profile-and-calibration-release-bindings",
+    "isolated-package-update-and-index-rebuild-recovery",
+    "DirectML-current-route-retirement"
+  ],
+  "worker_recovery": {
+    "receipt_sha256": "46205d39521bde40d1228c837e57f88e1a941d42f6846f4a244c48b513347cc6",
+    "all_three_consumer_outputs_identical_after_owned_worker_crash": true,
+    "ettin_maximum_envelope": {
+      "tokens": 1153,
+      "max_abs_logit_delta_vs_cpu": 5.91278076171875e-05
+    }
+  },
+  "retained_reranker_public": {
+    "scope": "Retained matched public pools retrieved by upstream Gemma; upstream and production Ettin PyTorch FP16; not a new Vulkan public replay",
+    "source_sha256": "0814db560081af8376e4b1f3e3d2bcd5c4d9b8b208f7fb8b7172f113e8123ef3",
+    "panels": {
+      "fiqa": {
+        "queries": 648,
+        "ndcg@10": {
+          "upstream": 0.48603492061219766,
+          "finetuned": 0.452680922415071
+        }
+      },
+      "scifact": {
+        "queries": 300,
+        "ndcg@10": {
+          "upstream": 0.7487436478294831,
+          "finetuned": 0.7543833016238701
+        }
+      }
+    }
+  }
+}

evaluation/serving-summary.json

@@ -0,0 +1 @@
+{"queries":970,"models":{"cuda":{"3":{"hit":0.8536082474226804,"precision":0.7157640565712314,"macro_precision":0.7166666666666667,"ndcg":0.6576832474683237,"known_positive_recall":0.09521315062968948,"recall_queries":895,"useful":2075,"retained":2899,"excluded_positions":11,"mean_useful":2.1391752577319587,"mean_retained":2.988659793814433},"5":{"hit":0.8845360824742268,"precision":0.7028311634635255,"macro_precision":0.7031786941580757,"ndcg":0.6554718901879515,"known_positive_recall":0.15383335671735354,"recall_queries":895,"useful":3401,"retained":4839,"excluded_positions":11,"mean_useful":3.5061855670103093,"mean_retained":4.988659793814433},"10":{"hit":0.8989690721649485,"precision":0.67180070291503,"macro_precision":0.6722631320569465,"ndcg":0.6610872123112499,"known_positive_recall":0.2802986435290336,"recall_queries":895,"useful":6499,"retained":9674,"excluded_positions":26,"mean_useful":6.7,"mean_retained":9.97319587628866},"20":{"hit":0.9103092783505154,"precision":0.61794500723589,"macro_precision":0.6184956360634603,"ndcg":0.6840908385828889,"known_positive_recall":0.4884616340614924,"recall_queries":895,"useful":11956,"retained":19348,"excluded_positions":52,"mean_useful":12.32577319587629,"mean_retained":19.94639175257732},"selected":{"hit":0.8618556701030928,"precision":0.8033770583310076,"macro_precision":0.7040325511660611,"ndcg":0.6598955189009045,"known_positive_recall":0.2079138769097556,"recall_queries":895,"useful":5757,"retained":7166,"excluded_positions":17,"mean_useful":5.935051546391753,"mean_retained":7.387628865979382}},"vulkan":{"3":{"hit":0.8525773195876288,"precision":0.7157640565712314,"macro_precision":0.7166666666666667,"ndcg":0.6577263968717306,"known_positive_recall":0.09519545071125639,"recall_queries":895,"useful":2075,"retained":2899,"excluded_positions":11,"mean_useful":2.1391752577319587,"mean_retained":2.988659793814433},"5":{"hit":0.8845360824742268,"precision":0.7023563455973543,"macro_precision":0.702766323024055,"ndcg":0.6550340124265787,"known_positive_recall":0.15374094654114448,"recall_queries":895,"useful":3398,"retained":4838,"excluded_positions":12,"mean_useful":3.5030927835051546,"mean_retained":4.9876288659793815},"10":{"hit":0.8989690721649485,"precision":0.6723514211886304,"macro_precision":0.6727589592538046,"ndcg":0.6615310405020679,"known_positive_recall":0.2805978824757964,"recall_queries":895,"useful":6505,"retained":9675,"excluded_positions":25,"mean_useful":6.706185567010309,"mean_retained":9.974226804123711},"20":{"hit":0.9103092783505154,"precision":0.6178416373785404,"macro_precision":0.6183925432799551,"ndcg":0.6840389893720067,"known_positive_recall":0.4883516349357555,"recall_queries":895,"useful":11954,"retained":19348,"excluded_positions":52,"mean_useful":12.323711340206186,"mean_retained":19.94639175257732},"selected":{"hit":0.8608247422680413,"precision":0.8027855153203343,"macro_precision":0.7038533836490732,"ndcg":0.6599601279630594,"known_positive_recall":0.2082001919384574,"recall_queries":895,"useful":5764,"retained":7180,"excluded_positions":17,"mean_useful":5.942268041237114,"mean_retained":7.402061855670103}}},"public":{}}

evaluation/svg_figures.py

@@ -0,0 +1,576 @@
+"""Dependency-free SVG charts for tracked evaluation figures.
+
+The evaluation figures are generated from private per-row evidence into tracked SVG files.
+Keeping the renderer inside the repository, with no plotting dependency, means a figure can be
+regenerated by any contributor with the private inputs and compared byte for byte.
+
+Everything is emitted as presentation attributes rather than CSS. Markdown hosts sanitize
+embedded stylesheets and scripts out of SVG, so a figure that carries its styling in attributes
+renders the same in the repository, on a model-card host, and in a local viewer.
+
+Layout is measured rather than assumed: legend entries, panel titles, and reference-line labels
+are placed from an estimated text width, so a longer label reflows instead of overlapping its
+neighbour. ``text_width`` approximates a sans-serif advance table, which is enough to keep
+elements apart but is not a substitute for a real font metric.
+"""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass, field
+from xml.sax.saxutils import escape
+
+# Okabe-Ito, chosen because it stays distinguishable under the common colour-vision
+# deficiencies and prints legibly in greyscale.
+PALETTE = ("#0072B2", "#D55E00", "#009E73", "#CC79A7", "#E69F00", "#56B4E9", "#000000")
+FONT_STACK = "system-ui, Segoe UI, Roboto, Helvetica, Arial, sans-serif"
+FONT = f'font-family="{FONT_STACK}"'
+
+INK = "#1a1a1a"
+MUTED = "#5c5c5c"
+AXIS = "#8a8a8a"
+GRID = "#e8e8e8"
+GRID_STRONG = "#d0d0d0"
+
+# Per-character advance as a fraction of font size, for a humanist sans at normal weight.
+_NARROW = set("iljft.,;:|!()[]{}I '")
+_WIDE = set("mwMW@%")
+_DIGIT = set("0123456789")
+
+
+def text_width(text: str, size: float, *, weight: str = "normal") -> float:
+    """Estimate rendered width in user units."""
+
+    total = 0.0
+    for character in text:
+        if character in _NARROW:
+            total += 0.30
+        elif character in _WIDE:
+            total += 0.86
+        elif character in _DIGIT:
+            total += 0.56
+        elif character.isupper():
+            total += 0.66
+        else:
+            total += 0.52
+    if weight in {"600", "700", "bold"}:
+        total *= 1.05
+    return total * size
+
+
+def _fit_lines(text: str, size: float, limit: float, *, weight: str = "normal") -> list[str]:
+    """Wrap to at most two lines, breaking on whitespace."""
+
+    if text_width(text, size, weight=weight) <= limit:
+        return [text]
+    words = text.split(" ")
+    line: list[str] = []
+    for index, word in enumerate(words):
+        candidate = " ".join([*line, word])
+        if line and text_width(candidate, size, weight=weight) > limit:
+            return [" ".join(line), " ".join(words[index:])]
+        line.append(word)
+    return [" ".join(line)]
+
+
+@dataclass
+class Series:
+    label: str
+    x: list[float]
+    y: list[float]
+    color: str = PALETTE[0]
+    dash: str | None = None
+    marker: bool = True
+    width: float = 1.9
+    marker_size: float = 2.6
+
+
+@dataclass
+class Band:
+    """A shaded interval drawn behind its series."""
+
+    x: list[float]
+    low: list[float]
+    high: list[float]
+    color: str = PALETTE[0]
+    opacity: float = 0.16
+
+
+@dataclass
+class Counts:
+    """A support strip under the plot: how many rows sit behind each x position."""
+
+    x: list[float]
+    values: list[float]
+    color: str = MUTED
+    label: str = "rows per bin"
+
+
+@dataclass
+class Panel:
+    title: str
+    series: list[Series] = field(default_factory=list)
+    bands: list[Band] = field(default_factory=list)
+    xlabel: str = ""
+    ylabel: str = ""
+    xlim: tuple[float, float] | None = None
+    ylim: tuple[float, float] | None = None
+    xticks: list[tuple[float, str]] | None = None
+    yticks: list[tuple[float, str]] | None = None
+    xscale: str = "linear"
+    hlines: list[tuple[float, str, str]] = field(default_factory=list)
+    diagonal: bool = False
+    notes: list[str] = field(default_factory=list)
+    counts: Counts | None = None
+
+
+def _fmt(value: float) -> str:
+    if abs(value) >= 1e6:
+        return f"{value:.3g}"
+    text = f"{value:.3f}".rstrip("0").rstrip(".")
+    return text or "0"
+
+
+def _auto_ticks(low: float, high: float, count: int = 5) -> list[tuple[float, str]]:
+    if high <= low:
+        high = low + 1.0
+    step = (high - low) / count
+    magnitude = 10 ** math.floor(math.log10(step)) if step > 0 else 1.0
+    for factor in (1, 2, 2.5, 5, 10):
+        if step <= factor * magnitude:
+            step = factor * magnitude
+            break
+    start = math.ceil(low / step) * step
+    ticks = []
+    value = start
+    while value <= high + 1e-9:
+        ticks.append((value, _fmt(0.0 if abs(value) < step * 1e-6 else value)))
+        value += step
+    return ticks
+
+
+def _text(
+    x: float,
+    y: float,
+    body: str,
+    *,
+    size: float,
+    fill: str = INK,
+    anchor: str = "start",
+    weight: str | None = None,
+) -> str:
+    weight_attr = f' font-weight="{weight}"' if weight else ""
+    return (
+        f'<text x="{x:.1f}" y="{y:.1f}" text-anchor="{anchor}" font-size="{size}" '
+        f'fill="{fill}"{weight_attr} {FONT}>{escape(body)}</text>'
+    )
+
+
+_TITLE_SIZE = 12.5
+
+# Everything below the plot box is stacked in fixed bands rather than placed at absolute
+# offsets, so an x-axis label, a support strip, and a note can coexist without overlapping.
+_TICK_BAND = 18.0
+_XLABEL_BAND = 16.0
+_COUNTS_BAND = 30.0
+_NOTE_BAND = 12.0
+_NOTE_LEAD = 6.0
+_FLOOR_SLACK = 8.0
+
+
+def _panel_bottom(panel: Panel) -> float:
+    bottom = _TICK_BAND + _FLOOR_SLACK
+    if panel.xlabel:
+        bottom += _XLABEL_BAND
+    if panel.counts:
+        bottom += _COUNTS_BAND
+    if panel.notes:
+        bottom += _NOTE_LEAD + _NOTE_BAND * len(panel.notes)
+    return bottom
+
+
+def _panel_svg(panel: Panel, width: float, height: float, *, title_rows: int | None = None) -> str:
+    left, right = 54.0, 14.0
+    plot_w = width - left - right
+    title_lines = _fit_lines(panel.title, _TITLE_SIZE, plot_w, weight="600")
+    top = 18.0 + 14.0 * (title_rows or len(title_lines))
+    plot_h = height - top - _panel_bottom(panel)
+
+    xs = [x for s in panel.series for x in s.x] or [0.0, 1.0]
+    ys = [y for s in panel.series for y in s.y] or [0.0, 1.0]
+    ys += [value for band in panel.bands for value in (*band.low, *band.high)]
+    ys += [level for level, _, _ in panel.hlines]
+    xlim = panel.xlim or (min(xs), max(xs))
+    ylim = panel.ylim or (min(ys), max(ys))
+    if ylim[0] == ylim[1]:
+        ylim = (ylim[0] - 0.5, ylim[1] + 0.5)
+
+    def tx(value: float) -> float:
+        if panel.xscale == "log":
+            lo, hi = math.log(xlim[0]), math.log(xlim[1])
+            return left + (math.log(value) - lo) / (hi - lo) * plot_w
+        return left + (value - xlim[0]) / (xlim[1] - xlim[0]) * plot_w
+
+    def ty(value: float) -> float:
+        return top + plot_h - (value - ylim[0]) / (ylim[1] - ylim[0]) * plot_h
+
+    parts: list[str] = []
+    for index, line in enumerate(title_lines):
+        parts.append(
+            _text(
+                left + plot_w / 2,
+                16.0 + 14.0 * index,
+                line,
+                size=_TITLE_SIZE,
+                anchor="middle",
+                weight="600",
+            )
+        )
+
+    xticks = panel.xticks or _auto_ticks(*xlim)
+    yticks = panel.yticks or _auto_ticks(*ylim)
+    for value, label in yticks:
+        if not ylim[0] - 1e-9 <= value <= ylim[1] + 1e-9:
+            continue
+        y = ty(value)
+        parts.append(
+            f'<line x1="{left}" y1="{y:.1f}" x2="{left + plot_w:.1f}" y2="{y:.1f}" '
+            f'stroke="{GRID}" stroke-width="1"/>'
+        )
+        parts.append(_text(left - 6, y + 3.4, label, size=10, fill=MUTED, anchor="end"))
+    for value, label in xticks:
+        if not xlim[0] - 1e-9 <= value <= xlim[1] + 1e-9:
+            continue
+        x = tx(value)
+        parts.append(
+            f'<line x1="{x:.1f}" y1="{top}" x2="{x:.1f}" y2="{top + plot_h:.1f}" '
+            f'stroke="{GRID}" stroke-width="1"/>'
+        )
+        parts.append(_text(x, top + plot_h + 13, label, size=10, fill=MUTED, anchor="middle"))
+
+    if panel.diagonal:
+        parts.append(
+            f'<line x1="{tx(xlim[0]):.1f}" y1="{ty(ylim[0]):.1f}" '
+            f'x2="{tx(xlim[1]):.1f}" y2="{ty(ylim[1]):.1f}" stroke="{AXIS}" '
+            f'stroke-width="1.1" stroke-dasharray="4 3"/>'
+        )
+
+    for band in panel.bands:
+        forward = " ".join(
+            f"{tx(x):.1f},{ty(y):.1f}" for x, y in zip(band.x, band.high, strict=True)
+        )
+        backward = " ".join(
+            f"{tx(x):.1f},{ty(y):.1f}"
+            for x, y in zip(reversed(band.x), reversed(band.low), strict=True)
+        )
+        parts.append(
+            f'<polygon points="{forward} {backward}" fill="{band.color}" '
+            f'fill-opacity="{band.opacity}" stroke="none"/>'
+        )
+
+    # A reference line carries its label at the left margin over a solid backing box, so the
+    # label never lands on the data it is a reference for.
+    for value, label, color in panel.hlines:
+        y = ty(value)
+        parts.append(
+            f'<line x1="{left}" y1="{y:.1f}" x2="{left + plot_w:.1f}" y2="{y:.1f}" '
+            f'stroke="{color}" stroke-width="1.2" stroke-dasharray="5 3"/>'
+        )
+        if not label:
+            continue
+        label_w = text_width(label, 9.5) + 8.0
+        parts.append(
+            f'<rect x="{left + 3:.1f}" y="{y - 12:.1f}" width="{label_w:.1f}" height="11.5" '
+            f'fill="white" fill-opacity="0.9" stroke="none"/>'
+        )
+        parts.append(_text(left + 7, y - 3.5, label, size=9.5, fill=color))
+
+    for series in panel.series:
+        points = " ".join(
+            f"{tx(x):.1f},{ty(y):.1f}" for x, y in zip(series.x, series.y, strict=True)
+        )
+        dash = f' stroke-dasharray="{series.dash}"' if series.dash else ""
+        parts.append(
+            f'<polyline points="{points}" fill="none" stroke="{series.color}" '
+            f'stroke-width="{series.width}" stroke-linejoin="round" '
+            f'stroke-linecap="round"{dash}/>'
+        )
+        if series.marker:
+            for x, y in zip(series.x, series.y, strict=True):
+                parts.append(
+                    f'<circle cx="{tx(x):.1f}" cy="{ty(y):.1f}" r="{series.marker_size}" '
+                    f'fill="{series.color}"/>'
+                )
+
+    parts.append(
+        f'<rect x="{left}" y="{top}" width="{plot_w:.1f}" height="{plot_h:.1f}" fill="none" '
+        f'stroke="{AXIS}" stroke-width="1"/>'
+    )
+
+    cursor = top + plot_h + _TICK_BAND
+    if panel.xlabel:
+        parts.append(
+            _text(
+                left + plot_w / 2,
+                cursor + 11,
+                panel.xlabel,
+                size=10.5,
+                fill=MUTED,
+                anchor="middle",
+            )
+        )
+        cursor += _XLABEL_BAND
+    if panel.counts:
+        counts = panel.counts
+        if not counts.values:
+            raise ValueError("a support strip needs at least one count")
+        strip_top = cursor + 2.0
+        strip_h = 15.0
+        peak = max(counts.values) or 1.0
+        slot = plot_w / max(len(counts.x), 1) * 0.7
+        for x, value in zip(counts.x, counts.values, strict=True):
+            bar_h = (value / peak) * strip_h
+            parts.append(
+                f'<rect x="{tx(x) - slot / 2:.1f}" y="{strip_top + strip_h - bar_h:.1f}" '
+                f'width="{slot:.1f}" height="{bar_h:.1f}" fill="{counts.color}" '
+                f'fill-opacity="0.5"/>'
+            )
+        parts.append(
+            f'<line x1="{left}" y1="{strip_top + strip_h:.1f}" x2="{left + plot_w:.1f}" '
+            f'y2="{strip_top + strip_h:.1f}" stroke="{GRID_STRONG}" stroke-width="1"/>'
+        )
+        parts.append(_text(left, strip_top + strip_h + 9, counts.label, size=9, fill=MUTED))
+        parts.append(
+            _text(
+                left + plot_w,
+                strip_top + strip_h + 9,
+                f"tallest {int(peak):,}",
+                size=9,
+                fill=MUTED,
+                anchor="end",
+            )
+        )
+        cursor += _COUNTS_BAND
+    for index, note in enumerate(panel.notes):
+        parts.append(
+            _text(left, cursor + _NOTE_LEAD + 9 + _NOTE_BAND * index, note, size=9.5, fill=MUTED)
+        )
+    if panel.ylabel:
+        parts.append(
+            f'<text transform="translate(13,{top + plot_h / 2:.1f}) rotate(-90)" '
+            f'text-anchor="middle" font-size="10.5" fill="{MUTED}" {FONT}>'
+            f"{escape(panel.ylabel)}</text>"
+        )
+    return "\n".join(parts)
+
+
+def _legend_svg(
+    entries: list[tuple[str, str, str | None]],
+    *,
+    x: float,
+    y: float,
+    max_width: float,
+) -> tuple[str, float]:
+    """Flow legend entries across as many rows as their measured widths need."""
+
+    swatch, gap, pad = 22.0, 7.0, 22.0
+    parts: list[str] = []
+    cursor_x, cursor_y, rows = x, y, 1
+    for label, color, dash in entries:
+        entry_w = swatch + gap + text_width(label, 11) + pad
+        if cursor_x > x and cursor_x + entry_w - pad > x + max_width:
+            cursor_x, cursor_y, rows = x, cursor_y + 16.0, rows + 1
+        dash_attr = f' stroke-dasharray="{dash}"' if dash else ""
+        parts.append(
+            f'<line x1="{cursor_x:.1f}" y1="{cursor_y:.1f}" x2="{cursor_x + swatch:.1f}" '
+            f'y2="{cursor_y:.1f}" stroke="{color}" stroke-width="2.4" '
+            f'stroke-linecap="round"{dash_attr}/>'
+        )
+        parts.append(_text(cursor_x + swatch + gap, cursor_y + 3.8, label, size=11))
+        cursor_x += entry_w
+    return "\n".join(parts), 16.0 * rows
+
+
+def _dedupe(entries: list[tuple[str, str, str | None]]) -> list[tuple[str, str, str | None]]:
+    seen: set[tuple[str, str, str | None]] = set()
+    unique = []
+    for entry in entries:
+        if entry not in seen:
+            seen.add(entry)
+            unique.append(entry)
+    return unique
+
+
+def _open_svg(width: float, height: float, title: str, description: str) -> list[str]:
+    return [
+        f'<svg xmlns="http://www.w3.org/2000/svg" width="{width:.0f}" height="{height:.0f}" '
+        f'viewBox="0 0 {width:.0f} {height:.0f}" role="img" aria-label="{escape(title)}">',
+        f"<title>{escape(title)}</title>",
+        f"<desc>{escape(description)}</desc>",
+        '<rect width="100%" height="100%" fill="white"/>',
+    ]
+
+
+def render_grid(
+    panels: list[Panel],
+    *,
+    columns: int,
+    title: str,
+    subtitle: str = "",
+    panel_width: float = 352.0,
+    panel_height: float = 256.0,
+    legend: list[tuple[str, str, str | None]] | None = None,
+    description: str = "",
+) -> str:
+    """Render panels on a grid with one shared title and a flowed legend.
+
+    A final short row is centred, so a five-panel figure on three columns has no empty cell.
+    """
+
+    if not panels:
+        raise ValueError("render_grid needs at least one panel")
+    if columns < 1:
+        raise ValueError("render_grid needs at least one column")
+    legend = _dedupe(legend or [])
+    rows = math.ceil(len(panels) / columns)
+    width = columns * panel_width
+    header = 26.0 + (16.0 if subtitle else 0.0)
+    legend_svg, legend_h = "", 0.0
+    if legend:
+        legend_svg, legend_h = _legend_svg(legend, x=18.0, y=header + 10.0, max_width=width - 36.0)
+        legend_h += 8.0
+    height = header + legend_h + rows * panel_height
+    parts = _open_svg(width, height, title, description or title)
+    parts.append(_text(width / 2, 19, title, size=14.5, anchor="middle", weight="700"))
+    if subtitle:
+        parts.append(_text(width / 2, 34, subtitle, size=11, fill=MUTED, anchor="middle"))
+    if legend_svg:
+        parts.append(legend_svg)
+    # One title row count for the whole grid, so a panel whose title wraps does not push its
+    # plot box below its neighbours'.
+    title_rows = max(
+        len(_fit_lines(panel.title, _TITLE_SIZE, panel_width - 68.0, weight="600"))
+        for panel in panels
+    )
+    for index, panel in enumerate(panels):
+        row, column = divmod(index, columns)
+        in_row = min(columns, len(panels) - row * columns)
+        offset = (columns - in_row) * panel_width / 2.0
+        px = offset + column * panel_width
+        py = header + legend_h + row * panel_height
+        parts.append(f'<g transform="translate({px:.1f},{py:.1f})">')
+        parts.append(_panel_svg(panel, panel_width, panel_height, title_rows=title_rows))
+        parts.append("</g>")
+    parts.append("</svg>")
+    return "\n".join(parts) + "\n"
+
+
+def render_bars(
+    groups: list[str],
+    series: list[tuple[str, list[float], str]],
+    *,
+    title: str,
+    ylabel: str,
+    subtitle: str = "",
+    ylim: tuple[float, float] = (0.0, 1.0),
+    reference: list[tuple[str, list[float], str]] | None = None,
+    notes: list[str] | None = None,
+    separator_before: int | None = None,
+    width: float = 780.0,
+    height: float = 350.0,
+    description: str = "",
+) -> str:
+    """Render grouped bars with per-group dashed reference levels.
+
+    Bars keep a zero baseline. Value labels are drawn only where a bar is wide enough to hold
+    one, because a crowded label is worse than none.
+    """
+
+    if not groups or not series:
+        raise ValueError("render_bars needs at least one group and one series")
+    if any(len(values) != len(groups) for _, values, _ in series):
+        raise ValueError("every bar series must carry one value per group")
+    notes = notes or []
+    left, right, top = 54.0, 16.0, 26.0 + (15.0 if subtitle else 0.0)
+    legend_entries = [(label, color, None) for label, _, color in series]
+    legend_entries += [(label, color, "5 3") for label, _, color in reference or []]
+    legend_svg, legend_h = _legend_svg(
+        _dedupe(legend_entries), x=18.0, y=top + 12.0, max_width=width - 36.0
+    )
+    top += legend_h + 10.0
+    bottom = 46.0 + 12.0 * len(notes)
+    plot_w, plot_h = width - left - right, height - top - bottom
+    group_w = plot_w / len(groups)
+    bar_w = group_w * 0.74 / len(series)
+
+    def ty(value: float) -> float:
+        return top + plot_h - (value - ylim[0]) / (ylim[1] - ylim[0]) * plot_h
+
+    parts = _open_svg(width, height, title, description or title)
+    parts.append(_text(width / 2, 19, title, size=14.5, anchor="middle", weight="700"))
+    if subtitle:
+        parts.append(_text(width / 2, 34, subtitle, size=11, fill=MUTED, anchor="middle"))
+    parts.append(legend_svg)
+    for value, label in _auto_ticks(*ylim):
+        y = ty(value)
+        parts.append(
+            f'<line x1="{left}" y1="{y:.1f}" x2="{left + plot_w:.1f}" y2="{y:.1f}" '
+            f'stroke="{GRID}" stroke-width="1"/>'
+        )
+        parts.append(_text(left - 6, y + 3.4, label, size=10, fill=MUTED, anchor="end"))
+
+    label_fits = bar_w - 2 >= text_width("0.000", 8.5) + 2
+    for g_index, group in enumerate(groups):
+        gx = left + g_index * group_w + group_w * 0.13
+        for s_index, (_, values, color) in enumerate(series):
+            value = values[g_index]
+            x = gx + s_index * bar_w
+            parts.append(
+                f'<rect x="{x:.1f}" y="{ty(value):.1f}" width="{bar_w - 2:.1f}" '
+                f'height="{ty(ylim[0]) - ty(value):.1f}" fill="{color}"/>'
+            )
+            if label_fits:
+                parts.append(
+                    _text(
+                        x + (bar_w - 2) / 2,
+                        ty(value) - 4,
+                        f"{value:.3f}",
+                        size=8.5,
+                        anchor="middle",
+                    )
+                )
+        for _, values, color in reference or []:
+            y = ty(values[g_index])
+            parts.append(
+                f'<line x1="{gx - 3:.1f}" y1="{y:.1f}" '
+                f'x2="{gx + group_w * 0.74 + 3:.1f}" y2="{y:.1f}" stroke="{color}" '
+                f'stroke-width="1.6" stroke-dasharray="5 3"/>'
+            )
+        parts.append(
+            _text(
+                left + g_index * group_w + group_w / 2,
+                top + plot_h + 16,
+                group,
+                size=11,
+                anchor="middle",
+            )
+        )
+    if separator_before is not None and 0 < separator_before < len(groups):
+        x = left + separator_before * group_w
+        parts.append(
+            f'<line x1="{x:.1f}" y1="{top}" x2="{x:.1f}" y2="{top + plot_h + 6:.1f}" '
+            f'stroke="{GRID_STRONG}" stroke-width="1.4"/>'
+        )
+    parts.append(
+        f'<rect x="{left}" y="{top}" width="{plot_w:.1f}" height="{plot_h:.1f}" fill="none" '
+        f'stroke="{AXIS}" stroke-width="1"/>'
+    )
+    parts.append(
+        f'<text transform="translate(13,{top + plot_h / 2:.1f}) rotate(-90)" '
+        f'text-anchor="middle" font-size="10.5" fill="{MUTED}" {FONT}>'
+        f"{escape(ylabel)}</text>"
+    )
+    for index, note in enumerate(notes):
+        parts.append(_text(left, top + plot_h + 34 + 12 * index, note, size=9.5, fill=MUTED))
+    parts.append("</svg>")
+    return "\n".join(parts) + "\n"

model.directml.onnx → model.vulkan.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:05abdc03628b599712a993381919aef1e22fcc5ad6384608abc2e069c402e369
-size 2242802
+oid sha256:cf27725a15982be3a0c329aec48486e5e048de5b4d3028f0cf88915eafa79b40
+size 2243288

publication-manifest.json

@@ -1,6 +1,6 @@
 {
   "schema": "daecore.retrieval-publication-manifest",
-  "tool_sha256": "2369a55f4fa3f007f800834880b6e4a65624c91589b469f48133ebc058c3d2ef",
+  "tool_sha256": "5ac2ef9465fe9a3454bfd2e61c7aa651b08fbdfef9473fb5b3455585ba560a7b",
   "model_id": "ettin-150m-memory-reranker-ft-v1",
   "public_repo": "Daecore/ettin-150m-memory-reranker-ft-v1",
   "role": "cross-encoder-reranker",
@@ -9,21 +9,30 @@
     "revision": "025501c4e0f9bbeb4c5b198318e0089ff061cc14",
     "tree_sha256": "e0edfb38cc495f23507de50233d0965f4e72ad0872e70e6f4e8fc889d394ea01"
   },
-  "export_receipt_sha256": "4faf07d568cc5da6400e9aa2645d73b2a2562cb720742c64106b2a70f43b12ba",
+  "export_receipt_sha256": "92cc733460256c481547efc3e3aaeee460bd35b2f417caf40cd5b312693f7d46",
   "serving_sha256": "ce05a6a810007372c6de0f4c56dc94f652e1d28f2eeddf1878b64a170b4c48c2",
   "source_model_tree_sha256": "bd8028b5dbc6ab2cd2ab9a6019346de7b25536f50022617194bcb8503d4467ad",
-  "staged_at": "2026-09-11T14:57:19+00:00",
+  "retirement": {
+    "parent_revision": "2e1d6d0bef0e3a1a801496fac1fbb9c0d87effd4",
+    "predecessor_manifest_sha256": "dfb9d18a795c431ea75dd9fde14509cd4e12e8be962ebc5b591e7b069c099999",
+    "files": {
+      "model.directml.onnx": {
+        "sha256": "05abdc03628b599712a993381919aef1e22fcc5ad6384608abc2e069c402e369",
+        "size": 2242802
+      },
+      "serving.directml.json": {
+        "sha256": "7a43e53663efc6e82db9b29a13fa3775a35d73bd6731e1163601f8f36748925f",
+        "size": 711
+      }
+    }
+  },
+  "staged_at": "2026-09-29T03:04:18+00:00",
   "files": {
     "config.json": {
       "sha256": "a897acffc99f97238bb5bc0e1bccd6b7fe8fdfebe1450d4db3a9ab026cce6795",
       "size": 2020,
       "binding": "export receipt (file hash)"
     },
-    "model.directml.onnx": {
-      "sha256": "05abdc03628b599712a993381919aef1e22fcc5ad6384608abc2e069c402e369",
-      "size": 2242802,
-      "binding": "export receipt (file hash)"
-    },
     "model.onnx": {
       "sha256": "7df4a3b3cd89d216c016abd7d169f0fd215e9f79c06a2ffed57bac1abf556073",
       "size": 2231565,
@@ -34,9 +43,9 @@
       "size": 299270144,
       "binding": "export receipt (file hash)"
     },
-    "serving.directml.json": {
-      "sha256": "7a43e53663efc6e82db9b29a13fa3775a35d73bd6731e1163601f8f36748925f",
-      "size": 711,
+    "model.vulkan.onnx": {
+      "sha256": "cf27725a15982be3a0c329aec48486e5e048de5b4d3028f0cf88915eafa79b40",
+      "size": 2243288,
       "binding": "export receipt (file hash)"
     },
     "serving.json": {
@@ -44,6 +53,11 @@
       "size": 704,
       "binding": "export receipt (file hash)"
     },
+    "serving.vulkan.json": {
+      "sha256": "908e092f9c2197fe1028ece2aa8aa142dc088528dacc7237b25c81bfdd44a837",
+      "size": 946,
+      "binding": "export receipt (file hash)"
+    },
     "tokenizer.json": {
       "sha256": "6c8aaa9a542084f2457eab775d4eeb51f92a70c0fd9de28d5edb0ddec3c08d30",
       "size": 3583228,
@@ -54,10 +68,10 @@
       "size": 488,
       "binding": "export receipt (file hash)"
     },
-    "README.md": {
-      "sha256": "19b97e33bddbcb587446bbbb164cd091fc2cb534911cf0aafb0efc8013bda68c",
-      "size": 11806,
-      "binding": "packaging record"
+    "vulkan-derivation.json": {
+      "sha256": "e3b08005807fb6b9974d327fa8db1d90df317109879762be55397d8a4c42baa3",
+      "size": 972,
+      "binding": "export receipt (file hash)"
     },
     "LICENSE": {
       "sha256": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
@@ -70,9 +84,65 @@
       "binding": "packaging record"
     },
     "MODIFICATIONS.md": {
-      "sha256": "1f8c677387fb72ef31b7a107e866868a13821b6282debebc6f8150eac9866b18",
-      "size": 1958,
+      "sha256": "18e06084d4a6e34b6c7a976da739cb8b28383f7132552a89c2d4d0f0f7189257",
+      "size": 2164,
+      "binding": "packaging record"
+    },
+    "evaluation/README.md": {
+      "sha256": "3215e6b0e302fdecd86ef1457e46e7d033fa18fb1ebcbd0dcb47c2b8abe7c174",
+      "size": 8825,
+      "binding": "packaging record"
+    },
+    "evaluation/metrics.py": {
+      "sha256": "7c25f29a03e0f61d5f0e59f7781269489f80512ae6862920f91a2d333c058bc4",
+      "size": 11135,
+      "binding": "packaging record"
+    },
+    "evaluation/reranker.json": {
+      "sha256": "20b5760a462c3c2d2281607f2c5045d3f60210123a0c11c8b2d92580c3cccd3e",
+      "size": 1145459,
       "binding": "packaging record"
+    },
+    "evaluation/reranker-summary.json": {
+      "sha256": "05face1266e65d5a97a3053eed857c190fb32119581e395339770f3697bd5287",
+      "size": 5939,
+      "binding": "packaging record"
+    },
+    "evaluation/serving.json": {
+      "sha256": "0c017dbd0e66734c465d617eef24b2affb2670a96f7fd5b2c001514c3cc1381a",
+      "size": 500118,
+      "binding": "packaging record"
+    },
+    "evaluation/serving-summary.json": {
+      "sha256": "3a7c6a2c96495917e65a7dabde9c02c7c457b5416df51cbb58bb7267d5c74fda",
+      "size": 3162,
+      "binding": "packaging record"
+    },
+    "evaluation/serving-qualification.json": {
+      "sha256": "bf7deda3f25306d15454798ad4c8e785c25fba970fc774efbe557225fc5d42e0",
+      "size": 4667,
+      "binding": "packaging record"
+    },
+    "evaluation/figures.py": {
+      "sha256": "9944765ee42148f2bb135f428ee1dacb822af0ca5c77be9f82b507eb8da236d4",
+      "size": 4097,
+      "binding": "packaging record"
+    },
+    "evaluation/svg_figures.py": {
+      "sha256": "c68b7e26e072e3875f111b592f8031322261272fcf285ae2a3edbe378e70740d",
+      "size": 21536,
+      "binding": "packaging record"
+    },
+    "figures/ettin-comparison.svg": {
+      "sha256": "5d34b26e75afd868632cd1bca16081cdd6d28a60f85bbdc8592dffd80cea6814",
+      "size": 15067,
+      "binding": "packaging record"
+    },
+    "README.md": {
+      "sha256": "d81407ff4ec675405036dcd2179f825cc2dade7595ba55791a7d886d4ff41a89",
+      "size": 7628,
+      "binding": "model card with upload-relative links",
+      "source_sha256": "d1fe7d1b266f383ce8248e0682f4d22bb3cccb64d7076175d88f3dfa5351bda0"
     }
   }
 }

serving.directml.json → serving.vulkan.json

@@ -6,7 +6,7 @@
     "max_batch": 8,
     "max_sequence": 1153
   },
-  "graph": "model.directml.onnx",
+  "graph": "model.vulkan.onnx",
   "inputs": [
     "input_ids",
     "attention_mask"
@@ -14,12 +14,21 @@
   "output": "scores",
   "precision": "float32",
   "provider_chain": [
-    "DmlExecutionProvider",
+    "WebGpuExecutionProvider",
     "CPUExecutionProvider"
   ],
+  "provider_options": {
+    "dawnBackendType": "Vulkan",
+    "enableInt64": "1",
+    "storageBufferCacheMode": "lazyRelease"
+  },
   "role": "cross-encoder-reranker",
+  "runtime_requirements": [
+    "onnxruntime==1.24.4",
+    "onnxruntime-ep-webgpu==0.4.0"
+  ],
   "schema": "daecore.retrieval-onnx-serving.v1",
-  "sha256": "b83ffa3a44f19d145ecb87e36a56da290b25def03db267e33c1c57a7a46dda76",
+  "sha256": "bdace46b267a6fed5c084fde8798ed60c199e38c59553090d7f44618a65e2ca3",
   "source_model_tree_sha256": "bd8028b5dbc6ab2cd2ab9a6019346de7b25536f50022617194bcb8503d4467ad",
   "torch_required_at_runtime": false
 }

vulkan-derivation.json

@@ -0,0 +1,33 @@
+{
+  "artifacts": [
+    {
+      "bytes": 2243288,
+      "path": "model.vulkan.onnx",
+      "sha256": "cf27725a15982be3a0c329aec48486e5e048de5b4d3028f0cf88915eafa79b40"
+    },
+    {
+      "bytes": 299270144,
+      "path": "model.onnx.data",
+      "sha256": "78ddab681c2f506d62bbcd475fec5c10c62d867ed60d97a0170828efaff5bdde"
+    }
+  ],
+  "changes": [
+    {
+      "dtype": 9,
+      "name": "node_GatherND_32",
+      "operation": "GatherND"
+    },
+    {
+      "dtype": 7,
+      "name": "node_abs_1",
+      "operation": "Abs"
+    }
+  ],
+  "max_sequence": 1153,
+  "role": "ettin",
+  "schema": "daecore.vulkan-mask-derivation.v1",
+  "source_derivation_sha256": "d874e9b6d92ae885733ae23d33e5612596af1f11c2bee725bf33649b8884357a",
+  "source_graph_sha256": "05abdc03628b599712a993381919aef1e22fcc5ad6384608abc2e069c402e369",
+  "source_weights_sha256": "78ddab681c2f506d62bbcd475fec5c10c62d867ed60d97a0170828efaff5bdde",
+  "weights_unchanged": true
+}