README.md

---
license: apache-2.0
base_model: cross-encoder/ettin-reranker-150m-v1
pipeline_tag: text-ranking
tags:
  - ettin
  - modernbert
  - cross-encoder
  - reranker
  - fine-tuned
  - onnx
---

# Ettin-150M memory reranker v1

## Summary

This model reorders the fixed 50-passage pool that a hybrid first stage
produces for a memory query. It is a fine-tune of
[`cross-encoder/ettin-reranker-150m-v1`](https://huggingface.co/cross-encoder/ettin-reranker-150m-v1)
trained with a graded listwise objective on 144,100 unique query-passage pairs
across 2,882 queries.

On the 441 development queries whose fixed pool holds at least one useful
passage, it raised nDCG@10 from 0.562 for the fused first stage to 0.762 and
Hit@1 from 0.644 to 0.807. In the protected promotion read, on the same 865
answerable pools, it scored nDCG@10 0.728 against 0.595 for the previous
reranker, +0.133 with a one-sided 95% lower bound of +0.119.

All relevance grades are frontier-model judgments under a frozen protocol,
the corpus is mostly generated documents, and the per-query evidence behind
the development and gate reads was lost after the aggregates were recorded.
The limitations section spells out each.

## Model details

| Item | Value |
|---|---|
| Architecture | Ettin-150M (ModernBERT-family) cross-encoder with a trained scoring head; rank-16 LoRA+ adapter merged into the weights |
| Input | one query and one passage, 1,153 tokens total |
| Pool | a fixed 50-passage candidate set; the model reorders it and never adds to it |
| Output | one relevance score per pair, used for ordering |
| Serving precision | FP16 on CUDA; FP32 calculations on DirectML, sharing the same stored weights |
| Deployment state | CUDA release plus an approved DirectML graph; installation is a separate update |

## Intended use

Final ordering inside a local memory-retrieval pipeline: the dense retriever
and BM25 each retrieve, reciprocal-rank fusion merges them, and this model
scores the top 50. It cannot recover a passage the first stage missed; on the
full 553-query development panel the 112 queries with no useful passage in
their pool stay at zero before and after reranking.

## Training data

| Item | Value |
|---|---|
| Queries | 2,882 answerable, plus 30 explicit no-positive rows kept out of the ranking loss |
| Unique pairs | 144,100, every one seen; 184,448 list presentations |
| Unique passages | 35,006 |
| Grade counts 0 / 1 / 2 / 3 | 32,331 / 37,258 / 43,731 / 30,780 |
| Query mix | 1,318 anchor-conditioned, 1,270 unanchored, 294 lexical-style |
| Protected queries or families in gradients | 0 |

Grades come from final consensus ledgers, so every grade-1 label is usable
supervision rather than a masked unknown. The lexical-style queries (terse,
exact, pasted, or lightly misspelled) are there so the model learns the
lexical lane's cases as well as rich semantic questions.

## Recipe

Grade-gain ListNet over lists of 16 with an adjacent-grade RankNet auxiliary
at weight 0.2; LoRA+ at rank 16 and alpha 32 on all linear layers, learning
rate 5e-5 for the A matrices and the score head and 8e-4 for the B matrices;
2,882 optimizer updates; FP16. Merge and reload changed scores by 0.

## Evaluation

**Development panel.** 553 queries with every one of the 27,650 pool
passages graded; 441 have a useful passage in their pool.

| Stage | nDCG@10 | Hit@1 | Hit@5 | Hit@10 | Hit@20 | Hit@50 |
|---|---:|---:|---:|---:|---:|---:|
| fused first stage, 441 answerable | 0.5617 | 0.6440 | 0.8662 | 0.9206 | 0.9569 | 1.0000 |
| this model, 441 answerable | 0.7621 | 0.8073 | 0.9410 | 0.9615 | 0.9841 | 1.0000 |
| fused first stage, all 553 | 0.4479 | 0.5136 | 0.6908 | 0.7342 | 0.7631 | 0.7975 |
| this model, all 553 | 0.6078 | 0.6438 | 0.7505 | 0.7667 | 0.7848 | 0.7975 |

Pairwise accuracy on the answerable pools is 0.820. The all-query rows are the
honest end-to-end view; Hit@50 is unchanged by design.

**How it was chosen.** Against the consensus hard-label baseline (0.680
nDCG@10 on this panel), a Lambda-style objective lost, ListNet at width 8 tied,
and width 16 without a higher learning rate lost. The adjacent-grade term at
5e-5 reached 0.715. Restarting from the base over every paid pair gave 0.728
with pure ListNet, 0.731 with the auxiliary and standard LoRA, and 0.732 with
DoRA. LoRA+ at rank 16 reached 0.762 and was the only large optimization gain;
rank 32 tied it (+0.0006) and a further top-10 continuation lost 0.005.

**Protected gate, same-pool comparison.** On the 865 sealed queries whose
candidate pool holds a useful passage, this model scored nDCG@10 0.7280
against 0.5952 for the previous reranker, +0.1328 with a one-sided 95% lower
bound of +0.1191 against a registered 0.02 margin. The judge-noise band for
this comparison was 0.0036. The full pipeline reached Hit@1 0.7609 and nDCG@10
0.6692 on the 895 union-answerable queries, against 0.6078 and 0.4988 before.

## Runtime

The FP16 ONNX export removes CPU-hosted sequence operations from all 22 MLP
layers so the whole graph runs on the GPU. It is bounded-quality equivalent
rather than bit-exact to the Torch model: as recorded at qualification,
+0.0002 nDCG@10 and +0.0023 Hit@1 on the development pools, 4 of 441 top
choices and 15 of 441 top-10 memberships changed, and a 95th-percentile score
drift of 0.0098. A top-50 pool reranks in 0.664 s median and 1.032 s at the
95th percentile on an RTX 3060 Ti; the one-second target is advisory.

The export regenerated on 2026-09-04 reproduced the qualified graph byte for
byte. A parity recheck the same day had to use the training partition (2,882
queries, 144,100 pairs) because the qualification pools were lost, and it
fails the frozen policy on two checks: Hit@1 moved by −0.0003, one query,
and one passage moved 16 places against a limit of 4. All 10 top-1 changes
and all 73 top-10 membership changes on that panel occur between passages
whose FP16 scores are exactly tied in one run and one unit apart in the
other, so the tie-break decides the order; the largest score change, 0.08,
sits at rank 47. On that panel the export reranked a pool in 0.584 s median
and 0.912 s at the 95th percentile, and the Torch FP16 model in 0.532 s and
0.777 s.

### DirectML comparison

The DirectML package adds `model.directml.onnx` alongside the unchanged CUDA
`model.onnx`. Both read the same `model.onnx.data`; no second parameter file
or training framework is needed at runtime. DirectML uses full-precision
calculations because the half-precision graph had larger score differences.
CUDA keeps its faster half-precision graph. Each provider plans batches from
available memory and pair length, with measured ceilings of 16 rows for CUDA
and 8 for DirectML.

On a current 970-query replay (48,500 fixed-pool pairs), DirectML completed
without a failure: 2.231 seconds median, 3.247 seconds at the 95th percentile,
and 4.286 seconds maximum on an RTX 3060 Ti. No call exceeded ten seconds.
This panel retains its per-query evidence; 2,498 pairs are unjudged, and their
grades remain unknown. It is separate from the lost historical gate evidence.

Against the published CUDA graph, nDCG@10 changed by +0.000017. Eight first
choices and 36 top-ten sets changed. Both versions returned known-useful
evidence for 816 of 970 queries; DirectML returned 5,528 known-useful passages
versus 5,524. The unchanged calibration was applied for comparison only; its
DirectML registration is not implied by these results.

Two strict cross-precision checks fail and were explicitly accepted:
Hit@1 and Hit@5 each lose one query, and one passage moves five positions
against a limit of four. The affected useful passages remain in the returned
sets; the five-position move is rank 33 to 28, outside the returned three.
On the 482 fully judged pools, no Hit metric decreases.

Running the same FP32 graph on CUDA for all 165 queries with changed
deliveries or relevant ranking differences isolates the provider error:
8,250 pairs have a 95th-percentile score difference of 0.00000954 and a
maximum of 0.000149. One pair exceeds the diagnostic 0.0001 bound; one full
ordering changes and no first choice changes. The two Hit losses and the
five-position move also occur on CUDA FP32, identifying precision rather
than DirectML as their cause.

On the supported sequential build policy, concurrent embedding and recall
kept reranking below 3.2 seconds in three rounds. Eight queued recalls across
four threads completed within 9.8 seconds. Forcing all three GPU models to
remain resident on this 8 GB card exceeded the memory budget and failed the
ten-second limit; those forced-overload timings are not a passing result.
Worker-kill recovery completed for reranking, embedding, and classification.

## Limitations, ranked

1. **Model-judged relevance.** No human grades; session agreement 0.858 and
   kappa 0.682 on a 419-query audit. Scores measure agreement with that
   instrument.
2. **Generated corpus.** Mostly fictional document worlds chosen for breadth;
   behavior on other real corpora is unmeasured.
3. **Lost per-query evidence.** The development ledger, gate judgments, pool
   scores, and gate report were destroyed in a local storage incident after
   these aggregates were recorded. The numbers stand as recorded and cannot be
   re-audited; the recipe-search arms survive only as recorded results.
4. **One training run per arm.** Several selection margins (rank 16 versus 32,
   width 8 versus 16) are inside a plausible seed effect; compute set that limit.
5. **First-stage ceiling.** 112 of 553 development queries and 95 of 990 gate
   queries have no useful passage in any top-50 pool; ranks 51 to 200 were
   never judged.
6. **Provider limits.** CPU reranking is not qualified. DirectML has been
   measured on an NVIDIA RTX 3060 Ti, not AMD or Intel hardware, and its
   cross-precision policy exceptions above were accepted with disclosure.
   Hosts without a qualified reranker serve fusion order with a typed notice.
7. **The reproducible parity read fails the frozen policy.** The recorded
   development-panel pass cannot be re-run, and the 2026-09-04 recheck on the
   training partition fails two checks at exact FP16 ties. The operator
   accepted the recorded qualification with this disclosure rather than
   exporting an FP32 graph or amending the policy. The current FP32 provider
   comparison is separate evidence and does not erase that historical result.

## Reproducibility

- Model tree SHA-256 `bd8028b5dbc6ab2cd2ab9a6019346de7b25536f50022617194bcb8503d4467ad`;
  base revision `025501c4e0f9bbeb4c5b198318e0089ff061cc14`, tree
  `e0edfb38cc495f23507de50233d0965f4e72ad0872e70e6f4e8fc889d394ea01`.
- Training data manifest `286318992c584ae790a72a5a86b0e7d03d8860f8ae93c35f6cead4b08b4d395a`,
  schedule `e98d3cec2533b6edcded7c0b5017cba432e4af0e73fd68fca84997f934fa8f63`.
- Recipe id `final-listnet-adjacent-loraplus16-full-coverage-w16-lr5e5`;
  registry manifest `../model_registry/ettin-150m-memory-reranker-ft-v1.json`;
  experiment log `../history/README.md`.
- CUDA graph SHA-256 `7df4a3b3cd89d216c016abd7d169f0fd215e9f79c06a2ffed57bac1abf556073`.
- DirectML graph SHA-256 `05abdc03628b599712a993381919aef1e22fcc5ad6384608abc2e069c402e369`.
- Shared weights SHA-256 `78ddab681c2f506d62bbcd475fec5c10c62d867ed60d97a0170828efaff5bdde`.
- CUDA serving-contract SHA-256 `ce05a6a810007372c6de0f4c56dc94f652e1d28f2eeddf1878b64a170b4c48c2`.
- Build stack: Transformers 5.2.0, Sentence Transformers 5.5.1, PyTorch
  2.10.0; export opset 18.

## License

Apache 2.0, following the upstream model; the distribution includes the
license, a notice, and a modification disclosure.