Correct the headline to +59.7% on a clean, document-isolated split

Measured on a split that isolates at the source-document level. See the notice at the top of the card.

README.md

@@ -17,6 +17,42 @@ language:
 
 # Quanfire Legal Embedding — `embed-legal-en` (Supreme Court judgments)
 
+> ### ⚠️ Measurement correction — 2026-08-21
+>
+> **The headline this card used to carry (+76.2 % Recall@1) was measured on a
+> contaminated evaluation split. The corrected figure is +59.7 %.**
+>
+> The bug: the training filter dropped a pair only when its *positive* was in the
+> held-out set, and pairs drawn from a held-out judgment were never excluded at
+> all. Adjacent-passage pairs from the same judgment therefore sat on both sides
+> of the split — the model was partly trained on the documents it was scored on.
+>
+> The adapter was retrained and rescored on a split that isolates at the
+> **judgment-document** level (1,379 documents held out, 48,304 pairs dropped
+> from the pool). **Every in-distribution number in *Results* below is the clean
+> measurement.** This was the best-surviving of the three corrections we pushed
+> that day: the gain lost about a quarter of its magnitude and kept its
+> direction, its significance and its shape.
+>
+> Two things a reader deserves to know:
+>
+> 1. **The clean figures are for a retrain (`legal-indic-e1c`), not for the
+>    weight file currently in this repo.** The published weights saw the
+>    evaluation documents during training, so no honest score for *them* exists.
+>    Publishing the clean adapter as a new revision is pending.
+> 2. **The runs are not volume-matched** — 46,115 training pairs clean against
+>    92,419 before. Do not read the difference as a clean measure of "what the
+>    leak was worth".
+>
+> The **out-of-origin transfer** result is unaffected in kind — it is scored on a
+> different corpus, so the split bug cannot reach it — but the figures below
+> belong to the withdrawn adapter and have not been recomputed for the retrain.
+>
+> Fix: `without_held_out()` in
+> [quanfire-multilingual-embedding](https://github.com/quanfire-ai/quanfire-multilingual-embedding)
+> (commits `66470fe`, `6fe7e6b`, `3deaf8d`).
+
+
 A retrieval adapter for **English-language Indian Supreme Court judgment text**. It
 is a LoRA adaptation over a frozen
 [`intfloat/multilingual-e5-small`](https://huggingface.co/intfloat/multilingual-e5-small)
@@ -31,7 +67,7 @@ you both, measured.
 - **Framework & code:** [github.com/quanfire-ai/quanfire-multilingual-embedding](https://github.com/quanfire-ai/quanfire-multilingual-embedding) (Apache-2.0)
 - **PyPI:** `pip install quanfire-multilingual-embedding`
 - **Weights licence:** Apache-2.0 (see *Licence & provenance* — the training text is statutory public domain, so no share-alike floor applies)
-- **Internal run:** `legal-indic-e1` · base e5-small · rank 32 / alpha 64, LoRA on `query,value` · 589,824 adapter params · 1 epoch, lr 1e-4, batch 256 (bf16, CUDA)
+- **Internal run:** `legal-indic-e1` · base e5-small · rank 32 / alpha 64, LoRA on `query,value` · 589,824 adapter params · 1 epoch, lr 1e-4, batch 256 (bf16, CUDA) · the clean re-measure reported below is `legal-indic-e1c`: identical configuration, retrained on a document-isolated split
 
 ## What it is for
 
@@ -43,7 +79,7 @@ It embeds a query and a passage into the same 384-d space; cosine similarity ran
 
 | | Validated? |
 |---|---|
-| English Supreme Court **judgment** retrieval | ✅ **Yes** — strong, measured below |
+| English Supreme Court **judgment** retrieval | ✅ **Yes** — +59.7 % Recall@1 on a clean, document-isolated split; measured below |
 | **Statutory / bare-act** text, FAQs, other legal registers | ❌ **No** — transfer tested, came back flat (see below) |
 | **Non-English** legal text (Hindi, Tamil, …) | ❌ **No** — the model and its training data are English-only |
 
@@ -56,23 +92,29 @@ contracts, or non-English legal material, use the base model or a purpose-built
 **In-distribution — 2,000 held-out Supreme Court judgment pairs.** The published base
 is the only honest baseline; the adapter is scored on the *same* held-out pairs.
 
-| Metric | base e5-small | **embed-legal-en** | change |
-|---|---|---|---|
-| Recall@1 | 0.309 | **0.545** | **+76.2 %** |
-| Recall@5 | 0.494 | **0.772** | +56.3 % |
-| Recall@10 | 0.567 | **0.829** | +46.2 % |
-| MRR | 0.399 | **0.647** | +62.2 % |
-| nDCG@10 | 0.433 | **0.687** | +58.7 % |
+| Metric | base e5-small | **clean retrain (`legal-indic-e1c`)** | change | withdrawn figure |
+|---|---|---|---|---|
+| Recall@1 | 0.3090 | **0.4935** | **+59.7 %** | ~~0.545 / +76.2 %~~ |
+| Recall@5 | 0.4940 | **0.7180** | +45.3 % | ~~0.772~~ |
+| Recall@10 | 0.5670 | **0.7910** | +39.5 % | ~~0.829~~ |
+| MRR | 0.3988 | **0.5996** | +50.4 % | ~~0.647~~ |
+| nDCG@10 | 0.4326 | **0.6408** | +48.1 % | ~~0.687~~ |
+
+The base column is unchanged — the contamination only ever inflated the adapter, so the
+correction is confined to one column.
 
-The Recall@1 95 % confidence intervals are disjoint (base `[0.289, 0.330]` →
-adapter `[0.523, 0.566]`), so the gain is not sampling noise. It holds where it is
+The Recall@1 95 % confidence intervals are disjoint (base `[0.2891, 0.3296]` →
+clean retrain `[0.4716, 0.5154]`), so the gain is not sampling noise. It holds where it is
 hardest: on the **low-lexical-overlap** bucket (`<0.3`, pure-semantic matches, no
-shared words to lean on) Recall@1 more than doubles, **0.156 → 0.325**.
+shared words to lean on) Recall@1 rises **0.156 → 0.290** (+85.7 %; the withdrawn figure
+was 0.325).
 
 **Out-of-origin transfer — 1,578 English *statutory* adjacency pairs** (a different
 legal register: bare-act sections and regulatory FAQs, origin-walled from the
 judgment training corpus). This is the honest generalization test, and it is
-**flat**:
+**flat**. *These two rows are the withdrawn adapter's numbers and have not been recomputed
+for the clean retrain; the split bug could not reach them (different corpus), but a retrained
+adapter is a different adapter:*
 
 | Metric | base e5-small | embed-legal-en |
 |---|---|---|
@@ -81,9 +123,10 @@ judgment training corpus). This is the honest generalization test, and it is
 
 The instrument is informative, not degenerate — the base model has real signal on it
 (high-overlap Recall@5 0.76). The adapter simply does not improve statutory
-retrieval. **We publish this row on purpose:** the +76 % is a judgment-domain result,
-not a "legal English" result, and the difference is exactly what a buyer needs to
-know.
+retrieval. **We publish this row on purpose:** the in-distribution gain is a
+judgment-domain result, not a "legal English" result, and the difference is exactly what a
+buyer needs to know. For statutory text, use the purpose-built sibling
+[`embed-statute-en`](https://huggingface.co/quanfire-ai/embed-statute-en).
 
 ## Usage