A 0.6B model at #2 on MTEB(Law)

We shipped a small legal embedding model that lands #2 on MTEB(Law) — second only to a 7–8B specialist, and ahead of every 4B, 8B, and 14B open retriever on the board. It's 0.6B parameters, fine-tuned on a single GPU, on a clean commercial license.

We had a target to hit, and two things in short supply getting there: time and compute. This is the recipe we built to hit it under those constraints. With more of either you'd make different calls — this is just what worked for us. Below is each step and why it's in the recipe.

The recipe:

1. Define "better" as a benchmark, and measure every change against the base model.
2. Fine-tune a small base with LoRA.
3. Mine hard negatives by gradient alignment, not embedding distance.
4. Read the training and eval data directly to find what's missing.
5. Fix catastrophic forgetting with a weight-space merge (WiSE-FT).
6. Soup two variants, then merge.

1. Define "better," then measure every change against the base

Pick the benchmark that defines success and turn it into one number you can watch. Run it on your untouched base model. That score is your floor, and it's the whole discipline of the project: from here, every change — a new way to mine negatives, more training data, a merge — is accepted or rejected by one rule. Does it beat the floor on that benchmark? If not, it's out, however good the idea sounded.

We used MTEB(Law) v1 as the benchmark and Qwen/Qwen3-Embedding-0.6B as the base. Score the base first, write the number down, and hold every later result against it.

floor = evaluate(base_model, benchmark="MTEB(Law) v1")     # your baseline number
# then, for every change you try:
score = evaluate(changed_model, benchmark="MTEB(Law) v1")
keep  = score > floor                                       # otherwise throw it out

2. Fine-tune a small base with LoRA

The setup is standard, and worth naming so you can look each piece up:

• LoRA (low-rank adaptation) — train small low-rank adapter matrices instead of all the weights, so a fine-tune is cheap and the base stays frozen.
• Last-token pooling — use the hidden state of the final token as the sentence embedding, the standard for decoder-based embedders like E5-mistral and Qwen3-Embedding.
• InfoNCE — the contrastive loss: pull a query toward its correct document and push it away from the other documents in the batch.

We kept the base at 0.6B to keep each training run fast and cheap, which is what our timeline needed. The steps that actually moved the score are the negatives and the merge below.

3. Mine hard negatives by gradient alignment

A hard negative is a document that looks relevant to a query but isn't the right answer — close enough to fool the model, which is what makes it worth training against. For the query "how tall is the Eiffel Tower?", the gold answer is the passage giving its height; an easy negative is a cake recipe (obviously unrelated); a hard negative is a passage about when the Eiffel Tower was built — same entity, same topic, wrong answer. Easy negatives teach the model nothing. Hard negatives are where it learns the fine distinctions.

The textbook way to find them is to take the most-similar document that isn't the gold. In legal retrieval that backfires: the nearest non-gold document is often a near-duplicate of the gold, so training to push it away also pushes away the answer — a false negative wearing a hard-negative costume.

A better selector asks a different question: of all the candidate negatives, which one's training gradient points the same direction as the gradient that would close the gap on the eval? That's gradient-based data selection — the idea behind LESS (Xia et al., ICML 2024), which picks training examples by the similarity of their gradient to a target task's gradient. We adapt it to negatives: isolate each candidate's contribution by subtracting the shared query→gold pull, then score its gradient against the eval-gap gradient.

# does this candidate negative move weights toward closing the gap, or away?
g_pos    = grad(infonce(q, gold, in_batch_only))            # shared query->gold pull
g_target = grad(infonce(q, gold, real_confusions)) - g_pos  # what the eval gap needs
g_cand   = grad(infonce(q, gold, candidate_negs))  - g_pos  # what this strategy does
alignment = cosine(g_cand, g_target)                        # > 0 helps, < 0 fights

Scored this way, the ordering comes out backwards from the usual advice: gradient-selected negatives (+0.093) beat random (+0.073), and the embedding-nearest "hardest" negative (+0.018) is barely above zero. Perturbing the gold to synthesize a negative — swap the entity, the date, a number — scores below random, because a 99%-identical perturbation is effectively the gold.

One filter does most of the work. A few documents are near-neighbors of almost every query (in legal text it's a handful of omnibus statutes). They aren't hard negatives, they're gravity wells, and they contaminate the mined set. Drop them and the gradient alignment jumps from 0.54 to 0.98:

# hub-filter: remove documents that appear as a top neighbor for too many queries
hub = {d for d, freq in neighbor_counts.items() if freq > 0.05 * n_queries}
negatives = [d for d in mined if d not in hub]

So: real, in-domain, retrieval-confusable documents, hub-filtered. Not the nearest one, and never a synthetic one.

4. Read the data directly, not just the aggregates

Aggregate scores tell you a task is stuck; they never tell you why. For that you have to open the actual queries, the actual gold documents, and the actual things your model retrieved instead — row by row. Almost every real fix we found came from reading data, not from staring at a metric.

Our statute-retrieval task was flat, and the aggregate suggested the obvious move: "train deeper on the main code." Reading the eval corpus said the opposite. The task retrieves over ~200 statutes spanning the constitution, procedure codes, evidence, arbitration, contract law — a whole shelf — while our training pool was one code, deep. The model had never seen most of the language it was being tested on. That's a coverage gap, and no amount of extra depth on the one code fixes it: a model can't rank language it was never trained on.

The fix was breadth from independent public-domain text — hundreds of additional acts, split into (heading → body) pairs. It lifted 7 of the 8 tasks; the statute task went 0.722 → 0.783.

When you add training data that overlaps your eval domain, add a leak guard: a check that drops any training example too similar to something in the eval set, so you teach the model the domain instead of memorizing the test. Exact-hash dedup isn't enough — transcriptions differ character by character and slip through. We used shingle containment: break each text into overlapping 8-word windows (shingles) and drop any training doc that shares more than 30% of its shingles with an eval document.

# drop any training statute whose 8-word shingle overlap with the eval corpus is too high
if shingle_containment(candidate, eval_corpus) >= 0.30:
    drop(candidate)

5. Fix catastrophic forgetting with a weight-space merge (WiSE-FT)

The problem: you fine-tune the model, it gets better at the task you trained on — and worse at everything else. Tasks it used to handle, especially ones with no examples in your training set, degrade or break. This is catastrophic forgetting: optimizing hard for the in-distribution task pulls the weights away from the general-purpose ones the base had, so out-of-distribution performance collapses.

The first thing we tried was replay: mix a fraction of general, non-task data back into training, so the model keeps practicing the old skills while learning the new one. It works, but only as a trade — the more general data you replay to protect the old tasks, the less the model learns the new task. You're diluting the signal you fine-tuned for, and there's no ratio that gives you both.

WiSE-FT (Wortsman et al., CVPR 2022) fixes it a different way, and it's almost free. The insight: a fine-tune moves the weights, but the general capability you lost still lives in the base weights you started from — so instead of protecting it during training, you recover it after, by ensembling in weight space. Interpolate between base and fine-tuned weights, θ = (1−α)·base + α·fine-tuned, and sweep α. The original paper introduced this to keep target-task accuracy while restoring robustness under distribution shift; here it recovers the forgotten tasks while keeping the learned ones — no retraining, one knob.

Because a LoRA on a frozen base starts from a zero delta, the interpolation reduces to just scaling the LoRA B matrices by α:

# theta = (1-a)*base + a*ft. base delta is zero (lora_B inits to 0),
# so this reduces exactly to: scale lora_B by a.
for name, p in adapter.items():
    if "lora_B" in name:
        p *= alpha        # a=0 -> exactly the base; a=1 -> fully fine-tuned

Sweep α and eval. It recovers the forgotten tasks and keeps the learned ones in one non-dilutive move; the peak is almost never at α=1. This is why you can now train with low replay — max signal — and let α recover the general tasks afterward. One check makes the merge trustworthy: α=0 must reproduce the base model exactly, or the merge math is wrong and every other α is lying.

6. Soup two variants, then merge

A model soup (Wortsman et al., ICML 2022) is just the average of several fine-tuned models' weights — often better than any single one, at no extra inference cost. We train the model twice — once with the mined hard negatives, once with in-batch negatives only — average the two adapters, then run the α merge-back on the average:

soup = {k: sum(ckpt[k] for ckpt in ckpts) / len(ckpts) for k in ckpts[0]}
# then apply step 5 (the alpha sweep) to `soup`.

The reason it helps: the in-batch-only variant barely forgets, so the average is less damaged, so you can inject more fine-tune before forgetting bites — it rides the α curve further up. Include a low-forgetting member on purpose.

One boundary worth stating: negatives don't transfer across base sizes. Reusing the 0.6B-mined negatives on a 4B base inverted the gains, because a near-miss that's hard for a weak representation is correct-and-easy for a stronger one. Mine negatives in the geometry of the base you're actually training.

Where I used it

The model and the method are open: Hanno-Labs/dinghy-law-0.6b-v1 — a 0.6B legal retriever at #2 on MTEB(Law), on a clean commercial license.