narcolepticchicken/legalbenchrag-cuad-topone-reranker

LegalBenchRAG CUAD Top-One Reranker

This is a legal retrieval reranker fine-tuned from cross-encoder/ms-marco-MiniLM-L6-v2 on the public orgrctera/legalbenchrag_cuad retrieval dataset.

The model scores (question, passage) pairs and is intended to rerank candidate legal/contract passages. It does not generate answers and does not perform first-stage retrieval by itself.

Source Data and Base Model

• Base model: cross-encoder/ms-marco-MiniLM-L6-v2
• Base model license: Apache-2.0
• Training/evaluation dataset: orgrctera/legalbenchrag_cuad
• Dataset visibility: public, non-gated
• Dataset license listed on Hugging Face: CC-BY-4.0
• Dataset revision used: 0729087702c19847e72c648e8c22f859295a3e0c

Training Recipe

The promoted checkpoint is the best local pilot run from a controlled top-one listwise ablation:

• Run ID: 10_topone_listwise_pilot_A_temp2p0_20260623
• Starting checkpoint: local incumbent from 10_legalbenchrag_hard_negative_reranker_pilot_20260623T1p0p8_checkpoint_selection
• Candidate list size: 10
• Objective: query-averaged top-one listwise loss
• Temperature: 2.0
• Learning rate: 1e-5
• Steps: 80
• Batch source: clean incumbent training lists only
• Checkpoint selection: train-only inner-validation MRR@10, then Hits@1, then Recall@5

The loss was:

logsumexp(all_candidate_scores / temperature)
- logsumexp(positive_candidate_scores / temperature)

No sigmoid was applied before the listwise loss.

Evaluation

Evaluation used the fixed LegalBenchRAG CUAD pilot split and candidate construction from the local labpack. The final model improved over the previous local incumbent but did not meet the earlier stretch target of MRR@10 > 0.85.

Model	MRR@10	Recall@5	Hits@1	Top1 / 400	Top5 / 400	Over10
Original frozen baseline	0.6448	0.7700	n/a	n/a	n/a	n/a
Previous local incumbent	0.7909	0.9500	0.6875	275	380	8
This checkpoint	0.8288	0.9625	0.7350	294	385	1

Paired bootstrap versus the previous local incumbent:

• Mean MRR delta: +0.0379285714
• 95% CI: [+0.0188350446, +0.0577749752]
• Positive-delta bootstrap share: 1.0

Rank distribution for this checkpoint:

{
  "1": 294,
  "2": 45,
  "3": 24,
  "4": 13,
  "5": 9,
  "6": 6,
  "7": 1,
  "8": 4,
  "9": 3,
  "25": 1
}

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("narcolepticchicken/legalbenchrag-cuad-topone-reranker")

pairs = [
    (
        "Consider the contract; What is the governing law?",
        "This Agreement shall be governed by and construed in accordance with the laws of ..."
    ),
    (
        "Consider the contract; What is the governing law?",
        "The parties shall provide written notice within thirty days ..."
    ),
]

scores = model.predict(pairs)

Higher scores indicate passages that the model ranks as more likely to answer the query.

Intended Use

Use this model as a second-stage reranker for English contract/legal RAG systems, especially CUAD-like clause retrieval. It expects candidate passages from a separate retriever and ranks them for relevance to the query.

Limitations

• This is a reranker, not a legal reasoning or answer-generation model.
• It was evaluated locally on a fixed pilot split; performance may not transfer to other legal corpora without validation.
• The model improves MRR@10 versus the local incumbent, but it did not hit the earlier experimental stretch goal of MRR@10 > 0.85.
• Legal outputs should be reviewed by qualified humans before operational use.

Artifacts Included

This repository includes:

• model.safetensors and Sentence Transformers config/tokenizer files
• metrics.json
• rank_distribution.json
• train_metrics.json
• run_manifest.json

These local artifacts document the promoted run and evaluation evidence.