Maestro-Retriever-Llama-3.1-8B

An instruction-following dense retriever (LoRA on Llama-3.1-8B) whose instruction-following is verified to be genuine composition — not lexical keyword gaming.

Most "instruction-following" retrievers are scored by FollowIR's p-MRR, a metric that a 200-line bag-of-words keyword rule can top without understanding a single instruction. This model is different: its following ability is validated by a placebo control (CADIF), which scores the real instruction against a word-order-shuffled placebo of the same words. A model that only matches vocabulary scores the two identically (Δ_comp = 0); a model that actually composes the instruction scores higher on the real one.

Maestro clears that control decisively — Δ_comp = +0.118 (95% CI [0.076, 0.160]) on the 104-query FollowIR census — above shipped Promptriever-Llama3.1 (+0.092). In other words, it follows instructions by understanding their composition, not by keyword overlap.

What it is

• A LoRA adapter for NousResearch/Meta-Llama-3.1-8B, used as a dense (bi-encoder) retriever in the RepLLaMA / Promptriever style: last-token (EOS) pooling, L2-normalized embeddings, query: / passage: prefixes; the instruction is appended to the query.
• Trained with a contrastive (InfoNCE) objective on samaya-ai/msmarco-w-instructions (instruction-bearing MS MARCO), with the full 128-in-batch-negative recipe and the instance-level instruction-negatives that teach following.

Why it's different

• Verified instruction-following, not gamed. Validated with a de-confounded placebo control (Δ_comp), so the following signal is composition, not lexical expansion. Two independent newswire collections agree (robust04 +0.131, news21 +0.130).
• Llama-3.1-8B backbone — a strong, widely-supported open base; drop-in for anyone already running Llama-3.1.
• Compact to ship — a LoRA adapter (~tens of MB), merged onto the base at load time.

Intended use

Instruction-following text retrieval and reranking: given a query plus a natural-language instruction that narrows relevance (e.g. "…only papers that propose a new method, not surveys"), rank passages by how well they satisfy the composed query+instruction.

Usage

import torch
from transformers import AutoTokenizer, AutoModel
from peft import PeftModel

BASE = "NousResearch/Meta-Llama-3.1-8B"
ADAPTER = "tardellirs/Maestro-Retriever-Llama-3.1-8B"

tok = AutoTokenizer.from_pretrained(BASE)
tok.pad_token = tok.eos_token; tok.padding_side = "right"
model = PeftModel.from_pretrained(
    AutoModel.from_pretrained(BASE, torch_dtype=torch.bfloat16, attn_implementation="sdpa"),
    ADAPTER).merge_and_unload().cuda().eval()

def embed(texts):
    enc = tok(texts, max_length=511, truncation=True, return_attention_mask=False)
    enc["input_ids"] = [ids + [tok.eos_token_id] for ids in enc["input_ids"]]
    enc = tok.pad(enc, padding=True, return_attention_mask=True, return_tensors="pt").to("cuda")
    with torch.no_grad():
        h = model(**enc).last_hidden_state
    idx = enc["attention_mask"].sum(1) - 1          # EOS position
    reps = h[torch.arange(h.shape[0]), idx]
    return torch.nn.functional.normalize(reps.float(), p=2, dim=-1)

query = "best pizza in rome? I want only places open after midnight."
q = embed([f"query:  {query}"])
docs = ["passage:  Da Enzo closes at 23:00 ...", "passage:  Bonci is open until 2am ..."]
scores = q @ embed(docs).T

(Prefixes are two spaces after the colon, matching mteb's RepLLaMA convention.)

Evaluation

• CADIF Δ_comp (composition-following, placebo-controlled), FollowIR census (104 queries, pool 1000): +0.118, 95% CI [0.076, 0.160]; randomization p ≈ 1e-4; TOST rejects equivalence to "no composition"; Cohen d_z 0.53; power ≈ 1.0. Per collection: robust04 +0.131 (n=52), news21 +0.130 (n=32), core17 +0.062 (n=20). BM25 lexical floor = 0.000.
• General IR benchmark numbers (BEIR nDCG, MS MARCO MRR) are not yet measured — this is a research checkpoint focused on validated instruction-following; broad-benchmark results are forthcoming.

Training

Base NousResearch/Meta-Llama-3.1-8B; LoRA r=32, α=64, targets = q,k,v,o,gate,up,down; LR 1e-4, temperature 0.01, train_group_size 16 (first 3 = instruction-negatives), bf16, gradient checkpointing; 128 in-batch negatives via 4-GPU DDP on 4×H200; ~1 epoch over a 100k instruction-bearing MS MARCO subsample. Data: samaya-ai/msmarco-w-instructions (MS MARCO, non-commercial research use).

Status & limitations

Research checkpoint (~1 epoch). Validated on instruction-following composition (Δ_comp), English newswire (FollowIR). Not yet tuned/benchmarked for general-purpose retrieval; not evaluated for non-English or safety-sensitive use. Inherits Llama 3.1 and MS MARCO licensing/usage terms.

Citation

The de-confounded evaluation (CADIF: a placebo-controlled assay for instruction-following retrieval) is described in a forthcoming paper; citation to follow.