README.md

---
language: en
license: apache-2.0
library_name: transformers
base_model: jhu-clsp/ettin-encoder-150m
model_name: cross-encoder-ettin-150m-DistillRankNET
source: https://github.com/xpmir/cross-encoders
paper: http://arxiv.org/abs/2603.03010
tags:
- cross-encoder
- sequence-classification
- tensorboard
datasets:
- msmarco
pipeline_tag: text-classification
---

# cross-encoder-ettin-150m-DistillRankNET

[![Paper](https://img.shields.io/badge/Paper-Arxiv-red)](http://arxiv.org/abs/2603.03010)
[![All Models](https://img.shields.io/badge/🤗%20Hugging%20Face%20Models-blue)](https://huggingface.co/collections/xpmir/reproducing-cross-encoders)
[![GitHub](https://img.shields.io/badge/GitHub-Code-blue)](https://github.com/xpmir/cross-encoders)

This model is a cross-encoder based on `jhu-clsp/ettin-encoder-150m`. It was trained on Ms-Marco using loss `distillRankNET` as part of a reproducibility paper for training cross encoders: "**[Reproducing and Comparing Distillation Techniques for Cross-Encoders](http://arxiv.org/abs/2603.03010)**", see the paper for more details.


### Contents
- [Model Description](#model-description)
- [Usage](#usage)
- [Evals](#evaluations)


## Model Description

This model is intended for **re-ranking** the top results returned by a retrieval system (like BM25, Bi-Encoders or SPLADE).

- **Training Data:** MS MARCO Passage
- **Language:** English
- **Loss** distillRankNET

Training can be easily reproduced using the assiciated repository. 
The exact training configuration used for this model is also detailed in [config.yaml](./config.yaml).

## Usage

Quick Start:
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-ettin-150m-DistillRankNET")
model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-150m-DistillRankNET")

features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")

model.eval()
with torch.no_grad():
    scores = model(**features).logits
    print(scores)
```

## Evaluations

We provide evaluations of this cross-encoder re-ranking the top `1000` documents retrieved by `naver/splade-v3-distilbert`.

| dataset            | RR@10     | nDCG@10   |
|:-------------------|:----------|:----------|
| msmarco_dev        | 36.30     | 42.93     |
| trec2019           | 96.98     | 75.59     |
| trec2020           | 93.83     | 72.27     |
| fever              | 80.10     | 79.82     |
| arguana            | 14.52     | 22.21     |
| climate_fever      | 27.00     | 19.76     |
| dbpedia            | 75.55     | 45.75     |
| fiqa               | 47.54     | 39.63     |
| hotpotqa           | 85.28     | 66.73     |
| nfcorpus           | 57.92     | 35.41     |
| nq                 | 53.64     | 58.68     |
| quora              | 75.29     | 77.46     |
| scidocs            | 28.04     | 15.73     |
| scifact            | 68.06     | 70.51     |
| touche             | 66.31     | 36.81     |
| trec_covid         | 96.50     | 77.81     |
| robust04           | 73.96     | 49.37     |
| lotte_writing      | 73.65     | 64.31     |
| lotte_recreation   | 62.02     | 56.42     |
| lotte_science      | 51.11     | 42.43     |
| lotte_technology   | 56.74     | 47.53     |
| lotte_lifestyle    | 72.68     | 63.49     |
| **Mean In Domain** | **75.70** | **63.60** |
| **BEIR 13**        | **59.67** | **49.72** |
| **LoTTE (OOD)**    | **65.03** | **53.92** |