Text Ranking
Transformers
ONNX
Safetensors
sentence-transformers
Transformers.js
Turkish
modernbert
text-classification
text-embeddings-inference
reranker
cross-encoder
Eval Results (legacy)
Instructions to use ytu-ce-cosmos/modernbert-tr-reranker with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ytu-ce-cosmos/modernbert-tr-reranker with Transformers:
# Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker") model = AutoModelForSequenceClassification.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker", device_map="auto") - sentence-transformers
How to use ytu-ce-cosmos/modernbert-tr-reranker with sentence-transformers:
from sentence_transformers import CrossEncoder model = CrossEncoder("ytu-ce-cosmos/modernbert-tr-reranker") query = "Which planet is known as the Red Planet?" passages = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", "Saturn, famous for its rings, is sometimes mistaken for the Red Planet." ] scores = model.predict([(query, passage) for passage in passages]) print(scores) - Transformers.js
How to use ytu-ce-cosmos/modernbert-tr-reranker with Transformers.js:
// npm i @huggingface/transformers import { pipeline } from '@huggingface/transformers'; // Allocate pipeline const pipe = await pipeline('text-ranking', 'ytu-ce-cosmos/modernbert-tr-reranker'); - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -91,3 +91,91 @@ model-index:
|
|
| 91 |
- type: ndcg_at_10
|
| 92 |
value: 97.86
|
| 93 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
- type: ndcg_at_10
|
| 92 |
value: 97.86
|
| 93 |
---
|
| 94 |
+
|
| 95 |
+
<p align="center">
|
| 96 |
+
<img src="assets/logo.webp" width="20%" alt="ModernBERT Reranker" />
|
| 97 |
+
</p>
|
| 98 |
+
<h1 align="center">ModernBERT-TR Reranker</h1>
|
| 99 |
+
|
| 100 |
+
A 150M-parameter Turkish cross-encoder reranker to score `(query, document)` relevance.
|
| 101 |
+
|
| 102 |
+
- Base model: [`ytu-ce-cosmos/modernbert-tr-base`](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-base).
|
| 103 |
+
- Distilled from `Qwen/Qwen3-Reranker-8B`.
|
| 104 |
+
|
| 105 |
+
## Results
|
| 106 |
+
|
| 107 |
+
Reranking the top-100 of a first-stage retriever ([`ytu-ce-cosmos/modernbert-tr-embed`](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-embed)) at `max_seq=512`. The uplift (Δ) is the reranker's contribution.
|
| 108 |
+
|
| 109 |
+
| Task | First-stage NDCG@10 | + Reranker | Δ |
|
| 110 |
+
|---|---|---|---|
|
| 111 |
+
| ArguAnaTR | 37.01 | **54.75** | **+17.74** |
|
| 112 |
+
| SquadTRRetrieval | 75.94 | **90.11** | +14.17 |
|
| 113 |
+
| SciFactTR | 77.07 | **86.34** | +9.27 |
|
| 114 |
+
| TQuadRetrieval | 87.48 | **94.00** | +6.52 |
|
| 115 |
+
| CQADupstackGamingRetrievalTR | 56.44 | **61.10** | +4.66 |
|
| 116 |
+
| XQuADRetrieval | 95.03 | **97.86** | +2.83 |
|
| 117 |
+
| **Mean Δ** | | | **+9.20** |
|
| 118 |
+
|
| 119 |
+
## How was this model trained?
|
| 120 |
+
|
| 121 |
+
Question answering and counter argument distillation of `Qwen3-Reranker-8B` relevance scores into the 150M cross-encoder over Turkish question answering / information retrieval data using [listwise KL](https://proceedings.mlr.press/v130/reddi21a.html).
|
| 122 |
+
|
| 123 |
+
## Usage
|
| 124 |
+
|
| 125 |
+
### transformers
|
| 126 |
+
|
| 127 |
+
```python
|
| 128 |
+
import torch
|
| 129 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 130 |
+
|
| 131 |
+
tok = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker")
|
| 132 |
+
model = AutoModelForSequenceClassification.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker").eval()
|
| 133 |
+
|
| 134 |
+
query = "Türkiye'nin başkenti neresidir?"
|
| 135 |
+
docs = ["Ankara, Türkiye'nin başkentidir.", "İstanbul en kalabalık şehirdir."]
|
| 136 |
+
enc = tok([query] * len(docs), docs, padding=True, truncation="longest_first",
|
| 137 |
+
max_length=8192, return_tensors="pt")
|
| 138 |
+
with torch.no_grad():
|
| 139 |
+
scores = model(**enc).logits.squeeze(-1)
|
| 140 |
+
ranking = sorted(zip(docs, scores.tolist()), key=lambda x: x[1], reverse=True)
|
| 141 |
+
```
|
| 142 |
+
|
| 143 |
+
### sentence-transformers
|
| 144 |
+
|
| 145 |
+
```python
|
| 146 |
+
from sentence_transformers import CrossEncoder
|
| 147 |
+
model = CrossEncoder("ytu-ce-cosmos/modernbert-tr-reranker")
|
| 148 |
+
scores = model.predict([(query, d) for d in docs])
|
| 149 |
+
```
|
| 150 |
+
|
| 151 |
+
### ONNX Runtime
|
| 152 |
+
|
| 153 |
+
The `onnx/` folder has the full graph, the output is the relevance logit:
|
| 154 |
+
|
| 155 |
+
```python
|
| 156 |
+
import onnxruntime, numpy as np
|
| 157 |
+
sess = onnxruntime.InferenceSession("onnx/model.onnx")
|
| 158 |
+
feed = {k: v.numpy() for k, v in enc.items() if k in {i.name for i in sess.get_inputs()}}
|
| 159 |
+
logits = sess.run(None, feed)[0].squeeze(-1)
|
| 160 |
+
```
|
| 161 |
+
|
| 162 |
+
### Text Embeddings Inference (TEI)
|
| 163 |
+
|
| 164 |
+
```bash
|
| 165 |
+
text-embeddings-router --model-id ytu-ce-cosmos/modernbert-tr-reranker --dtype float16
|
| 166 |
+
# POST /rerank {"query": "soru", "texts": ["aday 1", "aday 2"]}
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
## Training data
|
| 170 |
+
|
| 171 |
+
We used Turkish datasets msmarco-tr, squad-tr, fiqa-tr, nfcorpus-tr, quora-tr, scifact-tr for distillation by `Qwen3-Reranker-8B`, and Turkish counter-argument pairs from ArguAna machine-translated with TranslateGemma-27B. All training data was text-hash chceked against every MTEB(Turkish) test split.
|
| 172 |
+
|
| 173 |
+
## Limitations
|
| 174 |
+
|
| 175 |
+
- Reported NDCG is rerank-of-top-100 over a first-stage retriever; absolute scores depend on that first stage.
|
| 176 |
+
- int8 ONNX reorders scores meaningfully lossy for a reranker; use fp32 for quality-sensitive ranking.
|
| 177 |
+
- Due to the lack of long form data in our training, the model's performance may degrade on long context input.
|
| 178 |
+
|
| 179 |
+
## License & attribution
|
| 180 |
+
|
| 181 |
+
- License: `apache-2.0`.
|