README.md

---
language:
- tr
license: apache-2.0
library_name: transformers
base_model: ytu-ce-cosmos/modernbert-tr-base
pipeline_tag: text-ranking
tags:
- sentence-transformers
- text-embeddings-inference
- transformers.js
- reranker
- cross-encoder
- modernbert
- onnx
model-index:
- name: modernbert-tr-reranker
  results:
  - task:
      type: Retrieval
      name: ArguAnaTR
    dataset:
      type: trmteb/arguana-tr
      name: MTEB ArguAnaTR
      config: default
      split: test
      revision: main
    metrics:
    - type: ndcg_at_10
      value: 54.75
  - task:
      type: Retrieval
      name: CQADupstackGamingRetrievalTR
    dataset:
      type: trmteb/cqadupstack-gaming-tr
      name: MTEB CQADupstackGamingRetrievalTR
      config: default
      split: test
      revision: main
    metrics:
    - type: ndcg_at_10
      value: 61.10
  - task:
      type: Retrieval
      name: SciFactTR
    dataset:
      type: trmteb/scifact-tr
      name: MTEB SciFactTR
      config: default
      split: test
      revision: main
    metrics:
    - type: ndcg_at_10
      value: 86.34
  - task:
      type: Retrieval
      name: SquadTRRetrieval
    dataset:
      type: trmteb/squad-tr
      name: MTEB SquadTRRetrieval
      config: default
      split: test
      revision: main
    metrics:
    - type: ndcg_at_10
      value: 90.11
  - task:
      type: Retrieval
      name: TQuadRetrieval
    dataset:
      type: trmteb/tquad
      name: MTEB TQuadRetrieval
      config: default
      split: test
      revision: main
    metrics:
    - type: ndcg_at_10
      value: 94.00
  - task:
      type: Retrieval
      name: XQuADRetrieval
    dataset:
      type: google/xquad
      name: MTEB XQuADRetrieval
      config: default
      split: validation
      revision: 51adfef1c1287aab1d2d91b5bead9bcfb9c68583
    metrics:
    - type: ndcg_at_10
      value: 97.86
---

<p align="center">
  <img src="assets/logo.webp" width="20%" alt="ModernBERT Reranker" />
</p>
<h1 align="center">ModernBERT-TR Reranker</h1>

A 150M-parameter Turkish cross-encoder reranker to score `(query, document)` relevance.

- Base model: [`ytu-ce-cosmos/modernbert-tr-base`](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-base).
- Distilled from `Qwen/Qwen3-Reranker-8B`.

## Results

Reranking the top-100 of a first-stage retriever ([`ytu-ce-cosmos/modernbert-tr-embed`](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-embed)) at `max_seq=512`. The uplift (Δ) is the reranker's contribution.

| Task | First-stage NDCG@10 | + Reranker | Δ |
|---|---|---|---|
| ArguAnaTR | 37.01 | **54.75** | **+17.74** |
| SquadTRRetrieval | 75.94 | **90.11** | +14.17 |
| SciFactTR | 77.07 | **86.34** | +9.27 |
| TQuadRetrieval | 87.48 | **94.00** | +6.52 |
| CQADupstackGamingRetrievalTR | 56.44 | **61.10** | +4.66 |
| XQuADRetrieval | 95.03 | **97.86** | +2.83 |
| **Mean Δ** | | | **+9.20** |

## How was this model trained?

Question answering and counter argument distillation of `Qwen3-Reranker-8B` relevance scores into the 150M cross-encoder over Turkish question answering / information retrieval data using [listwise KL](https://proceedings.mlr.press/v130/reddi21a.html).

## Usage

### transformers

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker")
model = AutoModelForSequenceClassification.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker").eval()

query = "Türkiye'nin başkenti neresidir?"
docs = ["Ankara, Türkiye'nin başkentidir.", "İstanbul en kalabalık şehirdir."]
enc = tok([query] * len(docs), docs, padding=True, truncation="longest_first",
          max_length=8192, return_tensors="pt")
with torch.no_grad():
    scores = model(**enc).logits.squeeze(-1)
ranking = sorted(zip(docs, scores.tolist()), key=lambda x: x[1], reverse=True)
```

### sentence-transformers

```python
from sentence_transformers import CrossEncoder
model = CrossEncoder("ytu-ce-cosmos/modernbert-tr-reranker")
scores = model.predict([(query, d) for d in docs])
```

### ONNX Runtime

The `onnx/` folder has the full graph, the output is the relevance logit:

```python
import onnxruntime, numpy as np
sess = onnxruntime.InferenceSession("onnx/model.onnx")
feed = {k: v.numpy() for k, v in enc.items() if k in {i.name for i in sess.get_inputs()}}
logits = sess.run(None, feed)[0].squeeze(-1)
```

### Text Embeddings Inference (TEI)

```bash
text-embeddings-router --model-id ytu-ce-cosmos/modernbert-tr-reranker --dtype float16
# POST /rerank  {"query": "soru", "texts": ["aday 1", "aday 2"]}
```

## Training data

We used Turkish datasets msmarco-tr, squad-tr, fiqa-tr, nfcorpus-tr, quora-tr, scifact-tr for distillation by `Qwen3-Reranker-8B`, and Turkish counter-argument pairs from ArguAna machine-translated with TranslateGemma-27B. All training data was text-hash chceked against every MTEB(Turkish) test split.

## Limitations

- Reported NDCG is rerank-of-top-100 over a first-stage retriever; absolute scores depend on that first stage.
- int8 ONNX reorders scores meaningfully lossy for a reranker; use fp32 for quality-sensitive ranking.
- Due to the lack of long form data in our training, the model's performance may degrade on long context input.

## License & attribution

- License: `apache-2.0`.