mrbesher commited on
Commit
1aa9a4c
·
verified ·
1 Parent(s): 044870c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +88 -0
README.md CHANGED
@@ -91,3 +91,91 @@ model-index:
91
  - type: ndcg_at_10
92
  value: 97.86
93
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
91
  - type: ndcg_at_10
92
  value: 97.86
93
  ---
94
+
95
+ <p align="center">
96
+ <img src="assets/logo.webp" width="20%" alt="ModernBERT Reranker" />
97
+ </p>
98
+ <h1 align="center">ModernBERT-TR Reranker</h1>
99
+
100
+ A 150M-parameter Turkish cross-encoder reranker to score `(query, document)` relevance.
101
+
102
+ - Base model: [`ytu-ce-cosmos/modernbert-tr-base`](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-base).
103
+ - Distilled from `Qwen/Qwen3-Reranker-8B`.
104
+
105
+ ## Results
106
+
107
+ Reranking the top-100 of a first-stage retriever ([`ytu-ce-cosmos/modernbert-tr-embed`](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-embed)) at `max_seq=512`. The uplift (Δ) is the reranker's contribution.
108
+
109
+ | Task | First-stage NDCG@10 | + Reranker | Δ |
110
+ |---|---|---|---|
111
+ | ArguAnaTR | 37.01 | **54.75** | **+17.74** |
112
+ | SquadTRRetrieval | 75.94 | **90.11** | +14.17 |
113
+ | SciFactTR | 77.07 | **86.34** | +9.27 |
114
+ | TQuadRetrieval | 87.48 | **94.00** | +6.52 |
115
+ | CQADupstackGamingRetrievalTR | 56.44 | **61.10** | +4.66 |
116
+ | XQuADRetrieval | 95.03 | **97.86** | +2.83 |
117
+ | **Mean Δ** | | | **+9.20** |
118
+
119
+ ## How was this model trained?
120
+
121
+ Question answering and counter argument distillation of `Qwen3-Reranker-8B` relevance scores into the 150M cross-encoder over Turkish question answering / information retrieval data using [listwise KL](https://proceedings.mlr.press/v130/reddi21a.html).
122
+
123
+ ## Usage
124
+
125
+ ### transformers
126
+
127
+ ```python
128
+ import torch
129
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
130
+
131
+ tok = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker")
132
+ model = AutoModelForSequenceClassification.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker").eval()
133
+
134
+ query = "Türkiye'nin başkenti neresidir?"
135
+ docs = ["Ankara, Türkiye'nin başkentidir.", "İstanbul en kalabalık şehirdir."]
136
+ enc = tok([query] * len(docs), docs, padding=True, truncation="longest_first",
137
+ max_length=8192, return_tensors="pt")
138
+ with torch.no_grad():
139
+ scores = model(**enc).logits.squeeze(-1)
140
+ ranking = sorted(zip(docs, scores.tolist()), key=lambda x: x[1], reverse=True)
141
+ ```
142
+
143
+ ### sentence-transformers
144
+
145
+ ```python
146
+ from sentence_transformers import CrossEncoder
147
+ model = CrossEncoder("ytu-ce-cosmos/modernbert-tr-reranker")
148
+ scores = model.predict([(query, d) for d in docs])
149
+ ```
150
+
151
+ ### ONNX Runtime
152
+
153
+ The `onnx/` folder has the full graph, the output is the relevance logit:
154
+
155
+ ```python
156
+ import onnxruntime, numpy as np
157
+ sess = onnxruntime.InferenceSession("onnx/model.onnx")
158
+ feed = {k: v.numpy() for k, v in enc.items() if k in {i.name for i in sess.get_inputs()}}
159
+ logits = sess.run(None, feed)[0].squeeze(-1)
160
+ ```
161
+
162
+ ### Text Embeddings Inference (TEI)
163
+
164
+ ```bash
165
+ text-embeddings-router --model-id ytu-ce-cosmos/modernbert-tr-reranker --dtype float16
166
+ # POST /rerank {"query": "soru", "texts": ["aday 1", "aday 2"]}
167
+ ```
168
+
169
+ ## Training data
170
+
171
+ We used Turkish datasets msmarco-tr, squad-tr, fiqa-tr, nfcorpus-tr, quora-tr, scifact-tr for distillation by `Qwen3-Reranker-8B`, and Turkish counter-argument pairs from ArguAna machine-translated with TranslateGemma-27B. All training data was text-hash chceked against every MTEB(Turkish) test split.
172
+
173
+ ## Limitations
174
+
175
+ - Reported NDCG is rerank-of-top-100 over a first-stage retriever; absolute scores depend on that first stage.
176
+ - int8 ONNX reorders scores meaningfully lossy for a reranker; use fp32 for quality-sensitive ranking.
177
+ - Due to the lack of long form data in our training, the model's performance may degrade on long context input.
178
+
179
+ ## License & attribution
180
+
181
+ - License: `apache-2.0`.