MenteEAI
/

mentee-embed-v4

@@ -1,5 +1,7 @@
 ---
 license: apache-2.0
 language:
 - en
 - ar
@@ -10,31 +12,269 @@ tags:
 - retrieval
 - contrastive-learning
 - multilingual
 ---
-# mentee-embed-v4 (41M) — trilingual embeddings trained from scratch
-A compact **41M-parameter, 384-dim** text embedding model for **Arabic, English and Urdu**, trained entirely from scratch (no pretrained base) in two stages on a single consumer GPU.
-## Recipe
-1. **Stage A — Masked language modeling** on ~31M tokens across the three languages
-2. **Stage B — Contrastive fine-tuning** (symmetric InfoNCE, in-batch negatives + hard-negative column, temperature 0.05) on 2589852 triplets incl. **300K English↔Urdu parallel pairs (OPUS-100)**
-## Results (Protocol B — 15K-doc corpus-pool retrieval, higher is better)
-| 1 | paraphrase-multilingual-mpnet-base-v2 | MRR 0.829 · R@5 0.950 · R@100 0.997 | MRR 0.622 · R@5 0.757 · R@100 0.947 | MRR 0.534 · R@5 0.680 · R@100 0.903 | **0.662** |
-| 2 | paraphrase-multilingual-MiniLM-L12-v2 | MRR 0.817 · R@5 0.920 · R@100 0.993 | MRR 0.591 · R@5 0.710 · R@100 0.943 | MRR 0.469 · R@5 0.557 · R@100 0.847 | **0.626** |
-| 3 | mentee-embed-v4 (ours) ⭐ **ours** | MRR 0.695 · R@5 0.823 · R@100 0.970 | MRR 0.623 · R@5 0.727 · R@100 0.927 | MRR 0.336 · R@5 0.373 · R@100 0.643 | **0.551** |
-| 4 | all-MiniLM-L6-v2 | MRR 0.844 · R@5 0.967 · R@100 1.000 | MRR 0.100 · R@5 0.000 · R@100 0.010 | MRR 0.106 · R@5 0.007 · R@100 0.013 | **0.350** |
-Our model reaches **0.551** vs leader paraphrase-multilingual-mpnet-base-v2 at **0.662** on this suite.
-*Protocol details: queries ranked against the full pool; baselines run with their recommended usage (E5 query/passage prefixes applied).*
 ## Usage
 ```python
-from sentence_transformers import SentenceTransformer
-model = SentenceTransformer("<your-hf-user>/mentee-embed-v1")
-emb = model.encode(["مرحبا بالعالم", "hello world", "دنیا میں خوش آمدید"])
 ```
 ## Limitations
-NLI+parallel derived training data; small-pool protocol is easier than full MIRACL; expect open-domain gaps versus web-scale models.

 ---
 license: apache-2.0
+pipeline_tag: sentence-similarity
+library_name: transformers
 language:
 - en
 - ar
 - retrieval
 - contrastive-learning
 - multilingual
+- from-scratch
+datasets:
+- sentence-transformers/all-NLI
+- facebook/xnli
+- Helsinki-NLP/opus-100
+- sentence-transformers/msmarco-bm25
+- nlpai-lab/miracl-multilingual-triplets
+metrics:
+- mrr@10
+- spearman_correlation
 ---
+# mentee-embed-v4
+**A 41M-parameter trilingual text embedding model trained entirely from scratch — no pretrained backbone, no BERT, no RoBERTa. Random initialization only.**
+v4 adds mMARCO Arabic retrieval data (500K triplets) and a third distillation round with hard negatives, yielding significant improvements over v3 across all languages and benchmarks.
+Developed by **Team MenteE AI** (menteeai.org) as part of an ongoing research effort to build competitive multilingual embeddings from the ground up for Arabic, English, and Urdu.
+---
+## Key Facts
+| Property | Value |
+|----------|-------|
+| Parameters | **41M** |
+| Embedding dimension | 384 |
+| Max sequence length | 128 tokens |
+| Languages | Arabic · English · Urdu |
+| Architecture | 12-layer Transformer, custom BPE tokenizer (50K vocab) |
+| Initialization | **Random** (trained from scratch) |
+| Training data | ~2.6M triplets (NLI + MS-MARCO + OPUS + MIRACL + mMARCO Arabic) |
+| Pooling | Mean pooling |
+| Training objective | Relational distillation + InfoNCE contrastive (teacher: `multilingual-e5-base`) |
+---
+## What "From Scratch" Means
+Most embedding models fine-tune an existing pretrained encoder (BERT, RoBERTa, MPNet). **mentee-embed-v4 does not.** We:
+1. Trained a **custom BPE tokenizer** on Arabic, English, and Urdu text
+2. Initialized a **12-layer Transformer with random weights**
+3. Ran **masked language modeling** pretraining (50K steps on trilingual corpus)
+4. Applied **three-round contrastive distillation** with hard negative mining
+No pretrained checkpoint was used at any stage.
+---
+## v4 vs v3 Improvements
+| Benchmark | v3 | v4 | Change |
+|-----------|----|----|--------|
+| Bench MRR@10 (all) | 0.103 | **0.252** | +146% |
+| MIRACL EN MRR@10 | 0.768 | **0.916** | +19% |
+| MIRACL AR MRR@10 | 0.447 | **0.874** | +96% |
+| MIRACL UR MRR@10 | 0.354 | **0.572** | +62% |
+| MS-MARCO MRR@10 | 0.592 | **0.706** | +19% |
+| STS-B Spearman | 0.683 | **0.725** | +6% |
+Key improvements come from:
+- **mMARCO Arabic** — 500K Arabic retrieval triplets added to training
+- **50K MLM steps** — 6x more pretraining than v3 (8K steps)
+- **3 distillation rounds** — extra hard-negative round for better discrimination
+---
+## Benchmark Results
+All baselines evaluated under identical conditions on the same hardware (RTX 5090).
+### MenteE Embed Bench (Custom EN/AR/UR Pilot)
+*121 queries · 18 domains · dialect Arabic · Roman Urdu · hard negatives*
+| Model | Params | EN MRR@10 | AR MRR@10 | UR MRR@10 | Dialect AR | Roman UR | All MRR@10 |
+|---|---|---|---|---|---|---|---|
+| **mentee-embed-v4** | **41M** | 0.369 | 0.120 | 0.261 | 0.095 | 0.399 | **0.252** |
+| paraphrase-mpnet-base-v2 | 278M | 0.764 | 0.602 | 0.611 | 0.445 | 0.661 | 0.661 |
+| paraphrase-MiniLM-L12-v2 | 118M | 0.682 | 0.568 | 0.519 | 0.385 | 0.480 | 0.592 |
+| multilingual-e5-base | 278M | 0.667 | 0.352 | 0.413 | 0.310 | 0.430 | 0.481 |
+| all-MiniLM-L6-v2 | 23M | 0.873 | 0.087 | 0.289 | 0.084 | 0.486 | 0.425 |
+### Protocol A — In-batch Retrieval (MIRACL)
+*Format: acc@1 / MRR@10*
+| Model | MIRACL-EN | MIRACL-AR | MIRACL-UR |
+|-------|-----------|-----------|-----------|
+| **mentee-embed-v4** | 0.870 / **0.916** | 0.825 / **0.874** | 0.475 / **0.572** |
+| paraphrase-mpnet-base-v2 | 0.965 / 0.982 | 0.860 / 0.898 | 0.755 / 0.824 |
+| paraphrase-MiniLM-L12-v2 | 0.965 / 0.979 | 0.815 / 0.868 | 0.720 / 0.785 |
+| multilingual-e5-base | 0.980 / 0.990 | 0.940 / 0.958 | 0.960 / 0.970 |
+| all-MiniLM-L6-v2 | 0.980 / 0.990 | 0.005 / 0.027 | 0.015 / 0.031 |
+> v4 achieves strong MIRACL scores despite being 6.8x smaller than mpnet-base.
+### Protocol C — MS-MARCO Corpus Retrieval (10K passages)
+| Model | MRR@10 | R@5 | R@100 |
+|-------|--------|-----|-------|
+| all-MiniLM-L6-v2 | 0.877 | 0.992 | 0.992 |
+| multilingual-e5-base | 0.856 | 0.992 | 0.996 |
+| paraphrase-mpnet-base-v2 | 0.803 | 0.964 | 0.988 |
+| **mentee-embed-v4** | **0.706** | **0.913** | **0.980** |
+### STS-B (Spearman Correlation)
+| Model | Spearman |
+|-------|----------|
+| paraphrase-mpnet-base-v2 | 0.868 |
+| bge-small-en-v1.5 | 0.859 |
+| paraphrase-MiniLM-L12-v2 | 0.844 |
+| multilingual-e5-base | 0.842 |
+| **mentee-embed-v4** | **0.725** |
+---
+## Speed & Efficiency
+Measured on RTX 5090.
+| Model | Params | Sents/sec (bs=128) | Latency@1 (ms) | VRAM (MB) | Cost/1B sents |
+|---|---|---|---|---|---|
+| **mentee-embed-v4** | **41M** | **18,115** | **5.15** | **175** | **$7.15** |
+| all-MiniLM-L6-v2 | 23M | 12,445 | 4.71 | 103 | $10.40 |
+| multilingual-e5-small | 118M | 9,749 | 7.88 | 482 | $13.28 |
+| paraphrase-MiniLM-L12-v2 | 118M | 9,456 | 8.17 | 573 | $13.69 |
+| paraphrase-mpnet-base-v2 | 278M | 5,158 | 8.04 | 1126 | $25.10 |
+| multilingual-e5-base | 278M | 5,379 | 8.07 | 1126 | $24.07 |
+> v4 is the **fastest model tested** — 3.5x faster than mpnet-base, 45% faster than MiniLM-L12.
+### Efficiency Metrics
+| Model | Bench MRR/1M params | Sents/sec per 1M params |
+|---|---|---|
+| **mentee-embed-v4** | **0.525** | **441.82** |
+| paraphrase-MiniLM-L12-v2 | 0.452 | 80.34 |
+| paraphrase-mpnet-base-v2 | 0.224 | 18.55 |
+| multilingual-e5-base | 0.166 | 19.35 |
+---
+## Training Data
+| Source | Language | Triplets | Type |
+|--------|----------|----------|------|
+| all-NLI (sentence-transformers) | EN | 558K | NLI triplets |
+| XNLI | AR | 128K | NLI triplets |
+| XNLI | UR | 125K | NLI triplets |
+| OPUS-100 EN-UR | EN/UR | 300K | Parallel translation |
+| OPUS-100 AR-EN | AR/EN | 300K | Parallel translation |
+| MS-MARCO BM25 triplets | EN | 500K | Passage retrieval |
+| MS-MARCO hard negatives | EN | 200K | Hard retrieval |
+| mMARCO Arabic | AR | 500K | Passage retrieval |
+| MIRACL | EN/AR/UR | ~9K | Wikipedia retrieval |
+| **Total** | | **~2.6M** | |
+---
+## Training Pipeline
+```
+Stage 1 — MLM Pretraining (50K steps)
+  Random init → masked language modeling on trilingual corpus
+  50,000 steps · batch=64 · vocab=50K BPE
+Stage 2 — Teacher Encoding
+  Teacher: intfloat/multilingual-e5-base (768-dim)
+  Encode all 2.6M triplets to soft labels
+Stage 3 — Distillation Round 1 (no hard negatives)
+  InfoNCE contrastive + relational distillation
+  4,000 steps · batch=512 · temp=0.05
+Stage 4 — Hard Negative Mining Round 1
+  GPU-accelerated top-5 mining across full 2.6M corpus
+Stage 5 — Distillation Round 2 (with hard negatives)
+  Same objective + mined hard negatives per anchor
+  10,000 steps · batch=512 · temp=0.05
+Stage 6 — Hard Negative Mining Round 2 (on improved model)
+  Re-mine with the better round-2 model
+Stage 7 — Distillation Round 3 (final polish)
+  Train on round-2 mined negatives
+  15,000 cumulative steps · batch=512 · temp=0.05
+```
+---
 ## Usage
 ```python
+# pip install torch transformers tokenizers huggingface_hub
+from transformers import AutoModel, AutoTokenizer
+tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
+model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",   trust_remote_code=True)
+sentences = [
+    "Hello, how are you?",
+    "مرحبا، كيف حالك؟",
+    "ہیلو، آپ کیسے ہیں؟"
+]
+embeddings = model.encode(sentences, tokenizer=tok)
+print(embeddings.shape)  # torch.Size([3, 384])
 ```
+> `trust_remote_code=True` is required — standard for custom-architecture models on HuggingFace. The code runs entirely on your machine.
+### Similarity search
+```python
+from transformers import AutoModel, AutoTokenizer
+tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
+model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",   trust_remote_code=True)
+query    = model.encode(["What is machine learning?"], tokenizer=tok)
+passages = model.encode([
+    "Machine learning is a subset of artificial intelligence.",
+    "The weather today is sunny.",
+    "تعلم الآلة هو فرع من فروع الذكاء الاصطناعي.",
+], tokenizer=tok)
+scores = query @ passages.T
+print(scores)  # tensor([[0.81, 0.60, 0.79]])
+```
+---
 ## Limitations
+- **Custom benchmark gap** — v4 scores 0.252 vs 0.661 for mpnet-base on the custom bench; the gap reflects the difference between 2.6M training pairs and web-scale data
+- **Arabic and Urdu** lag behind English on the custom bench, though MIRACL scores are strong (AR 0.874, UR 0.572)
+- Vocabulary limited to 50K tokens trained on ~2.6M sentences — rare scripts and dialects may tokenize poorly
+- Not evaluated on MTEB full suite yet
+---
+## Citation
+```bibtex
+@misc{mentee-embed-v4-2026,
+  title   = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
+             A Compute-Efficient Study of Arabic, English, and Urdu},
+  author  = {Shah, Syed Syab Ahmad and Sania, Shakeel and Hamza, Rustam and Mahboob, Iqbal},
+  year    = {2026},
+  doi     = {10.5281/zenodo.22117673},
+  url     = {https://doi.org/10.5281/zenodo.22117673},
+  note    = {MenteE AI. Apache-2.0 License}
+}
+```
+---
+## About MenteE AI
+Built by **Syed Syab Ahmad Shah** and Team MenteE AI.
+[menteeai.org](https://menteeai.org) · syab@menteeai.org
+Research paper: [10.5281/zenodo.22117673](https://doi.org/10.5281/zenodo.22117673)