vivekkopthsd/distilluse-taxfinance-multilingual

Distiluse Multilingual Tax-Finance — Cross-Lingual Distillation

A ~135M-parameter, genuinely multilingual sentence-embedding model for Indian personal income-tax and finance document retrieval, produced by knowledge distillation from a 300M-parameter, domain- and language-tuned teacher.

This is a research artifact. All reported metrics are three-seed means ± standard deviations on held-out test splits, with the teacher and the untrained student as upper/lower reference bounds. No test-set leakage: model selection (early stopping) uses a separate validation split.

Intended Use

Semantic search and RAG over tax and finance corpora, with emphasis on cross-lingual retrieval — Hindi-language queries retrieving English documents (and English queries retrieving English documents).

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("vivekkopthsd/distilluse-taxfinance-multilingual")
emb = model.encode([
    "How do I claim House Rent Allowance (HRA)?",          # EN query
    "हाउस रेंट अलाउंस (HRA) कैसे क्लेम करूं?"                # HI query
], normalize_embeddings=True)

# cross-lingual: compute cosine similarity between the two
import numpy as np
print(emb[0] @ emb[1])   # similarity between EN and HI query

Model

Property	Value
Base architecture	distiluse-base-multilingual-cased-v2 (DistilBERT-6L, 768 hidden)
Total parameters (backbone, merged)	≈ 134.73M
Total parameters (incl. 2_Dense pooling projection)	≈ 135.1M
Pooling	mean pooling
Final projection	Dense (768→512, Tanh) + L2-normalization
Output embedding dim	512
Training paradigm	Parameter-efficient fine-tuning: LoRA (r=16, α=32, dropout=0.05)
Trainable parameters	≈ 2.9% of total

Teacher

The distillation teacher is a multilingual (English + Hindi + Hinglish) embedding model fine-tuned on Indian income-tax and finance question-answer data (embeddinggemma-300m-taxrag-ft, 300M parameters, 768-dim embeddings). The teacher is frozen and never updated during student training.

Training

Objective — temperature-scaled KL / cross-entropy on ranking

The student is trained so that, within each batch, its similarity distribution over candidate passages approximates the teacher's. The cross-entropy objective is dimension-agnostic (student 512-dim output vs teacher 768-dim) because it optimizes the ranking induced by the teacher, not a direct embedding regression.

For each batch, the loss is the KL divergence between the student's temperature-scaled softmax over candidates and the teacher's:

$$\mathcal{L}=\text{KL}(\sigma ({\mathbf{S}}_t\mathrm{/}T)\mathrm{\parallel }\sigma ({\mathbf{S}}_s\mathrm{/}T))\backslash mathcal\{L\}\; =\; \backslash text\{KL\}\backslash big(\backslash ,\backslash sigma(\backslash mathbf\{S\}\_t\; /\; T)\; \backslash ;\backslash |\backslash ;\; \backslash sigma(\backslash mathbf\{S\}\_s\; /\; T)\backslash ,\backslash big)$$

with temperature $T=1.0$, and $\mathbf{S}$ the query–passage cosine-similarity matrix.

Reimers-style parallel-pair training (cross-lingual alignment)

Following Reimers & Gurevych ("Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation", 2020): a translated sentence is mapped to the same location in the vector space as the original. For every English tax training passage, a Hindi translation (via Helsinki-NLP/opus-mt-en-hi) is added to the batch as a parallel positive candidate. The student therefore learns that an English query ranks both its English passage and the Hindi translation of that passage highly — directly teaching cross-lingual alignment rather than only measuring it at test time.

Comparison condition (direct fine-tuning)

To isolate the effect of distillation, an identical student is trained on the same data and same protocol with the same multi-positive InfoNCE objective, but with the teacher's soft targets replaced by hard one-hot labels. The only difference between the two conditions is the objective (teacher-rank distillation vs. direct hard-label supervision). Both conditions use identical seeds, data ordering, batch size, and hyperparameters.

Hyperparameters

Hyperparameter	Value
LoRA rank / alpha / dropout	16 / 32 / 0.05
Initial learning rate	2×10⁻⁴
Batch size	24
Epochs	6 (early-stopped by validation elbow)
Warmup ratio	0.1
LR schedule	Cosine decay
Temperature	1.0
Gradient clipping	1.0
Optimizer	AdamW (weight decay 0.01)
Seeds	42, 123, 456

Data

• Tax QA: indian-income-tax-qa — Indian personal income-tax QA pairs.
• Finance EN: AIR-Bench / qa_finance_en — English finance data.
• Hindi finance: native Hindi finance QA pairs (Nifty / EBITDA / breakout-level terminology).
• Parallel pairs: Hindi machine translations of English tax passages (opus-mt-en-hi) generated for cross-lingual training.

Splits (no leakage): 80/10/10 train/validation/test for tax (stratified by topic); finance and Hindi split analogously. Model selection (early stopping, LoRA hyperparameters fixed a priori) uses the validation split only; all reported metrics are on the untouched test split.

Evaluation

Three held-out retrieval conditions, each evaluated with standard IR metrics:

1. Monolingual EN: English queries → English documents (dense 839-candidate pool).
2. Native Hindi: Hindi queries → Hindi documents.
3. Cross-lingual (XL): Hindi-translated queries → English documents (the hard, discriminating condition that an English-only model cannot satisfy).

Results — three-seed mean ± std (nDCG@10)

Condition	Distilled (ours)	Direct fine-tune (control)	Teacher (300M)	Student, untrained
Monolingual EN	0.4856 ± 0.0018	0.5062 ± 0.0133	0.4872	0.3630
Native Hindi	0.9908 ± 0.0000	0.9331 ± 0.0111	0.9908	0.9548
Cross-lingual (XL)	0.3404 ± 0.0009	0.2517 ± 0.0110	0.3831	0.2414

Full metric detail, distilled model (mean ± std):

Metric	Monolingual EN	Native Hindi	Cross-lingual (XL)
nDCG@10	0.4856 ± 0.0018	0.9908 ± 0.0000	0.3404 ± 0.0009
MRR@10	0.3885 ± 0.0015	0.9875 ± 0.0000	0.2811 ± 0.0011
MAP@100	0.3988 ± 0.0013	0.9875 ± 0.0000	0.2936 ± 0.0011
Recall@1	0.8097 ± 0.0042	1.0000 ± 0.0000	0.5391 ± 0.0014
Recall@5	0.5391 ± 0.0030	1.0000 ± 0.0000	0.3711 ± 0.0028

Key findings

1. Cross-lingual superiority of distillation. The distilled student attains nDCG@10 = 0.3404 on cross-lingual retrieval, exceeding direct fine-tuning (0.2517) by +0.089 with non-overlapping error bars (σ ≈ 0.001 vs 0.011). Direct fine-tuning lifts cross-lingual performance only slightly above the untrained student (0.2517 vs 0.2414), i.e., hard-label supervision largely fails to transfer cross-lingual ability.
2. Native-Hindi parity with the teacher. The distilled student reaches 0.9908 Hindi nDCG@10 — statistically indistinguishable from the 300M teacher (0.9908) — while being ~2× smaller.
3. Recovered teacher capacity. The ~135M student recovers ≈ 89% of the teacher's cross-lingual retrieval performance (0.3404 / 0.3831) and fully matches it on Hindi.
4. Monolingual EN trade-off. Distillation slightly underperforms direct fine-tuning on English-only retrieval (0.4856 vs 0.5062). This is the expected cross-lingual/English capability trade-off: the teacher's domain- and language-tuned ranking favors cross-lingual generalization over English-only precision.

Ablation: effect of parallel-pair (Reimers) training

The same distillation setup without parallel-pair data attains cross-lingual nDCG@10 of 0.3064 ± 0.0031. Adding parallel EN↔HI training pairs raises it to 0.3404 ± 0.0009 — a +0.034 improvement, confirming that teaching the EN↔HI alignment during training (rather than testing it only at evaluation) materially improves cross-lingual transfer.

Limitations

• Cross-lingual coverage is EN↔HI only. Other Indian languages (Hindi-Hinglish, etc.) beyond the teacher's fine-tuning languages were not evaluated.
• Reference-bound, not MTEB/BeIR standard. Scores are reported on our held-out domain corpus, not on standard multilingual IR benchmarks (MTEB, MIRACL, BeIR). Absolute values should not be compared across different benchmarks.
• Hindi MT for training relies on NMT quality; translation noise may bound the achievable cross-lingual alignment.
• English-only retrieval is not state-of-the-art in this setup; users prioritizing pure-English retrieval over cross-lingual ability should compare against direct-finetuned or larger models on their own data.
• Evaluation pool is server-side fixed; in open-domain retrieval the candidate pool may differ and absolute metrics would change.

Verified Reproduction

The published weights in this repository were re-evaluated on the held-out test sets. The reproduced nDCG@10 values (EN 0.4835, Hindi 0.9908, cross-lingual 0.3395) match the reported three-seed means (0.4856/0.9908/0.3404) within one standard deviation, confirming that the downloadable artifact reproduces the metrics on this card. Embeddings are produced by the full model pipeline (mean pooling + Dense(768→512, Tanh) + L2-normalization).

Reproducibility

Configurations and full per-seed metric tables used to train and report this model are pinned. LoRA adapters for all three seeds (distill and direct) are available, allowing exact metric regeneration. Base model weights and teacher are downloadable from Hugging Face; training data sources are public as cited above.

Citation

If you find this model useful, please cite:

@software{distilluse_taxfinance_multilingual,
  author = {Vivek},
  title  = {Distiluse Multilingual Tax-Finance (Cross-Lingual Distillation)},
  year   = {2026},
  note   = {~135M multilingual sentence embedding distilled from a 300M domain-tuned teacher},
}