Qwen3-4B-wmt26-AMI-en-is-lora

A LoRA adapter fine-tuning arnastofnun/Qwen3-4B-wmt26-AMI-en-is (a mergekit blend of Qwen/Qwen3-4B-Instruct-2507 with a continually pretrained Icelandic checkpoint) for English→Icelandic machine translation, trained with Unsloth.

Built by Árnastofnun (The Árni Magnússon Institute for Icelandic Studies) as part of the AMI submission to the WMT 2026 General Translation shared task.

Model description

The base checkpoint was continually pretrained on Icelandic data from the Icelandic Gigaword Corpus (IGC) before this LoRA fine-tune was applied (see the base model card for the merge details).

Fine-tuned on a retrieval-augmented (RAG) prompt format: each training example pairs a domain-specific system prompt (general, news, social, software, or speech) with a user turn containing retrieved bilingual dictionary entries and similar example translations for the source sentence, and an assistant turn with the reference Icelandic translation. Loss is computed over the assistant turn only. Trained on 4,996 examples across the five domains above.

Training data

stofnun-arna-magnussonar/WMT2026_finetuning_dataset — wmt26_translate_train.jsonl, the combined ChatML SFT dataset (4,996 examples), with an explanation of how it was built (RAG-retrieved dictionary hits and example translations from the Icelandic Gigaword Corpus, assembled alongside WMT24/25 EN-IS test data and synthetically generated pairs).

At inference time, source sentences must go through the same retrieval/prompt-construction pipeline used in training — this adapter expects dictionary hits and example translations in the prompt context, not a bare source sentence.

Training hyperparameters

LoRA rank	16
LoRA alpha	16
LoRA dropout	0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Max sequence length	8192
Framework	Unsloth + PEFT 0.19.1

Uses

Serve the base model with the adapter attached, e.g. with vLLM:

vllm serve arnastofnun/Qwen3-4B-wmt26-AMI-en-is \
    --enable-lora \
    --lora-modules translate-lora=arnastofnun/Qwen3-4B-wmt26-AMI-en-is-lora

Or load directly with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("arnastofnun/Qwen3-4B-wmt26-AMI-en-is")
model = PeftModel.from_pretrained(base, "arnastofnun/Qwen3-4B-wmt26-AMI-en-is-lora")
tokenizer = AutoTokenizer.from_pretrained("arnastofnun/Qwen3-4B-wmt26-AMI-en-is-lora")

Out-of-scope use

Not intended for language pairs or directions other than English→Icelandic, and expects RAG-style context (dictionary/example retrieval) in its prompts — quality without that context is not evaluated.

Citation

If you use this model, please cite:

@inproceedings{steingrimsson2026drag,
  title     = {What a DRAG (It Is Being Small) - The AMI Submission to the WMT 2026 General Translation Shared Task},
  author    = {Steingr{\'i}msson, Stein{\th}{\'o}r and {\TH}{\'o}r{\dh}arson, Sveinbj{\"o}rn and Da{\dh}ason, J{\'o}n Fri{\dh}rik},
  booktitle = {Proceedings of the Eleventh Conference on Machine Translation},
  month     = oct,
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}

Framework versions

• PEFT 0.19.1