README.md

---
base_model: arnastofnun/Qwen3-4B-wmt26-AMI-en-is
language:
- en
- is
library_name: peft
pipeline_tag: translation
tags:
- base_model:adapter:arnastofnun/Qwen3-4B-wmt26-AMI-en-is
- lora
- transformers
- unsloth
- translation
- icelandic
license: apache-2.0
---

# Qwen3-4B-wmt26-AMI-en-is-lora

A LoRA adapter fine-tuning
[`arnastofnun/Qwen3-4B-wmt26-AMI-en-is`](https://huggingface.co/arnastofnun/Qwen3-4B-wmt26-AMI-en-is)
(a mergekit blend of `Qwen/Qwen3-4B-Instruct-2507` with a continually
pretrained Icelandic checkpoint) for **English→Icelandic machine
translation**, trained with [Unsloth](https://github.com/unslothai/unsloth).

Built by [Árnastofnun](https://arnastofnun.is) (The Árni Magnússon Institute
for Icelandic Studies) as part of the AMI submission to the WMT 2026 General
Translation shared task.

## Model description

The base checkpoint was continually pretrained on Icelandic data from the
[Icelandic Gigaword Corpus (IGC)](https://clarin.is/en/resources/gigaword/)
before this LoRA fine-tune was applied (see the
[base model card](https://huggingface.co/arnastofnun/Qwen3-4B-wmt26-AMI-en-is)
for the merge details).

Fine-tuned on a retrieval-augmented (RAG) prompt format: each training
example pairs a domain-specific system prompt (general, news, social,
software, or speech) with a user turn containing retrieved bilingual
dictionary entries and similar example translations for the source
sentence, and an assistant turn with the reference Icelandic translation.
Loss is computed over the assistant turn only. Trained on 4,996 examples
across the five domains above.

### Training data

[`stofnun-arna-magnussonar/WMT2026_finetuning_dataset`](https://github.com/stofnun-arna-magnussonar/WMT2026_finetuning_dataset)
— `wmt26_translate_train.jsonl`, the combined ChatML SFT dataset (4,996
examples), with an explanation of how it was built (RAG-retrieved
dictionary hits and example translations from the Icelandic Gigaword
Corpus, assembled alongside WMT24/25 EN-IS test data and synthetically
generated pairs).

At inference time, source sentences must go through the same
retrieval/prompt-construction pipeline used in training — this adapter
expects dictionary hits and example translations in the prompt context, not
a bare source sentence.

### Training hyperparameters

| | |
|---|---|
| LoRA rank | 16 |
| LoRA alpha | 16 |
| LoRA dropout | 0 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Max sequence length | 8192 |
| Framework | Unsloth + PEFT 0.19.1 |

## Uses

Serve the base model with the adapter attached, e.g. with
[vLLM](https://github.com/vllm-project/vllm):

```bash
vllm serve arnastofnun/Qwen3-4B-wmt26-AMI-en-is \
    --enable-lora \
    --lora-modules translate-lora=arnastofnun/Qwen3-4B-wmt26-AMI-en-is-lora
```

Or load directly with PEFT:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("arnastofnun/Qwen3-4B-wmt26-AMI-en-is")
model = PeftModel.from_pretrained(base, "arnastofnun/Qwen3-4B-wmt26-AMI-en-is-lora")
tokenizer = AutoTokenizer.from_pretrained("arnastofnun/Qwen3-4B-wmt26-AMI-en-is-lora")
```

### Out-of-scope use

Not intended for language pairs or directions other than English→Icelandic,
and expects RAG-style context (dictionary/example retrieval) in its
prompts — quality without that context is not evaluated.

## Citation

If you use this model, please cite:

```bibtex
@inproceedings{steingrimsson2026drag,
  title     = {What a DRAG (It Is Being Small) - The AMI Submission to the WMT 2026 General Translation Shared Task},
  author    = {Steingr{\'i}msson, Stein{\th}{\'o}r and {\TH}{\'o}r{\dh}arson, Sveinbj{\"o}rn and Da{\dh}ason, J{\'o}n Fri{\dh}rik},
  booktitle = {Proceedings of the Eleventh Conference on Machine Translation},
  month     = oct,
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}
```

### Framework versions

- PEFT 0.19.1