Abdusin/gemma-3-4b-est-v1

Gemma 3 4B Estonian (v1)

A Gemma 3 4B base model adapted for the Estonian language. The stock Gemma 3 4B knows surprisingly little Estonian; this model was trained to fix that while staying small enough to run comfortably on a phone.

What was done

Two stages, both with QLoRA on a single consumer GPU (RTX 5070 12 GB, ~26 hours total):

1. Continued pretraining on 161.5M tokens of Estonian web text (the Estonian subset of FineWeb-2), packed to 2048-token blocks, one epoch, learning rate 5e-5.
2. Supervised fine-tuning on ~30.5k examples (one epoch, lr 1e-4):
   • 15k Estonian instructions from tartuNLP/magpie-gemma-3-12b-it-100k-et
   • ~2.9k responses distilled from the open EstLLM-8B model on native Estonian prompts (dictionary explanations, inflection, reading comprehension)
   • ~11k pairs built from public Estonian language resources: EKI dictionary definitions → word (word_meanings_et), noun-phrase inflection (inflection_et), grammar correction with gold targets (grammar_et), extractive QA (EstQA), and news summarization (ERRnews)

Evaluation

Measured with the official Estonian LLM benchmark harness (LREC 2026, Lillepalu & Alumäe), full test sets, zero-shot with the chat template. Published numbers for reference models are from the benchmark paper (arXiv:2510.21193).

Task (exact match)	Gemma-3-4B-it (base)	This model	EstLLM-8B (published)
Inflection (1,400)	0.107	0.779	0.811
Word meanings (1,000)	0.133	0.252	0.327
Grammar correction (1,000)	0.083	0.223	0.275
Trivia (800)	—	0.276	0.586
News summarization, ROUGE-L (523)	~0.07	0.162	0.152
National exam (1,614, macro over 8 subjects)	—	0.445	0.575
6-task average	—	0.356	0.454

For context: Qwen3-4B-Instruct scores 0.212 and Llama-3.1-8B-Instruct 0.244 on the published version of this benchmark. This model beats the summarization score of EstLLM-8B and stays competitive with it on inflection, while being a 4B model trained on less than 2% of the Estonian pretraining data EstLLM used.

GGUF files

File	Size	Notes
gguf/gemma-3-4b-est-v1-q4_k_m.gguf	2.49 GB	Recommended; runs on 6 GB+ phones
gguf/gemma-3-4b-est-v1-q5_k_m.gguf	2.83 GB	Slightly better quality
gguf/gemma-3-4b-est-v1-f16.gguf	7.77 GB	Reference

Works out of the box with llama.cpp, Ollama, LM Studio, and on Android/iOS via PocketPal AI or ChatterUI (context up to 4096 recommended).

Usage

from transformers import AutoTokenizer, Gemma3ForConditionalGeneration
import torch

tok = AutoTokenizer.from_pretrained("Abdusin/gemma-3-4b-est-v1")
model = Gemma3ForConditionalGeneration.from_pretrained(
    "Abdusin/gemma-3-4b-est-v1", torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Tere! Räägi veidi Tartu linna ajaloost."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=300)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Limitations

• Trained primarily for Estonian; English and other languages were not evaluated after training and may have degraded somewhat.
• World-knowledge tasks (trivia, national exams) still trail much larger Estonian models — the continued-pretraining corpus here is 161M tokens, which is small.
• Standard LLM caveats apply: it can hallucinate confidently, and it should not be used for medical, legal, or other high-stakes decisions.

License

This model inherits the Gemma Terms of Use. By using or redistributing it you agree to those terms.

Credits

• Base model: Gemma 3 by Google
• Training recipe follows the published Estonian adaptation work: EstLLM (teacher model), Llammas, and the Estonian LLM benchmark by TartuNLP / TalTech
• Datasets by TartuNLP, TalTechNLP, the Institute of the Estonian Language (EKI), ERR, and the FineWeb-2 project — thank you for making them public