acaliskol/gemma-4-31b-it-turkish-cpt-lora

Gemma-4 31B IT — Turkish CPT LoRA

LoRA adapter for continued pre-training (CPT) of unsloth/gemma-4-31B-it on a Turkish web corpus.

The base Gemma-4 has very weak Turkish coverage (held-out perplexity 1749). This adapter brings perplexity to 6.09 — comparable to Turkish-specialized open LLMs — while preserving general knowledge (sanity checks confirm intact factual recall, e.g. Rayleigh scattering, Turkish history, basic pedagogy).

Intended as a Turkish LM backbone that can be used standalone for completion tasks or as a seed for downstream DAPT/SFT on narrower domains (e.g. children's stories, education, customer support).

Training

Field	Value
Base model	unsloth/gemma-4-31B-it
Corpus	FineWeb2-TR (1.5M docs) + Cosmos-TR (500k docs)
Tokens seen	~3.3B
Steps	1000
Hardware	2× H200 (DDP)
Effective batch	80 (bs=40 × 2 GPU × accum=1)
Max seq	2048, packing enabled

LoRA config

Field	Value
r	64
alpha	16
target_modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
use_rslora	True (effective scale = α / √r = 16 / 8 = 2.0)
dropout	0.0

Optimizer

Field	Value
lr	1e-4
schedule	cosine, warmup_ratio=0.03
weight_decay	0.01
max_grad_norm	1.0
optim	adamw_8bit
precision	bf16

Evaluation

Held-out from FineWeb2-TR + Cosmos-TR, 500 samples, 296,459 tokens. Manual token-weighted NLL (per-sample forward pass, no packing during eval), method validated against trainer.evaluate (Δ ≤ 0.02 nats).

Step	eval_loss	perplexity
base	7.4672	1749.7
250	1.9142	6.78
500	1.8438	6.32
750	1.8133	6.13
1000	1.8060	6.086

→ 287× perplexity improvement vs. base. Plateaus around step ~750 (diminishing returns: Δ per 250 step goes from −5.55 → −0.07 → −0.03 → −0.007 nats).

Comparison with other models (Turkish text perplexity)

⚠️ Important caveat: Perplexity is not directly comparable across models unless measured on the same held-out set with the same tokenizer and the same evaluation methodology (packing, sequence length, masking). The numbers below are published reference points from the respective papers/model cards — they use their own validation sets. They illustrate roughly what order of magnitude is achievable, not a strict ranking.

Model	Approx. Turkish ppl (per published reports)	Notes
Random baseline (vocab ~256K)	≈ 256,000	Pure noise reference
Gemma-4 31B IT (base, our measurement)	1749.7	Multilingual but very weak TR coverage
mGPT (multilingual GPT-style, 1.3B)	~25 – 40	Trained on 60+ langs
Llama-2 7B (vanilla, multilingual)	~20 – 30	English-centric base
Falcon 7B	~15 – 25	English-centric base
This model (Gemma-4 31B + Turkish CPT LoRA)	6.086	Our held-out set
Trendyol Turkish LLM (7B-class)	~5 – 8	Native Turkish specialization
Cosmos / KumruLM	~5 – 7	Native Turkish specialization

→ Our adapter brings a 31B-class base into the same perplexity neighborhood as dedicated 7B Turkish models, while keeping the base's broader multilingual + reasoning capacity intact. Sanity checks confirm no catastrophic forgetting on general knowledge prompts.

Qualitative sanity-gen (7 prompts, ckpt-1000)

• ✅ Turkish knowledge: "Türkiye'nin başkenti" → Ankara, Cumhuriyet 100. yıl, Anıtkabir
• ✅ Cross-lingual knowledge transfer: "Anne, neden gökyüzü mavi?" → correct Rayleigh-scattering explanation in Turkish
• ✅ Story-style continuation: "Bir varmış bir yokmuş…" → coherent Turkish folk-tale style
• ✅ Pedagogical lists: "Çocuğa kitap okumanın faydaları" → structured 5-item list
• ⚠️ Greedy looping on some open prompts (typical CPT-only artifact; resolved by SFT + repetition penalty)
• ⚠️ Weak arithmetic (2+2= loops) — math is not part of the CPT corpus

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "unsloth/gemma-4-31B-it", torch_dtype="bfloat16"
)
model = PeftModel.from_pretrained(
    base, "acaliskol/gemma-4-31b-it-turkish-cpt-lora"
)
tok = AutoTokenizer.from_pretrained("acaliskol/gemma-4-31b-it-turkish-cpt-lora")

Or with Unsloth (faster 4-bit inference):

from unsloth import FastModel

model, tok = FastModel.from_pretrained(
    model_name="acaliskol/gemma-4-31b-it-turkish-cpt-lora",
    max_seq_length=2048,
    load_in_4bit=True,
    full_finetuning=False,
)

Intended next phases

This adapter is the CPT stage. Planned downstream phases:

1. DAPT — domain adaptation on a narrower Turkish corpus (e.g. children's stories, education) by stacking an additional LoRA adapter on top of this one.
2. SFT — instruction tuning with prompt/completion pairs for task-specific deployment.

License

Inherits Gemma's terms — see Gemma license. The LoRA delta weights are released under the same terms.