LASR-Callum/2026-08-03-qwen36-lora-500k-numina-only

Qwen3.6-27B — 500k tokens, NuminaMath-CoT only

LoRA adapter trained on 497,968 tokens of NuminaMath-CoT with loss on assistant tokens only, for 1 epoch at lr 4e-5.

Training data: qwen3.6-27b-mixture-500k-numina-only.

Examples / tokens	934 / 497,968
Supervised	409,975 / 497,968 = 82.3%
Epochs / steps	1 / 59
lr / schedule	4e-5, cosine, 3% warmup
Runtime	36 min, 1x H100 80GB
r / alpha / dropout	32 / 64 / 0.05
batch x grad-accum	1 x 16
max seq len / packing	3072 / off
Final loss	0.538
Token accuracy	0.878

Reading the loss

0.538 is roughly half the loss of every mixed-corpus arm in this family (0.76-0.95), and token accuracy is ~8 points higher. That is the narrowness of the target, not model quality: NuminaMath solutions follow a highly regular mathematical structure, so next-token prediction is far easier than on open-ended instruction data. Loss converged by about step 15 of 59.

Run	Data	Loss	Token acc
this	NuminaMath only	0.538	0.878
500k maths-weighted	67% maths + mixed	0.878	0.793
500k da20 + maths	20% difficult-advice	0.946	0.776

What is supervised

Everything outside an assistant turn is -100. A supervised span starts immediately after the <|im_start|>assistant\n header and ends after the closing <|im_end|>. Verified before training: zero user or system tokens inside any supervised span.

loss_type: nll is set because TRL's default chunked-CE path patches the LM head and reads forward.__func__, which fails on this checkpoint's functools.partial forward.

Not yet evaluated.

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-08-03-qwen36-lora-500k-numina-only")
model = model.merge_and_unload()

Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language checkpoint.