dougalldeepmind/2026-07-31-wrongly-trained-qwen36-toolcalling-tulu-lora-20-80

Qwen3.6-27B — tool-calling + TULU3 LoRA (20/80 mixture)

LoRA adapter for Qwen/Qwen3.6-27B. The 20% target portion is entirely agentic tool-use data — conversations where the model itself is the actor holding live tools — and the other 80% is TULU3 replay.

This is the pure-tool-calling cell of a family that holds total tokens and the 20% target share fixed and varies only the composition of that 20%:

Arm	20% composition	ODCV-Bench	Agentic-misalignment
base (no SFT)	—	37.2%	65.5%
…-difficult-advice-tulu-lora-20-80	difficult-advice only	19.2%	25.3%
…-threeway-constitution-lora	equal thirds: embodied / difficult-advice / agentic	not yet run	not yet run
this	agentic tool-use only	not yet run	not yet run
…-tulu-100pct-lora	none (zero-dose control)	—	—

Training mixture

Published in full as LASR-Callum/2026-07-31-toolcalling-tulu-20-80-mixture.

Source	Examples	Tokens	Share	Rendering
agentic tool-use (approved_agentic, fullthink)	124	297,894	19.96%	reasoning kept where the source had it; no empty think blocks
TULU3 replay	1,878	1,194,548	80.04%	no <think> block at all
Total	2,002	1,492,442

25 of the 124 agentic documents actually emit tool calls — 92 <tool_call> spans in Qwen3.6's XML dialect, all verified balanced.

Training

bf16 LoRA (not QLoRA — bitsandbytes does not reliably cover this model's hybrid linear-attention/SSM layers), 1×H100 80GB SXM, 1h38m09s.

r / alpha / dropout	32 / 64 / 0.05
target modules	regex scoped to model.language_model.* (q/k/v/o/gate/up/down proj)
epochs / steps	1 / 126
batch x grad-accum	1 x 16
lr / schedule	1e-4, cosine, 3% warmup, annealed to 0
max seq len / packing	4096 / off
assistant_only_loss	false
seed	0
trainable params	159.4M

Loss: 2.7528 → 1.057 (epoch average); the last logged step (125) read 1.0199. Epoch-average mean token accuracy 0.7071, final grad_norm 0.3573, 1,492,498 tokens consumed.

Curves and the full log history are in LASR-Callum/2026-07-31-toolcalling-tulu-sft-run.

Why max_seq_len is 4096 and the sibling arms use 2048

These agentic conversations run 9–13 turns with a median of 2,348 tokens, and 99 of the 151 source documents exceed 2048. Measured: a 2048 cap keeps only 80.4% of the corpus and severs 11 of its 98 <tool_call> spans, inside exactly the long conversations the tool calls live in. At 4096 the mixture is truncated nowhere at all. The cost is that this arm differs from its siblings on one hyperparameter as well as on composition — read the head-to-head with that caveat.

Known caveats

1. Reasoning density. Only 30 of the 124 agentic rows (24%) carry a real reasoning trace, against every target example in the difficult-advice 20/80 arm. If the dose-response in this family is driven by reasoning rather than topic coverage, that is confounded with the composition change here. Inherent to the source corpus.
2. target_modules only half-applies. Qwen3.6-27B is hybrid: q/k/v/o_proj exist in 16 of 64 layers, the rest being linear-attention blocks with different module names. gate/up/down_proj attach to all 64. So this adapter tunes MLP throughout but attention in only a quarter of the stack. Same for every arm in the family, so comparisons are unaffected.
3. Not yet evaluated. No ODCV-Bench or agentic-misalignment number exists for this arm yet.

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-07-31-wrongly-trained-qwen36-toolcalling-tulu-lora-20-80")
model = model.merge_and_unload()  # vLLM LoRA support for this hybrid arch is unproven

Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language checkpoint. Merging drops the base model's 15 mtp.* tensors, so speculative decoding needs them grafted back.

Provenance

Repository https://github.com/Matthew-Bozoukov/teaching_claude_why_replication @ 639d85c. Built with src/experiments/build_toolcalling_mixture.py, trained with src/experiments/train_lora.py --config configs/train_lora_toolcalling.yaml.