dougalldeepmind/2026-07-31-qwen36-difficult-advice-tulu-lora-40-60-assistant-loss-only

Qwen3.6-27B — 40/60 difficult-advice + TULU3, assistant_loss_only

LoRA adapter for Qwen/Qwen3.6-27B, trained on 40% difficult-advice / 60% TULU3 replay with loss on assistant tokens only.

Controlled ablation of …-tulu-lora-40-60: same mixture file (md5 88f39a3d01e59ba9d592b26c1705c57f, 1,982 rows, 1,493,359 tokens), same seed, same hyperparameters, same rendered strings. The loss mask is the only difference.

Training data: qwen3.6-27b-sft-mixture-40-60_assistant_loss_only, published with per-row assistant spans so the mask reproduces without our code.

What is supervised

Source	Rows	Tokens	Supervised
difficult-advice	580	597,013	85.48%
TULU3 replay	1,402	896,346	77.98%
Total	1,982	1,493,359	80.98%

Supervision starts immediately after the <|im_start|>assistant\n header — which the model is given at inference and never has to produce — and ends after the closing <|im_end|>, which it must produce in order to stop. Everything else is -100.

<|im_start|>   MASKED
assistant      MASKED
\n             MASKED
<think>        LOSS     <- supervision starts at the first generated token

TRL's assistant_only_loss flag does not work on this model. It requires {% generation %} markers the chat template lacks, and it re-renders from messages, discarding the think-block convention baked into the pre-rendered mixture. Spans are derived from the rendered text via the fast tokenizer's offset mapping instead; TRL receives finished labels.

Reasoning traces

Data	Renders as
difficult-advice (580/580 rows)	<think>real reasoning</think>
TULU3 replay (0/1,402 rows)	no <think> block at all

Zero rows carry an empty <think></think> — that pattern is Qwen3.6's explicit do-not-deliberate marker and trains a model to stop reasoning.

Training

bf16 LoRA (not QLoRA — bitsandbytes does not reliably cover this model's hybrid linear-attention/SSM layers: 48 of 64 layers are Gated DeltaNet, and none of their projections receive an adapter, so quantization error there would be uncorrectable). 1×H100 80GB, 90 min.

r / alpha / dropout	32 / 64 / 0.05
target modules	regex scoped to model.language_model.* (q/k/v/o/gate/up/down proj)
epochs / steps	1 / 124
batch × grad-accum	1 × 16
lr / schedule	1e-4, cosine, 3% warmup
max seq len / packing	2048 / off

Final train loss 0.948, token accuracy 0.749. Loss is not comparable to the full-token arm by construction — a different set of tokens is scored.

Status

Not yet evaluated. For reference, the full-token sweep at the same total budget:

Difficult-advice share	ODCV-Bench MR	Agentic-misalignment
0% (base)	37.2%	65.5%
10%	24.7%	38.7%
20%	19.2%	25.3%
40%	15.4%	19.5%

The 40% row is this adapter's direct counterpart.

Sibling assistant-loss-only arms: 10-90 · 20-80 · 40-60

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-07-31-qwen36-difficult-advice-tulu-lora-40-60-assistant-loss-only")
model = model.merge_and_unload()  # vLLM LoRA support for this hybrid arch is unproven

Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language checkpoint. Merging drops the base model's 15 mtp.* tensors, so speculative decoding needs them grafted back.