dougalldeepmind/2026-07-31-qwen36-difficult-advice-tulu-lora-80-20-empty-think-tags

Qwen3.6-27B — 80_20_empty_think_tags

LoRA adapter for Qwen/Qwen3.6-27B trained on the 20% difficult-advice / 80% TULU3 mixture, with Qwen3.6's empty <think></think> marker on the replay rows conditioned on but excluded from the loss. Supervision is assistant-tokens-only.

Controlled ablation of …-20-80-assistant_loss_only: same rows, same seed, same hyperparameters, and an effectively identical supervised token set. The presence of the marker as context is the only variable.

Training data: qwen3.6-27b-sft-mixture-80_20_empty_think_tags.

The marker

Every TULU3 replay row carries <think>\n\n</think>\n\n on its final assistant turn -- Qwen3.6's explicit non-thinking marker, placed exactly where apply_chat_template puts it (the template emits it only on the final turn, never on historical ones; the insertion is asserted to reproduce the template byte-for-byte before any data is touched).

Those marker tokens are masked out of the loss. The model is conditioned on the marker -- which is how Qwen3.6 injects it as a prefill in non-thinking mode -- but is never trained to emit it, since learning to emit an empty think block is the documented reasoning-collapse pattern. Difficult-advice rows are untouched and their real <think> traces stay fully supervised.

<|im_start|>   MASKED
assistant      MASKED
<think>        MASKED   <- marker: context, not a target
</think>       MASKED
Pre            LOSS     <- supervision starts at the answer

	Rows	Tokens	Marker	Supervised
difficult-advice	291	299,455	0	85.45%
TULU3 replay	1,878	1,202,056	1,878	77.51%
Total	2,169	1,501,511	1,878	79.09%

1,187,560 supervised tokens, versus 1,187,563 in the plain assistant-only 20/80 arm -- the supervised set is effectively identical, so the marker's presence as context is the only variable. The 3-token gap is one row (index 1302) that sat at exactly 2,048 tokens and now reaches 2,052, truncating its trailing <|im_end|>. Left as-is so max_seq_len stays comparable across arms.

Training

bf16 LoRA (not QLoRA — bitsandbytes does not reliably cover this model's hybrid linear-attention/SSM layers: 48 of 64 layers are Gated DeltaNet and none of their projections receive an adapter, so quantization error there would be uncorrectable). 1×H100 SXM, 1h34m.

r / alpha / dropout	32 / 64 / 0.05
target modules	regex scoped to model.language_model.* (q/k/v/o/gate/up/down proj)
epochs / steps	1 / 136
batch × grad-accum	1 × 16
lr / schedule	1e-4, cosine, 3% warmup
max seq len / packing	2048 / off

Final train loss 0.848, token accuracy 0.803 (plain assistant-only 20/80 arm: 0.896 / 0.800 — near-identical, as expected when the supervised set matches).

Status

Not yet evaluated. For reference, the full-token sweep at the same budget:

Difficult-advice share	ODCV-Bench MR	Agentic-misalignment
0% (base)	37.2%	65.5%
10%	24.7%	38.7%
20%	19.2%	25.3%
40%	15.4%	19.5%

A useful check on this arm specifically: probe <think> length after training. The whole point of masking the marker is to avoid teaching reasoning suppression, so measured think-block length versus the plain arm is the direct test of whether that worked.

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-07-31-qwen36-difficult-advice-tulu-lora-80-20-empty-think-tags")
model = model.merge_and_unload()  # vLLM LoRA support for this hybrid arch is unproven

Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language checkpoint. Merging drops the base model's 15 mtp.* tensors, so speculative decoding needs them grafted back.