dougalldeepmind
/

2026-07-31-wrongly-trained-qwen36-toolcalling-tulu-lora-20-80

@@ -1,116 +1,116 @@
----
-base_model: Qwen/Qwen3.6-27B
-library_name: peft
-pipeline_tag: image-text-to-text
-tags:
-  - lora
-  - peft
-  - sft
-  - trl
-  - alignment
-  - agentic-misalignment
-  - tool-use
-datasets:
-  - LASR-Callum/2026-07-31-toolcalling-tulu-20-80-mixture
-  - LASR-Callum/2026-07-29-synthdoc-approved-constitution-sft
-  - LASR-Callum/tulu3-replay-80pct-qwen3.6-27b
-license: apache-2.0
----
-# Qwen3.6-27B — tool-calling + TULU3 LoRA (**20/80** mixture)
-LoRA adapter for [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). The 20% target
-portion is **entirely agentic tool-use data** — conversations where the model itself is the actor
-holding live tools — and the other 80% is TULU3 replay.
-This is the pure-tool-calling cell of a family that holds total tokens and the 20% target share
-fixed and varies only the **composition** of that 20%:
-| Arm | 20% composition | ODCV-Bench | Agentic-misalignment |
-|---|---|---|---|
-| [base (no SFT)](https://huggingface.co/Qwen/Qwen3.6-27B) | — | 37.2% | 65.5% |
-| [`…-difficult-advice-tulu-lora-20-80`](https://huggingface.co/LASR-Callum/qwen3.6-27b-difficult-advice-tulu-lora-20-80) | difficult-advice only | 19.2% | 25.3% |
-| [`…-threeway-constitution-lora`](https://huggingface.co/LASR-Callum/qwen3.6-27b-threeway-constitution-lora) | equal thirds: embodied / difficult-advice / agentic | not yet run | not yet run |
-| **this** | **agentic tool-use only** | **not yet run** | **not yet run** |
-| [`…-tulu-100pct-lora`](https://huggingface.co/LASR-Callum/qwen3.6-27b-tulu-100pct-lora) | none (zero-dose control) | — | — |
-## Training mixture
-Published in full as
-[`LASR-Callum/2026-07-31-toolcalling-tulu-20-80-mixture`](https://huggingface.co/datasets/LASR-Callum/2026-07-31-toolcalling-tulu-20-80-mixture).
-| Source | Examples | Tokens | Share | Rendering |
-|---|---:|---:|---:|---|
-| agentic tool-use (`approved_agentic`, `fullthink`) | 124 | 297,894 | 19.96% | reasoning kept where the source had it; **no** empty think blocks |
-| TULU3 replay | 1,878 | 1,194,548 | 80.04% | **no** `<think>` block at all |
-| **Total** | **2,002** | **1,492,442** | | |
-25 of the 124 agentic documents actually emit tool calls
-— 92 `<tool_call>` spans in Qwen3.6's XML dialect, all verified balanced.
-## Training
-bf16 LoRA (not QLoRA — bitsandbytes does not reliably cover this model's hybrid
-linear-attention/SSM layers), 1×H100 80GB SXM, **1h38m09s**.
-| | |
-|---|---|
-| r / alpha / dropout | 32 / 64 / 0.05 |
-| target modules | regex scoped to `model.language_model.*` (q/k/v/o/gate/up/down proj) |
-| epochs / steps | 1 / 126 |
-| batch x grad-accum | 1 x 16 |
-| lr / schedule | 1e-4, cosine, 3% warmup, annealed to 0 |
-| max seq len / packing | **4096** / off |
-| `assistant_only_loss` | false |
-| seed | 0 |
-| trainable params | 159.4M |
-**Loss:** 2.7528 → **1.057**
-(epoch average); the last logged step (125) read
-1.0199. Epoch-average mean token accuracy
-**0.7071**, final grad_norm 0.3573,
-1,492,498 tokens consumed.
-Curves and the full log history are in
-[`LASR-Callum/2026-07-31-toolcalling-tulu-sft-run`](https://huggingface.co/datasets/LASR-Callum/2026-07-31-toolcalling-tulu-sft-run).
-### Why `max_seq_len` is 4096 and the sibling arms use 2048
-These agentic conversations run 9–13 turns with a median of 2,348 tokens, and 99 of the 151 source
-documents exceed 2048. Measured: a 2048 cap keeps only 80.4% of the corpus and **severs 11 of its
-98 `<tool_call>` spans**, inside exactly the long conversations the tool calls live in. At 4096 the
-mixture is truncated nowhere at all. The cost is that this arm differs from its siblings on one
-hyperparameter as well as on composition — read the head-to-head with that caveat.
-## Known caveats
-1. **Reasoning density.** Only **30 of the 124 agentic rows (24%) carry a real
-   reasoning trace**, against every target example in the difficult-advice 20/80 arm. If the
-   dose-response in this family is driven by reasoning rather than topic coverage, that is
-   confounded with the composition change here. Inherent to the source corpus.
-2. **`target_modules` only half-applies.** Qwen3.6-27B is hybrid: `q/k/v/o_proj` exist in 16 of 64
-   layers, the rest being linear-attention blocks with different module names. `gate/up/down_proj`
-   attach to all 64. So this adapter tunes MLP throughout but attention in only a quarter of the
-   stack. Same for every arm in the family, so comparisons are unaffected.
-3. **Not yet evaluated.** No ODCV-Bench or agentic-misalignment number exists for this arm yet.
-## Usage
-```python
-from peft import PeftModel
-from transformers import AutoModelForImageTextToText
-model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
-model = PeftModel.from_pretrained(model, "LASR-Callum/qwen3.6-27b-toolcalling-tulu-lora-20-80")
-model = model.merge_and_unload()  # vLLM LoRA support for this hybrid arch is unproven
-```
-Use `AutoModelForImageTextToText`, not `AutoModelForCausalLM` — this is a vision-language
-checkpoint. Merging drops the base model's 15 `mtp.*` tensors, so speculative decoding needs them
-grafted back.
-## Provenance
-Repository https://github.com/Matthew-Bozoukov/teaching_claude_why_replication @ `639d85c`.
-Built with `src/experiments/build_toolcalling_mixture.py`, trained with
-`src/experiments/train_lora.py --config configs/train_lora_toolcalling.yaml`.

+---
+base_model: Qwen/Qwen3.6-27B
+library_name: peft
+pipeline_tag: image-text-to-text
+tags:
+  - lora
+  - peft
+  - sft
+  - trl
+  - alignment
+  - agentic-misalignment
+  - tool-use
+datasets:
+  - LASR-Callum/2026-07-31-toolcalling-tulu-20-80-mixture
+  - LASR-Callum/2026-07-29-synthdoc-approved-constitution-sft
+  - LASR-Callum/2026-07-31-tulu3-replay-80-pct-qwen36-mixture
+license: apache-2.0
+---
+# Qwen3.6-27B — tool-calling + TULU3 LoRA (**20/80** mixture)
+LoRA adapter for [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B). The 20% target
+portion is **entirely agentic tool-use data** — conversations where the model itself is the actor
+holding live tools — and the other 80% is TULU3 replay.
+This is the pure-tool-calling cell of a family that holds total tokens and the 20% target share
+fixed and varies only the **composition** of that 20%:
+| Arm | 20% composition | ODCV-Bench | Agentic-misalignment |
+|---|---|---|---|
+| [base (no SFT)](https://huggingface.co/Qwen/Qwen3.6-27B) | — | 37.2% | 65.5% |
+| [`…-difficult-advice-tulu-lora-20-80`](https://huggingface.co/LASR-Callum/2026-07-28-qwen36-difficult-advice-tulu-lora-20-80) | difficult-advice only | 19.2% | 25.3% |
+| [`…-threeway-constitution-lora`](https://huggingface.co/LASR-Callum/2026-07-30-qwen36-threeway-constitution-lora) | equal thirds: embodied / difficult-advice / agentic | not yet run | not yet run |
+| **this** | **agentic tool-use only** | **not yet run** | **not yet run** |
+| [`…-tulu-100pct-lora`](https://huggingface.co/LASR-Callum/2026-07-30-qwen36-tulu-100-pct-lora) | none (zero-dose control) | — | — |
+## Training mixture
+Published in full as
+[`LASR-Callum/2026-07-31-toolcalling-tulu-20-80-mixture`](https://huggingface.co/datasets/LASR-Callum/2026-07-31-toolcalling-tulu-20-80-mixture).
+| Source | Examples | Tokens | Share | Rendering |
+|---|---:|---:|---:|---|
+| agentic tool-use (`approved_agentic`, `fullthink`) | 124 | 297,894 | 19.96% | reasoning kept where the source had it; **no** empty think blocks |
+| TULU3 replay | 1,878 | 1,194,548 | 80.04% | **no** `<think>` block at all |
+| **Total** | **2,002** | **1,492,442** | | |
+25 of the 124 agentic documents actually emit tool calls
+— 92 `<tool_call>` spans in Qwen3.6's XML dialect, all verified balanced.
+## Training
+bf16 LoRA (not QLoRA — bitsandbytes does not reliably cover this model's hybrid
+linear-attention/SSM layers), 1×H100 80GB SXM, **1h38m09s**.
+| | |
+|---|---|
+| r / alpha / dropout | 32 / 64 / 0.05 |
+| target modules | regex scoped to `model.language_model.*` (q/k/v/o/gate/up/down proj) |
+| epochs / steps | 1 / 126 |
+| batch x grad-accum | 1 x 16 |
+| lr / schedule | 1e-4, cosine, 3% warmup, annealed to 0 |
+| max seq len / packing | **4096** / off |
+| `assistant_only_loss` | false |
+| seed | 0 |
+| trainable params | 159.4M |
+**Loss:** 2.7528 → **1.057**
+(epoch average); the last logged step (125) read
+1.0199. Epoch-average mean token accuracy
+**0.7071**, final grad_norm 0.3573,
+1,492,498 tokens consumed.
+Curves and the full log history are in
+[`LASR-Callum/2026-07-31-toolcalling-tulu-sft-run`](https://huggingface.co/datasets/LASR-Callum/2026-07-31-toolcalling-tulu-sft-run).
+### Why `max_seq_len` is 4096 and the sibling arms use 2048
+These agentic conversations run 9–13 turns with a median of 2,348 tokens, and 99 of the 151 source
+documents exceed 2048. Measured: a 2048 cap keeps only 80.4% of the corpus and **severs 11 of its
+98 `<tool_call>` spans**, inside exactly the long conversations the tool calls live in. At 4096 the
+mixture is truncated nowhere at all. The cost is that this arm differs from its siblings on one
+hyperparameter as well as on composition — read the head-to-head with that caveat.
+## Known caveats
+1. **Reasoning density.** Only **30 of the 124 agentic rows (24%) carry a real
+   reasoning trace**, against every target example in the difficult-advice 20/80 arm. If the
+   dose-response in this family is driven by reasoning rather than topic coverage, that is
+   confounded with the composition change here. Inherent to the source corpus.
+2. **`target_modules` only half-applies.** Qwen3.6-27B is hybrid: `q/k/v/o_proj` exist in 16 of 64
+   layers, the rest being linear-attention blocks with different module names. `gate/up/down_proj`
+   attach to all 64. So this adapter tunes MLP throughout but attention in only a quarter of the
+   stack. Same for every arm in the family, so comparisons are unaffected.
+3. **Not yet evaluated.** No ODCV-Bench or agentic-misalignment number exists for this arm yet.
+## Usage
+```python
+from peft import PeftModel
+from transformers import AutoModelForImageTextToText
+model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
+model = PeftModel.from_pretrained(model, "LASR-Callum/2026-07-31-wrongly-trained-qwen36-toolcalling-tulu-lora-20-80")
+model = model.merge_and_unload()  # vLLM LoRA support for this hybrid arch is unproven
+```
+Use `AutoModelForImageTextToText`, not `AutoModelForCausalLM` — this is a vision-language
+checkpoint. Merging drops the base model's 15 `mtp.*` tensors, so speculative decoding needs them
+grafted back.
+## Provenance
+Repository https://github.com/Matthew-Bozoukov/teaching_claude_why_replication @ `639d85c`.
+Built with `src/experiments/build_toolcalling_mixture.py`, trained with
+`src/experiments/train_lora.py --config configs/train_lora_toolcalling.yaml`.