# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM
processor = AutoProcessor.from_pretrained("nanoswe/qwen-3.6-35b-a3b-teacher")
model = AutoModelForMultimodalLM.from_pretrained("nanoswe/qwen-3.6-35b-a3b-teacher", device_map="auto")
messages = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
{"type": "text", "text": "What animal is on the candy?"}
]
},
]
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))Qwen3.6-35B-A3B Teacher
A lightly fine-tuned variant of Qwen/Qwen3.6-35B-A3B, trained to be a better generator of agentic software-engineering trajectories. We use it as the teacher for distillation experiments: it produces the SWE-bench-style trajectory data that smaller student models are then trained on.
The fine-tune is deliberately light — one epoch at LR 6e-6 — because the goal is to sharpen the base model's trajectory format and success rate, not to move it off its pretrained distribution.
Training data
Self-distillation on the base model's own rollouts. Qwen3.6-35B-A3B was sampled on SWE-smith task instances, each trajectory was graded by the SWE-smith harness, and the surviving set was curated as follows:
- Correct only — the trajectory resolves its instance under recovered grading.
- No cheating — trajectories whose assistant turns match a cheat-pattern regex are dropped.
- No easy instances — instances solved by more than 75% of attempts (with at least 4 attempts) are dropped entirely, so the model does not spend its budget on problems it already solves.
- Top-3 shortest per instance — among the correct trajectories for a base instance, keep the three with the fewest tokens, which favors direct solutions over meandering ones.
- ≤ 32,768 tokens.
The result is 14,268 training trajectories.
Training
Full-parameter bf16 SFT, one epoch:
| Steps | 223 (1 epoch) |
| Batch size | 64 sequences |
| Sequence length | 32,768 |
| Learning rate | 6e-6, cosine, 5% warmup |
| Optimizer | FusedAdam (Transformer Engine), β = (0.9, 0.95), weight decay 0.0 |
| Gradient clipping | 1.0 |
| Seed | 1234 |
| Hardware | 8 × B200 |
| Stack | verl + NeMo-Automodel, FSDP2 with expert parallelism 8 |
Usage
Serve it as you would the base model. Two things differ:
- Use the bundled
chat_template.jinja. It emits plain<|im_start|>role\ncontent<|im_end|>turns with no<think>/</think>injection, which is what the training trajectories look like. - Thinking is off. The trajectories were generated with
enable_thinking=False, and the model was fine-tuned on them in that form.
Weights are bf16. Our own trajectory generation runs it under vLLM with online FP8 quantization.
License
Apache 2.0, inherited from the base model.
- Downloads last month
- 55
Model tree for nanoswe/qwen-3.6-35b-a3b-teacher
Base model
Qwen/Qwen3.6-35B-A3B
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="nanoswe/qwen-3.6-35b-a3b-teacher") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)