---
license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - empero-ai
  - qwen3.6
  - qwen3.8
  - distillation
  - reasoning
  - function-calling
  - moe
  - sft
---

# Qwen3.8-35B-A3B

**Developed by [Empero](https://empero.org)**

> [!Note]
> This repository contains model weights and configuration files in the Hugging Face Transformers format.
>
> These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, and other standard runtimes with Qwen3.6 architecture support.

**Qwen3.8-35B-A3B** is a distillation of the **Qwen3.8** frontier models into the Qwen3.6-35B-A3B Mixture-of-Experts architecture. The student was trained on curated teacher traces from our internal Qwen3.8 distillation datasets — dense chain-of-thought spanning mathematics, code, general reasoning, instruction following, and tool use, quality-filtered before training.

The objective: bring the reasoning behavior of frontier-scale teachers into a sparse 35B that activates only 3B parameters per token and deploys on a single GPU.

## Highlights

- **Distilled chain-of-thought** — every answer opens with a `<think>` block learned directly from Qwen3.8 teacher traces rather than synthetic self-generated reasoning.
- **Mathematics and code emphasis** — the trace mix is deliberately weighted toward hard math and competitive programming, the domains where distillation moves the needle most at this scale.
- **Sparse MoE efficiency** — 35B total parameters, ~3B active per token; 256 experts with 8 routed per token.
- **Attention and experts both adapted** — our internal MoE training pipeline updates the attention path and the routed and shared expert stacks, not just attention.
- **Native function calling** per Qwen3.6's specification — no wrapper or tool-specific fine-tune required.
- **262,144-token native context**, inherited from the Qwen3.6 base.

## Model Overview

- Type: Causal Language Model (text path of a vision-language base)
- Base: [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
- Number of Parameters: 35B total / ~3B active per token
- Architecture: 40 layers, 256 experts, 8 experts per token, hybrid linear + full attention
- Training: SFT (off-policy distillation) on curated teacher traces via our internal MoE training pipeline
- Teachers: Qwen3.8 2.4T A95B and Qwen3.8 Flash Next (internal distillation datasets)
- Context Length: 262,144 natively

## Benchmark Results

Measured with [`lm-evaluation-harness`](https://github.com/EleutherAI/lm-evaluation-harness), HF backend, bfloat16, identical settings and seed for base and student. Zero-shot, loglikelihood scoring.

| Task | Metric | Qwen3.6-35B-A3B (base) | **Qwen3.8-35B-A3B** | Δ |
|---|---|---:|---:|---:|
| MMLU (57 subjects) | acc | 0.838 | **0.834** | −0.004 |
| ARC-Challenge | acc | 0.548 | **0.582** | **+0.034** |
| ARC-Challenge | acc_norm | 0.548 | **0.591** | **+0.044** |
| ARC-Easy | acc | 0.819 | **0.830** | **+0.011** |
| ARC-Easy | acc_norm | 0.717 | **0.766** | **+0.048** |

The MMLU difference is within noise (standard error 0.003 on each measurement). The ARC gains are outside it.

## Quickstart

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "empero-ai/Qwen3.8-35B-A3B-Distilled"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "A snail is at the bottom of a 10-meter well. Each day it climbs 3 meters, each night it slips back 2. How many days until it escapes?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True).to(model.device)

out = model.generate(**inputs, max_new_tokens=16384,
                     temperature=0.6, top_p=0.95, top_k=20, do_sample=True)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

A recent `transformers` release with Qwen3.6 support is required, along with the Gated DeltaNet kernels ([`flash-linear-attention`](https://github.com/fla-org/flash-linear-attention) and a CUDA-matched [`causal_conv1d`](https://github.com/Dao-AILab/causal-conv1d) build) — without them the linear-attention layers fall back to slow, memory-hungry PyTorch ops.

`AutoModelForCausalLM` loads the text path (34.7B parameters). The vision tower is retained in the checkpoint and is reachable via `AutoModelForImageTextToText`.

## Best Practices

- **Sampling**: `temperature=0.6, top_p=0.95, top_k=20`. Greedy decoding on long generations is a known repetition-loop failure mode for reasoning models in this class.
- **Output length**: allow generous `max_new_tokens` (16,384 recommended); every answer opens with a `<think>` block. Parse and strip the `<think>...</think>` span for end users.
- **Scope**: the model learned from teacher traces, not from its own rollouts — it inherits the teachers' reasoning style, including occasional over-long deliberation on easy questions.

## Limitations

- **Shorter responses.** The student was trained on 8,192-token examples and produces noticeably shorter outputs than the base. Long chains of thought are more likely to be cut short, so behaviour on long-form generation and long-context workloads may be degraded relative to the base.
- **A v2 is in training** with longer-context support, aimed squarely at the point above.
- **Vision is untouched.** The fine-tune is text-only; vision behaviour is inherited from the base and was not evaluated.

## Stay in the loop

Sign up for the Empero newsletter at **[empero.org](https://empero.org)** for releases, evals, and research notes.

## Support / Donate

If this model helped you, consider supporting the project:

- **BTC**: `bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v`
- **LTC**: `ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x`

---

## Provenance & licensing

Weights are released under **Apache-2.0**, inherited from the Qwen3.6-35B-A3B base. Shared for research and experimentation, as-is.

## Acknowledgements

- Developed and released by [Empero](https://empero.org)
- Base model: [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Alibaba Qwen team)
- Teachers: Qwen3.8 2.4T A95B and Qwen3.8 Flash Next
- Training: [TRL](https://github.com/huggingface/trl) + [Transformers](https://github.com/huggingface/transformers)
- Linear-attention kernels: [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), [causal_conv1d](https://github.com/Dao-AILab/causal-conv1d)
- Evaluation: [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) (EleutherAI)
- **Thanks to everyone who used our free community endpoint** — a portion of the prompts used to generate this model's synthetic training data came from those interactions.