README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
base_model_relation: quantized
tags:
- mlx
- mlx-optiq
- quantized
- oq8
- mixed-precision
- multi-token-prediction
- mtp
- moe
- qwen3.6
pipeline_tag: text-generation
library_name: mlx
model-index:
- name: Qwen3.6-35B-A3B-oQ8-mtp
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU-Pro
      type: TIGER-Lab/MMLU-Pro
      config: sampled_300_of_12032
      split: test
    metrics:
    - name: Accuracy
      type: accuracy
      value: 81
      verified: false
    source:
      name: Self-reported (local eval, MTP off)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mmlu_pro.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MathQA
      type: allenai/math_qa
      config: sampled_30_of_2985
      split: test
    metrics:
    - name: Accuracy
      type: accuracy
      value: 93.3
      verified: false
    source:
      name: Self-reported (local eval, MTP off)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mathqa.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HumanEval
      type: openai_humaneval
      split: test
    metrics:
    - name: pass@1
      type: pass@1
      value: 96.3
      verified: false
    source:
      name: Self-reported (local eval, MTP off)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_humaneval.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MBPP
      type: mbpp
      config: sampled_200_of_500
      split: test
    metrics:
    - name: pass@1
      type: pass@1
      value: 93.5
      verified: false
    source:
      name: Self-reported (local eval, MTP off)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mbpp.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: LiveCodeBench
      type: livecodebench/code_generation_lite
      config: sampled_100_of_1055
      split: test
    metrics:
    - name: pass@1
      type: pass@1
      value: 52
      verified: false
    source:
      name: Self-reported (local eval, MTP off)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_livecodebench.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU-Pro
      type: TIGER-Lab/MMLU-Pro
      config: sampled_300_of_12032
      split: test
    metrics:
    - name: Accuracy
      type: accuracy
      value: 82
      verified: false
    source:
      name: Self-reported (local eval, MTP on)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mmlu_pro-mtp_on.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MathQA
      type: allenai/math_qa
      config: sampled_30_of_2985
      split: test
    metrics:
    - name: Accuracy
      type: accuracy
      value: 97
      verified: false
    source:
      name: Self-reported (local eval, MTP on)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mathqa-mtp_on.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HumanEval
      type: openai_humaneval
      split: test
    metrics:
    - name: pass@1
      type: pass@1
      value: 96
      verified: false
    source:
      name: Self-reported (local eval, MTP on)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_humaneval-mtp_on.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MBPP
      type: mbpp
      config: sampled_200_of_500
      split: test
    metrics:
    - name: pass@1
      type: pass@1
      value: 94
      verified: false
    source:
      name: Self-reported (local eval, MTP on)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mbpp-mtp_on.csv
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: LiveCodeBench
      type: livecodebench/code_generation_lite
      config: sampled_100_of_1055
      split: test
    metrics:
    - name: pass@1
      type: pass@1
      value: 53
      verified: false
    source:
      name: Self-reported (local eval, MTP on)
      url: >-
        https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_livecodebench-mtp_on.csv
---


# Qwen3.6-35B-A3B-oQ8-mtp

An oQ8 mixed-precision quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) for [MLX](https://github.com/ml-explore/mlx), quantized with [mlx-optiq](https://github.com/ml-explore/mlx), with Multi-Token Prediction (MTP) heads grafted on from the reference `mtp.safetensors` structure.

This repo also includes benchmark data comparing MTP on vs. off ([MTP Performance](#mtp-performance)) and quality benchmarks for the quantized checkpoint ([Intelligence Benchmarks](#intelligence-benchmarks)) below. Summary on MTP: in this setup, enabling it does not provide a token generation speedup, and in single-request decoding it is measurably slower.

## Model Details

- **Base model:** [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Apache 2.0), a sparse MoE model with 35B total parameters and about 3B active parameters, with a 262,144 token native context extensible to about 1,010,000.
- **Quantization method:** [mlx-optiq](https://github.com/ml-explore/mlx), oQ8, a mixed-precision quantization targeting about 8-bit average bits per weight, with sensitive layers kept at higher precision.
- **MTP heads:** Grafted from the official Qwen3.6 reference `mtp.safetensors` structure (37 tensors, `group_size=64`), reverse-engineered to match the base model's MTP layer layout so the quantized checkpoint can run with speculative or MTP decoding enabled.
- **Format:** MLX-native safetensors, quantized for use with an MLX inference server (tested with [oMLX](https://github.com/ml-explore/mlx)).
- **Intended hardware:** Apple Silicon with unified memory (developed and tested on an M4 Max, 128GB).

## Usage

```bash
# with an oMLX-compatible server
omlx serve --model programmer-666/Qwen3.6-35B-A3B-oQ8-mtp
```

```python
# or load directly with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("programmer-666/Qwen3.6-35B-A3B-oQ8-mtp")
response = generate(
    model, tokenizer,
    prompt="Explain multi-token prediction in one paragraph.",
    max_tokens=256,
)
print(response)
```

To run without MTP, disable speculative decoding in your server config. The base weights are unaffected, and inference falls back to standard autoregressive decoding.

## MTP Performance

Benchmarks below compare this checkpoint with MTP enabled vs. MTP disabled, on the same hardware (M4 Max, 128GB unified memory), same prompt and generation lengths, and same inference server.

### Single Request Decoding

| Test | tg TPS (MTP On) | tg TPS (MTP Off) | Δ tg TPS | TTFT (MTP On, ms) | TTFT (MTP Off, ms) | Peak Mem (MTP On, GB) | Peak Mem (MTP Off, GB) |
|---|---|---|---|---|---|---|---|
| pp1024/tg128 | 81.5 | 83.0 | -1.8% | 810.8 | 817.2 | 36.22 | 35.38 |
| pp4096/tg128 | 79.8 | 90.5 | -11.8% | 2491.8 | 2590.2 | 37.00 | 36.16 |
| pp8192/tg128 | 74.8 | 90.2 | -17.1% | 5331.6 | 5235.2 | 37.34 | 36.50 |
| pp16384/tg128 | 70.9 | 86.0 | -17.6% | 12701.4 | 12424.1 | 37.97 | 37.13 |
| pp32768/tg128 | 64.7 | 77.6 | -16.6% | 31601.0 | 30047.3 | 39.31 | 38.47 |
| pp65536/tg128 | 52.9 | 63.2 | -16.3% | 82607.3 | 80079.9 | 42.00 | 41.15 |
| pp131072/tg128 | 32.6 | 36.8 | -11.4% | 328400.7 | 344694.8 | 47.37 | 46.53 |
| pp200000/tg128 | 24.1 | 28.7 | -16.0% | 695737.6 | 700247.4 | 53.03 | 52.19 |

- Token generation throughput (tg TPS) is consistently lower with MTP enabled, by an average of about 13.6% across context lengths, with the gap widening to 14 to 18% slower once context exceeds about 4K tokens.
- Peak memory is about 2.1% higher with MTP enabled across the board (the extra MTP head weights and their activation buffers).
- TTFT (prefill) is roughly a wash, about 0.3% average difference with no consistent direction. MTP heads don't participate in prefill, so this is expected and mostly run to run noise.

### Continuous Batching

| Batch | tg TPS (MTP On) | tg TPS (MTP Off) | Δ tg TPS | Speedup (MTP On) | Speedup (MTP Off) |
|---|---|---|---|---|---|
| 1x | 81.5 | 83.0 | -1.8% | 1.00x | 1.00x |
| 2x | 120.1 | 128.6 | -6.6% | 1.47x | 1.55x |
| 4x | 133.8 | 138.5 | -3.4% | 1.64x | 1.67x |
| 8x | 168.8 | 167.5 | +0.8% | 2.07x | 2.02x |

Under continuous batching, the gap mostly closes. MTP is within noise of the no-MTP baseline (-6.6% to +0.8%), with no consistent advantage at any batch size.

### Takeaway

MTP head grafting is functional (weights load, model runs, output is correct), but in this MLX/oMLX inference setup, enabling MTP does not deliver a speculative decoding speedup, and for single-request decoding it does the opposite. This is most likely because the MTP head isn't yet wired into an actual speculative decode/verify loop in the serving stack; it currently just adds a forward pass and memory overhead without skipping any accept-reject cycle. Treat the MTP tensors in this repo as a structurally correct starting point for that integration work, not as a ready speedup.

If you get MTP-accelerated decoding working with this checkpoint on a different server or runtime, opening a discussion on this repo with your results would be very welcome.

## Intelligence Benchmarks

Quality checks run on this quantized checkpoint with MTP disabled (thinking mode enabled for all runs). Equivalent runs with MTP enabled are in progress and will be added once complete; MTP only affects decoding strategy, not the model's weights, so these numbers reflect the checkpoint's underlying quality regardless of MTP status, but a direct on/off comparison isn't available yet.

These results are also embedded as structured `model-index` metadata in this README's YAML header, so they will surface in the Hub's evaluation results widget on the model page. Several benchmarks use a sampled subset rather than the full set, so treat MathQA and LiveCodeBench results (30 and 100 samples respectively) as indicative rather than tight estimates. MMLU-Pro, MBPP, and especially the full-set HumanEval run are more reliable.

| Benchmark | Accuracy | Correct / Total | Sampled | Time (s) | Thinking |
|---|---|---|---|---|---|
| MMLU-Pro | 81.0% | 243 / 300 | 300 / 12,032 | 8,867.2 | Yes |
| MathQA | 93.3% | 28 / 30 | 30 / 2,985 | 1,113.5 | Yes |
| HumanEval | 96.3% | 158 / 164 | 164 / 164 (full) | 5,940.7 | Yes |
| MBPP | 93.5% | 187 / 200 | 200 / 500 | 7,483.3 | Yes |
| LiveCodeBench | 52.0% | 52 / 100 | 100 / 1,055 | 20,696.7 | Yes |

Quantization to oQ8 preserves strong coding performance (HumanEval, MBPP) and general knowledge (MMLU-Pro) close to what's expected of the base model. LiveCodeBench, the hardest and most contamination-resistant of these, comes in noticeably lower, which lines up with it being the more demanding, competition-style coding benchmark rather than being oQ8-specific degradation. A same-precision base-model run on the same subset would be needed to isolate a quantization effect there.

## Quantization Notes

- Quantized with `mlx-optiq` targeting oQ8 mixed precision. Comparable full-precision/oQ7 runs on the same base architecture landed around 37GB, so expect a similar footprint here (see [Files and versions](.) for exact sizes).
- No calibration or fine-tuning was performed beyond quantization and MTP head grafting; this is a direct weight-space conversion of the base model.

## License

This is a derivative of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), released under the Apache 2.0 license. See the base model card for full terms, intended use, and limitations.