--- license: apache-2.0 base_model: Qwen/Qwen3.6-35B-A3B base_model_relation: quantized tags: - mlx - mlx-optiq - quantized - oq8 - mixed-precision - multi-token-prediction - mtp - moe - qwen3.6 pipeline_tag: text-generation library_name: mlx model-index: - name: Qwen3.6-35B-A3B-oQ8-mtp results: - task: type: text-generation name: Text Generation dataset: name: MMLU-Pro type: TIGER-Lab/MMLU-Pro config: sampled_300_of_12032 split: test metrics: - name: Accuracy type: accuracy value: 81 verified: false source: name: Self-reported (local eval, MTP off) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mmlu_pro.csv - task: type: text-generation name: Text Generation dataset: name: MathQA type: allenai/math_qa config: sampled_30_of_2985 split: test metrics: - name: Accuracy type: accuracy value: 93.3 verified: false source: name: Self-reported (local eval, MTP off) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mathqa.csv - task: type: text-generation name: Text Generation dataset: name: HumanEval type: openai_humaneval split: test metrics: - name: pass@1 type: pass@1 value: 96.3 verified: false source: name: Self-reported (local eval, MTP off) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_humaneval.csv - task: type: text-generation name: Text Generation dataset: name: MBPP type: mbpp config: sampled_200_of_500 split: test metrics: - name: pass@1 type: pass@1 value: 93.5 verified: false source: name: Self-reported (local eval, MTP off) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mbpp.csv - task: type: text-generation name: Text Generation dataset: name: LiveCodeBench type: livecodebench/code_generation_lite config: sampled_100_of_1055 split: test metrics: - name: pass@1 type: pass@1 value: 52 verified: false source: name: Self-reported (local eval, MTP off) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_livecodebench.csv - task: type: text-generation name: Text Generation dataset: name: MMLU-Pro type: TIGER-Lab/MMLU-Pro config: sampled_300_of_12032 split: test metrics: - name: Accuracy type: accuracy value: 82 verified: false source: name: Self-reported (local eval, MTP on) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mmlu_pro-mtp_on.csv - task: type: text-generation name: Text Generation dataset: name: MathQA type: allenai/math_qa config: sampled_30_of_2985 split: test metrics: - name: Accuracy type: accuracy value: 97 verified: false source: name: Self-reported (local eval, MTP on) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mathqa-mtp_on.csv - task: type: text-generation name: Text Generation dataset: name: HumanEval type: openai_humaneval split: test metrics: - name: pass@1 type: pass@1 value: 96 verified: false source: name: Self-reported (local eval, MTP on) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_humaneval-mtp_on.csv - task: type: text-generation name: Text Generation dataset: name: MBPP type: mbpp config: sampled_200_of_500 split: test metrics: - name: pass@1 type: pass@1 value: 94 verified: false source: name: Self-reported (local eval, MTP on) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mbpp-mtp_on.csv - task: type: text-generation name: Text Generation dataset: name: LiveCodeBench type: livecodebench/code_generation_lite config: sampled_100_of_1055 split: test metrics: - name: pass@1 type: pass@1 value: 53 verified: false source: name: Self-reported (local eval, MTP on) url: >- https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_livecodebench-mtp_on.csv --- # Qwen3.6-35B-A3B-oQ8-mtp An oQ8 mixed-precision quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) for [MLX](https://github.com/ml-explore/mlx), quantized with [mlx-optiq](https://github.com/ml-explore/mlx), with Multi-Token Prediction (MTP) heads grafted on from the reference `mtp.safetensors` structure. This repo also includes benchmark data comparing MTP on vs. off ([MTP Performance](#mtp-performance)) and quality benchmarks for the quantized checkpoint ([Intelligence Benchmarks](#intelligence-benchmarks)) below. Summary on MTP: in this setup, enabling it does not provide a token generation speedup, and in single-request decoding it is measurably slower. ## Model Details - **Base model:** [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Apache 2.0), a sparse MoE model with 35B total parameters and about 3B active parameters, with a 262,144 token native context extensible to about 1,010,000. - **Quantization method:** [mlx-optiq](https://github.com/ml-explore/mlx), oQ8, a mixed-precision quantization targeting about 8-bit average bits per weight, with sensitive layers kept at higher precision. - **MTP heads:** Grafted from the official Qwen3.6 reference `mtp.safetensors` structure (37 tensors, `group_size=64`), reverse-engineered to match the base model's MTP layer layout so the quantized checkpoint can run with speculative or MTP decoding enabled. - **Format:** MLX-native safetensors, quantized for use with an MLX inference server (tested with [oMLX](https://github.com/ml-explore/mlx)). - **Intended hardware:** Apple Silicon with unified memory (developed and tested on an M4 Max, 128GB). ## Usage ```bash # with an oMLX-compatible server omlx serve --model programmer-666/Qwen3.6-35B-A3B-oQ8-mtp ``` ```python # or load directly with mlx-lm from mlx_lm import load, generate model, tokenizer = load("programmer-666/Qwen3.6-35B-A3B-oQ8-mtp") response = generate( model, tokenizer, prompt="Explain multi-token prediction in one paragraph.", max_tokens=256, ) print(response) ``` To run without MTP, disable speculative decoding in your server config. The base weights are unaffected, and inference falls back to standard autoregressive decoding. ## MTP Performance Benchmarks below compare this checkpoint with MTP enabled vs. MTP disabled, on the same hardware (M4 Max, 128GB unified memory), same prompt and generation lengths, and same inference server. ### Single Request Decoding | Test | tg TPS (MTP On) | tg TPS (MTP Off) | Δ tg TPS | TTFT (MTP On, ms) | TTFT (MTP Off, ms) | Peak Mem (MTP On, GB) | Peak Mem (MTP Off, GB) | |---|---|---|---|---|---|---|---| | pp1024/tg128 | 81.5 | 83.0 | -1.8% | 810.8 | 817.2 | 36.22 | 35.38 | | pp4096/tg128 | 79.8 | 90.5 | -11.8% | 2491.8 | 2590.2 | 37.00 | 36.16 | | pp8192/tg128 | 74.8 | 90.2 | -17.1% | 5331.6 | 5235.2 | 37.34 | 36.50 | | pp16384/tg128 | 70.9 | 86.0 | -17.6% | 12701.4 | 12424.1 | 37.97 | 37.13 | | pp32768/tg128 | 64.7 | 77.6 | -16.6% | 31601.0 | 30047.3 | 39.31 | 38.47 | | pp65536/tg128 | 52.9 | 63.2 | -16.3% | 82607.3 | 80079.9 | 42.00 | 41.15 | | pp131072/tg128 | 32.6 | 36.8 | -11.4% | 328400.7 | 344694.8 | 47.37 | 46.53 | | pp200000/tg128 | 24.1 | 28.7 | -16.0% | 695737.6 | 700247.4 | 53.03 | 52.19 | - Token generation throughput (tg TPS) is consistently lower with MTP enabled, by an average of about 13.6% across context lengths, with the gap widening to 14 to 18% slower once context exceeds about 4K tokens. - Peak memory is about 2.1% higher with MTP enabled across the board (the extra MTP head weights and their activation buffers). - TTFT (prefill) is roughly a wash, about 0.3% average difference with no consistent direction. MTP heads don't participate in prefill, so this is expected and mostly run to run noise. ### Continuous Batching | Batch | tg TPS (MTP On) | tg TPS (MTP Off) | Δ tg TPS | Speedup (MTP On) | Speedup (MTP Off) | |---|---|---|---|---|---| | 1x | 81.5 | 83.0 | -1.8% | 1.00x | 1.00x | | 2x | 120.1 | 128.6 | -6.6% | 1.47x | 1.55x | | 4x | 133.8 | 138.5 | -3.4% | 1.64x | 1.67x | | 8x | 168.8 | 167.5 | +0.8% | 2.07x | 2.02x | Under continuous batching, the gap mostly closes. MTP is within noise of the no-MTP baseline (-6.6% to +0.8%), with no consistent advantage at any batch size. ### Takeaway MTP head grafting is functional (weights load, model runs, output is correct), but in this MLX/oMLX inference setup, enabling MTP does not deliver a speculative decoding speedup, and for single-request decoding it does the opposite. This is most likely because the MTP head isn't yet wired into an actual speculative decode/verify loop in the serving stack; it currently just adds a forward pass and memory overhead without skipping any accept-reject cycle. Treat the MTP tensors in this repo as a structurally correct starting point for that integration work, not as a ready speedup. If you get MTP-accelerated decoding working with this checkpoint on a different server or runtime, opening a discussion on this repo with your results would be very welcome. ## Intelligence Benchmarks Quality checks run on this quantized checkpoint with MTP disabled (thinking mode enabled for all runs). Equivalent runs with MTP enabled are in progress and will be added once complete; MTP only affects decoding strategy, not the model's weights, so these numbers reflect the checkpoint's underlying quality regardless of MTP status, but a direct on/off comparison isn't available yet. These results are also embedded as structured `model-index` metadata in this README's YAML header, so they will surface in the Hub's evaluation results widget on the model page. Several benchmarks use a sampled subset rather than the full set, so treat MathQA and LiveCodeBench results (30 and 100 samples respectively) as indicative rather than tight estimates. MMLU-Pro, MBPP, and especially the full-set HumanEval run are more reliable. | Benchmark | Accuracy | Correct / Total | Sampled | Time (s) | Thinking | |---|---|---|---|---|---| | MMLU-Pro | 81.0% | 243 / 300 | 300 / 12,032 | 8,867.2 | Yes | | MathQA | 93.3% | 28 / 30 | 30 / 2,985 | 1,113.5 | Yes | | HumanEval | 96.3% | 158 / 164 | 164 / 164 (full) | 5,940.7 | Yes | | MBPP | 93.5% | 187 / 200 | 200 / 500 | 7,483.3 | Yes | | LiveCodeBench | 52.0% | 52 / 100 | 100 / 1,055 | 20,696.7 | Yes | Quantization to oQ8 preserves strong coding performance (HumanEval, MBPP) and general knowledge (MMLU-Pro) close to what's expected of the base model. LiveCodeBench, the hardest and most contamination-resistant of these, comes in noticeably lower, which lines up with it being the more demanding, competition-style coding benchmark rather than being oQ8-specific degradation. A same-precision base-model run on the same subset would be needed to isolate a quantization effect there. ## Quantization Notes - Quantized with `mlx-optiq` targeting oQ8 mixed precision. Comparable full-precision/oQ7 runs on the same base architecture landed around 37GB, so expect a similar footprint here (see [Files and versions](.) for exact sizes). - No calibration or fine-tuning was performed beyond quantization and MTP head grafting; this is a direct weight-space conversion of the base model. ## License This is a derivative of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), released under the Apache 2.0 license. See the base model card for full terms, intended use, and limitations.