Create README.md

README.md

@@ -0,0 +1,198 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.6-35B-A3B
+base_model_relation: quantized
+tags:
+- mlx
+- mlx-optiq
+- quantized
+- oq8
+- mixed-precision
+- multi-token-prediction
+- mtp
+- moe
+- qwen3.6
+pipeline_tag: text-generation
+library_name: mlx
+model-index:
+- name: Qwen3.6-35B-A3B-oQ8-mtp
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MMLU-Pro
+      type: TIGER-Lab/MMLU-Pro
+      config: sampled_300_of_12032
+      split: test
+    metrics:
+    - name: Accuracy
+      type: accuracy
+      value: 81.0
+      verified: false
+    source:
+      name: Self-reported (local eval, M4 Max 128GB, MTP off)
+      url: https://huggingface.co/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MathQA
+      type: allenai/math_qa
+      config: sampled_30_of_2985
+      split: test
+    metrics:
+    - name: Accuracy
+      type: accuracy
+      value: 93.3
+      verified: false
+    source:
+      name: Self-reported (local eval, M4 Max 128GB, MTP off)
+      url: https://huggingface.co/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: HumanEval
+      type: openai_humaneval
+      split: test
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 96.3
+      verified: false
+    source:
+      name: Self-reported (local eval, M4 Max 128GB, MTP off)
+      url: https://huggingface.co/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MBPP
+      type: mbpp
+      config: sampled_200_of_500
+      split: test
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 93.5
+      verified: false
+    source:
+      name: Self-reported (local eval, M4 Max 128GB, MTP off)
+      url: https://huggingface.co/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: LiveCodeBench
+      type: livecodebench/code_generation_lite
+      config: sampled_100_of_1055
+      split: test
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 52.0
+      verified: false
+    source:
+      name: Self-reported (local eval, M4 Max 128GB, MTP off)
+      url: https://huggingface.co/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp
+---
+
+# Qwen3.6-35B-A3B-oQ8-mtp
+
+An oQ8 mixed-precision quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) for [MLX](https://github.com/ml-explore/mlx), quantized with [mlx-optiq](https://github.com/ml-explore/mlx), with Multi-Token Prediction (MTP) heads grafted on from the reference `mtp.safetensors` structure.
+
+This repo also includes benchmark data comparing MTP on vs. off ([MTP Performance](#mtp-performance)) and quality benchmarks for the quantized checkpoint ([Intelligence Benchmarks](#intelligence-benchmarks)) below. Summary on MTP: in this setup, enabling it does not provide a token generation speedup, and in single-request decoding it is measurably slower.
+
+## Model Details
+
+- **Base model:** [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Apache 2.0), a sparse MoE model with 35B total parameters and about 3B active parameters, with a 262,144 token native context extensible to about 1,010,000.
+- **Quantization method:** [mlx-optiq](https://github.com/ml-explore/mlx), oQ8, a mixed-precision quantization targeting about 8-bit average bits per weight, with sensitive layers kept at higher precision.
+- **MTP heads:** Grafted from the official Qwen3.6 reference `mtp.safetensors` structure (37 tensors, `group_size=64`), reverse-engineered to match the base model's MTP layer layout so the quantized checkpoint can run with speculative or MTP decoding enabled.
+- **Format:** MLX-native safetensors, quantized for use with an MLX inference server (tested with [oMLX](https://github.com/ml-explore/mlx)).
+- **Intended hardware:** Apple Silicon with unified memory (developed and tested on an M4 Max, 128GB).
+
+## Usage
+
+```bash
+# with an oMLX-compatible server
+omlx serve --model programmer-666/Qwen3.6-35B-A3B-oQ8-mtp
+```
+
+```python
+# or load directly with mlx-lm
+from mlx_lm import load, generate
+
+model, tokenizer = load("programmer-666/Qwen3.6-35B-A3B-oQ8-mtp")
+response = generate(
+    model, tokenizer,
+    prompt="Explain multi-token prediction in one paragraph.",
+    max_tokens=256,
+)
+print(response)
+```
+
+To run without MTP, disable speculative decoding in your server config. The base weights are unaffected, and inference falls back to standard autoregressive decoding.
+
+## MTP Performance
+
+Benchmarks below compare this checkpoint with MTP enabled vs. MTP disabled, on the same hardware (M4 Max, 128GB unified memory), same prompt and generation lengths, and same inference server.
+
+### Single Request Decoding
+
+| Test | tg TPS (MTP On) | tg TPS (MTP Off) | Δ tg TPS | TTFT (MTP On, ms) | TTFT (MTP Off, ms) | Peak Mem (MTP On, GB) | Peak Mem (MTP Off, GB) |
+|---|---|---|---|---|---|---|---|
+| pp1024/tg128 | 81.5 | 83.0 | -1.8% | 810.8 | 817.2 | 36.22 | 35.38 |
+| pp4096/tg128 | 79.8 | 90.5 | -11.8% | 2491.8 | 2590.2 | 37.00 | 36.16 |
+| pp8192/tg128 | 74.8 | 90.2 | -17.1% | 5331.6 | 5235.2 | 37.34 | 36.50 |
+| pp16384/tg128 | 70.9 | 86.0 | -17.6% | 12701.4 | 12424.1 | 37.97 | 37.13 |
+| pp32768/tg128 | 64.7 | 77.6 | -16.6% | 31601.0 | 30047.3 | 39.31 | 38.47 |
+| pp65536/tg128 | 52.9 | 63.2 | -16.3% | 82607.3 | 80079.9 | 42.00 | 41.15 |
+| pp131072/tg128 | 32.6 | 36.8 | -11.4% | 328400.7 | 344694.8 | 47.37 | 46.53 |
+| pp200000/tg128 | 24.1 | 28.7 | -16.0% | 695737.6 | 700247.4 | 53.03 | 52.19 |
+
+- Token generation throughput (tg TPS) is consistently lower with MTP enabled, by an average of about 13.6% across context lengths, with the gap widening to 14 to 18% slower once context exceeds about 4K tokens.
+- Peak memory is about 2.1% higher with MTP enabled across the board (the extra MTP head weights and their activation buffers).
+- TTFT (prefill) is roughly a wash, about 0.3% average difference with no consistent direction. MTP heads don't participate in prefill, so this is expected and mostly run to run noise.
+
+### Continuous Batching
+
+| Batch | tg TPS (MTP On) | tg TPS (MTP Off) | Δ tg TPS | Speedup (MTP On) | Speedup (MTP Off) |
+|---|---|---|---|---|---|
+| 1x | 81.5 | 83.0 | -1.8% | 1.00x | 1.00x |
+| 2x | 120.1 | 128.6 | -6.6% | 1.47x | 1.55x |
+| 4x | 133.8 | 138.5 | -3.4% | 1.64x | 1.67x |
+| 8x | 168.8 | 167.5 | +0.8% | 2.07x | 2.02x |
+
+Under continuous batching, the gap mostly closes. MTP is within noise of the no-MTP baseline (-6.6% to +0.8%), with no consistent advantage at any batch size.
+
+### Takeaway
+
+MTP head grafting is functional (weights load, model runs, output is correct), but in this MLX/oMLX inference setup, enabling MTP does not deliver a speculative decoding speedup, and for single-request decoding it does the opposite. This is most likely because the MTP head isn't yet wired into an actual speculative decode/verify loop in the serving stack; it currently just adds a forward pass and memory overhead without skipping any accept-reject cycle. Treat the MTP tensors in this repo as a structurally correct starting point for that integration work, not as a ready speedup.
+
+If you get MTP-accelerated decoding working with this checkpoint on a different server or runtime, opening a discussion on this repo with your results would be very welcome.
+
+## Intelligence Benchmarks
+
+Quality checks run on this quantized checkpoint with MTP disabled (thinking mode enabled for all runs). Equivalent runs with MTP enabled are in progress and will be added once complete; MTP only affects decoding strategy, not the model's weights, so these numbers reflect the checkpoint's underlying quality regardless of MTP status, but a direct on/off comparison isn't available yet.
+
+These results are also embedded as structured `model-index` metadata in this README's YAML header, so they will surface in the Hub's evaluation results widget on the model page. Several benchmarks use a sampled subset rather than the full set, so treat MathQA and LiveCodeBench results (30 and 100 samples respectively) as indicative rather than tight estimates. MMLU-Pro, MBPP, and especially the full-set HumanEval run are more reliable.
+
+| Benchmark | Accuracy | Correct / Total | Sampled | Time (s) | Thinking |
+|---|---|---|---|---|---|
+| MMLU-Pro | 81.0% | 243 / 300 | 300 / 12,032 | 8,867.2 | Yes |
+| MathQA | 93.3% | 28 / 30 | 30 / 2,985 | 1,113.5 | Yes |
+| HumanEval | 96.3% | 158 / 164 | 164 / 164 (full) | 5,940.7 | Yes |
+| MBPP | 93.5% | 187 / 200 | 200 / 500 | 7,483.3 | Yes |
+| LiveCodeBench | 52.0% | 52 / 100 | 100 / 1,055 | 20,696.7 | Yes |
+
+Quantization to oQ8 preserves strong coding performance (HumanEval, MBPP) and general knowledge (MMLU-Pro) close to what's expected of the base model. LiveCodeBench, the hardest and most contamination-resistant of these, comes in noticeably lower, which lines up with it being the more demanding, competition-style coding benchmark rather than being oQ8-specific degradation. A same-precision base-model run on the same subset would be needed to isolate a quantization effect there.
+
+## Quantization Notes
+
+- Quantized with `mlx-optiq` targeting oQ8 mixed precision. Comparable full-precision/oQ7 runs on the same base architecture landed around 37GB, so expect a similar footprint here (see [Files and versions](.) for exact sizes).
+- No calibration or fine-tuning was performed beyond quantization and MTP head grafting; this is a direct weight-space conversion of the base model.
+
+## License
+
+This is a derivative of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), released under the Apache 2.0 license. See the base model card for full terms, intended use, and limitations.