Instructions to use programmer-666/Qwen3.6-35B-A3B-oQ8-mtp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use programmer-666/Qwen3.6-35B-A3B-oQ8-mtp with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("programmer-666/Qwen3.6-35B-A3B-oQ8-mtp")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use programmer-666/Qwen3.6-35B-A3B-oQ8-mtp with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use programmer-666/Qwen3.6-35B-A3B-oQ8-mtp with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use programmer-666/Qwen3.6-35B-A3B-oQ8-mtp with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default programmer-666/Qwen3.6-35B-A3B-oQ8-mtp

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use programmer-666/Qwen3.6-35B-A3B-oQ8-mtp with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "programmer-666/Qwen3.6-35B-A3B-oQ8-mtp" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.6-35B-A3B-oQ8-mtp / README.md

programmer-666

Update README.md

e9d49e2 verified about 2 months ago

preview code

Raw

History Blame Contribute Delete

13.1 kB

metadata

license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
base_model_relation: quantized
tags:
  - mlx
  - mlx-optiq
  - quantized
  - oq8
  - mixed-precision
  - multi-token-prediction
  - mtp
  - moe
  - qwen3.6
pipeline_tag: text-generation
library_name: mlx
model-index:
  - name: Qwen3.6-35B-A3B-oQ8-mtp
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU-Pro
          type: TIGER-Lab/MMLU-Pro
          config: sampled_300_of_12032
          split: test
        metrics:
          - name: Accuracy
            type: accuracy
            value: 81
            verified: false
        source:
          name: Self-reported (local eval, MTP off)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mmlu_pro.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MathQA
          type: allenai/math_qa
          config: sampled_30_of_2985
          split: test
        metrics:
          - name: Accuracy
            type: accuracy
            value: 93.3
            verified: false
        source:
          name: Self-reported (local eval, MTP off)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mathqa.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HumanEval
          type: openai_humaneval
          split: test
        metrics:
          - name: pass@1
            type: pass@1
            value: 96.3
            verified: false
        source:
          name: Self-reported (local eval, MTP off)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_humaneval.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MBPP
          type: mbpp
          config: sampled_200_of_500
          split: test
        metrics:
          - name: pass@1
            type: pass@1
            value: 93.5
            verified: false
        source:
          name: Self-reported (local eval, MTP off)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mbpp.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: LiveCodeBench
          type: livecodebench/code_generation_lite
          config: sampled_100_of_1055
          split: test
        metrics:
          - name: pass@1
            type: pass@1
            value: 52
            verified: false
        source:
          name: Self-reported (local eval, MTP off)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_off_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_livecodebench.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU-Pro
          type: TIGER-Lab/MMLU-Pro
          config: sampled_300_of_12032
          split: test
        metrics:
          - name: Accuracy
            type: accuracy
            value: 82
            verified: false
        source:
          name: Self-reported (local eval, MTP on)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mmlu_pro-mtp_on.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MathQA
          type: allenai/math_qa
          config: sampled_30_of_2985
          split: test
        metrics:
          - name: Accuracy
            type: accuracy
            value: 97
            verified: false
        source:
          name: Self-reported (local eval, MTP on)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mathqa-mtp_on.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HumanEval
          type: openai_humaneval
          split: test
        metrics:
          - name: pass@1
            type: pass@1
            value: 96
            verified: false
        source:
          name: Self-reported (local eval, MTP on)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_humaneval-mtp_on.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MBPP
          type: mbpp
          config: sampled_200_of_500
          split: test
        metrics:
          - name: pass@1
            type: pass@1
            value: 94
            verified: false
        source:
          name: Self-reported (local eval, MTP on)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_mbpp-mtp_on.csv
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: LiveCodeBench
          type: livecodebench/code_generation_lite
          config: sampled_100_of_1055
          split: test
        metrics:
          - name: pass@1
            type: pass@1
            value: 53
            verified: false
        source:
          name: Self-reported (local eval, MTP on)
          url: >-
            https://github.com/programmer-666/llm-evaluations-hf/blob/main/oMLX/programmer-666/Qwen3.6-35B-A3B-oQ8-mtp/mtp_on_intelligence_evaluation_results/Qwen3.6-35B-A3B-oQ8-mtp_livecodebench-mtp_on.csv

Qwen3.6-35B-A3B-oQ8-mtp

An oQ8 mixed-precision quantization of Qwen/Qwen3.6-35B-A3B for MLX, quantized with mlx-optiq, with Multi-Token Prediction (MTP) heads grafted on from the reference mtp.safetensors structure.

This repo also includes benchmark data comparing MTP on vs. off (MTP Performance) and quality benchmarks for the quantized checkpoint (Intelligence Benchmarks) below. Summary on MTP: in this setup, enabling it does not provide a token generation speedup, and in single-request decoding it is measurably slower.

Model Details

Base model: Qwen/Qwen3.6-35B-A3B (Apache 2.0), a sparse MoE model with 35B total parameters and about 3B active parameters, with a 262,144 token native context extensible to about 1,010,000.
Quantization method: mlx-optiq, oQ8, a mixed-precision quantization targeting about 8-bit average bits per weight, with sensitive layers kept at higher precision.
MTP heads: Grafted from the official Qwen3.6 reference mtp.safetensors structure (37 tensors, group_size=64), reverse-engineered to match the base model's MTP layer layout so the quantized checkpoint can run with speculative or MTP decoding enabled.
Format: MLX-native safetensors, quantized for use with an MLX inference server (tested with oMLX).
Intended hardware: Apple Silicon with unified memory (developed and tested on an M4 Max, 128GB).

Usage

# with an oMLX-compatible server
omlx serve --model programmer-666/Qwen3.6-35B-A3B-oQ8-mtp

# or load directly with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("programmer-666/Qwen3.6-35B-A3B-oQ8-mtp")
response = generate(
    model, tokenizer,
    prompt="Explain multi-token prediction in one paragraph.",
    max_tokens=256,
)
print(response)

To run without MTP, disable speculative decoding in your server config. The base weights are unaffected, and inference falls back to standard autoregressive decoding.

MTP Performance

Benchmarks below compare this checkpoint with MTP enabled vs. MTP disabled, on the same hardware (M4 Max, 128GB unified memory), same prompt and generation lengths, and same inference server.

Single Request Decoding

Test	tg TPS (MTP On)	tg TPS (MTP Off)	Δ tg TPS	TTFT (MTP On, ms)	TTFT (MTP Off, ms)	Peak Mem (MTP On, GB)	Peak Mem (MTP Off, GB)
pp1024/tg128	81.5	83.0	-1.8%	810.8	817.2	36.22	35.38
pp4096/tg128	79.8	90.5	-11.8%	2491.8	2590.2	37.00	36.16
pp8192/tg128	74.8	90.2	-17.1%	5331.6	5235.2	37.34	36.50
pp16384/tg128	70.9	86.0	-17.6%	12701.4	12424.1	37.97	37.13
pp32768/tg128	64.7	77.6	-16.6%	31601.0	30047.3	39.31	38.47
pp65536/tg128	52.9	63.2	-16.3%	82607.3	80079.9	42.00	41.15
pp131072/tg128	32.6	36.8	-11.4%	328400.7	344694.8	47.37	46.53
pp200000/tg128	24.1	28.7	-16.0%	695737.6	700247.4	53.03	52.19

Token generation throughput (tg TPS) is consistently lower with MTP enabled, by an average of about 13.6% across context lengths, with the gap widening to 14 to 18% slower once context exceeds about 4K tokens.
Peak memory is about 2.1% higher with MTP enabled across the board (the extra MTP head weights and their activation buffers).
TTFT (prefill) is roughly a wash, about 0.3% average difference with no consistent direction. MTP heads don't participate in prefill, so this is expected and mostly run to run noise.

Continuous Batching

Batch	tg TPS (MTP On)	tg TPS (MTP Off)	Δ tg TPS	Speedup (MTP On)	Speedup (MTP Off)
1x	81.5	83.0	-1.8%	1.00x	1.00x
2x	120.1	128.6	-6.6%	1.47x	1.55x
4x	133.8	138.5	-3.4%	1.64x	1.67x
8x	168.8	167.5	+0.8%	2.07x	2.02x

Under continuous batching, the gap mostly closes. MTP is within noise of the no-MTP baseline (-6.6% to +0.8%), with no consistent advantage at any batch size.

Takeaway

MTP head grafting is functional (weights load, model runs, output is correct), but in this MLX/oMLX inference setup, enabling MTP does not deliver a speculative decoding speedup, and for single-request decoding it does the opposite. This is most likely because the MTP head isn't yet wired into an actual speculative decode/verify loop in the serving stack; it currently just adds a forward pass and memory overhead without skipping any accept-reject cycle. Treat the MTP tensors in this repo as a structurally correct starting point for that integration work, not as a ready speedup.

If you get MTP-accelerated decoding working with this checkpoint on a different server or runtime, opening a discussion on this repo with your results would be very welcome.

Intelligence Benchmarks

Quality checks run on this quantized checkpoint with MTP disabled (thinking mode enabled for all runs). Equivalent runs with MTP enabled are in progress and will be added once complete; MTP only affects decoding strategy, not the model's weights, so these numbers reflect the checkpoint's underlying quality regardless of MTP status, but a direct on/off comparison isn't available yet.

These results are also embedded as structured model-index metadata in this README's YAML header, so they will surface in the Hub's evaluation results widget on the model page. Several benchmarks use a sampled subset rather than the full set, so treat MathQA and LiveCodeBench results (30 and 100 samples respectively) as indicative rather than tight estimates. MMLU-Pro, MBPP, and especially the full-set HumanEval run are more reliable.

Benchmark	Accuracy	Correct / Total	Sampled	Time (s)	Thinking
MMLU-Pro	81.0%	243 / 300	300 / 12,032	8,867.2	Yes
MathQA	93.3%	28 / 30	30 / 2,985	1,113.5	Yes
HumanEval	96.3%	158 / 164	164 / 164 (full)	5,940.7	Yes
MBPP	93.5%	187 / 200	200 / 500	7,483.3	Yes
LiveCodeBench	52.0%	52 / 100	100 / 1,055	20,696.7	Yes

Quantization to oQ8 preserves strong coding performance (HumanEval, MBPP) and general knowledge (MMLU-Pro) close to what's expected of the base model. LiveCodeBench, the hardest and most contamination-resistant of these, comes in noticeably lower, which lines up with it being the more demanding, competition-style coding benchmark rather than being oQ8-specific degradation. A same-precision base-model run on the same subset would be needed to isolate a quantization effect there.

Quantization Notes

Quantized with mlx-optiq targeting oQ8 mixed precision. Comparable full-precision/oQ7 runs on the same base architecture landed around 37GB, so expect a similar footprint here (see Files and versions for exact sizes).
No calibration or fine-tuning was performed beyond quantization and MTP head grafting; this is a direct weight-space conversion of the base model.

License

This is a derivative of Qwen/Qwen3.6-35B-A3B, released under the Apache 2.0 license. See the base model card for full terms, intended use, and limitations.