Instructions to use aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.6-35B-A3B — MLX mixed-bit GPTQ @2.7 bpw (~12 GB)

The size-optimized sibling of Qwen3.6-35B-A3B-MLX-GPTQ-3.5bpw: same loss-aware + GPTQ pipeline, pushed to the practical floor of MLX scalar affine quantization (~2.73 effective bits/weight overall, 12.0 GB). Fits comfortably in 24 GB unified memory.

Recipe

Experts (91.7% of params): per-(layer,tensor) menu MCKP over (bits × group-size) candidates — chosen mix {2bit/gs128: 69, 3bit/gs128: 11} (avg 2.38 bpw) by empirical-Fisher gxw cost, then GPTQ (pooled per-layer Hessian, sym=False, MSE grid, act-order static groups).
Spine is GPTQ'd too (unlike the 3.5bpw build): linear-attention qkv/z/out, full-attention q/k/v/o, shared experts — 4bit gs64 with their own input Hessians; lm_head 6bit. Routers 8bit, embeddings 4bit, vision tower unquantized (bf16).

Quality (vs bf16, fakequant-proxy measured on CUDA)

holdout	KL	PPL
ja-think	0.288	9.75 → 11.46 (+17.5%)
multilingual	0.227	11.02 → 12.71 (+15.4%)

(Numbers measured with RTN spine; the shipped artifact additionally GPTQ-compensates the spine, so true quality is slightly better.) For reference, the 3.5bpw/15.4 GB sibling measures +6.6%/+6.9% — pick that one if you can spare 3.4 GB. This 2.7bpw build is for tight-memory Macs, where it is roughly the best scalar-affine MLX can do: plain-RTN mixed 2-bit at similar size measures ~+47%.

Run (Apple Silicon, ≥16 GB unified memory)

pip install -U mlx-lm            # needs >=0.31 (qwen3_5_moe support)

mlx_lm.generate --model aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw \
  --prompt "東京の観光名所を3つ教えてください。" --max-tokens 2048

mlx_lm.server --model aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw --port 8080

Notes

Text-only inference is the validated path (vision weights present, unquantized).
Thinking mode is on by default and verbose; pass "chat_template_kwargs": {"enable_thinking": false} for direct answers.

Provenance / license

Base model © Alibaba Cloud, Apache-2.0. Derived from the official bf16 checkpoint and the mlx-community 4bit conversion (structure/tokenizer).

Downloads last month: 69

Safetensors

Model size

35B params

Tensor type

U32

BF16

MLX

Hardware compatibility

4-bit

Model tree for aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(832)

this model