Instructions to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Use Docker

docker model run hf.co/xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

SGLang

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with Ollama:
```
ollama run hf.co/xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M
```
Unsloth Desktop

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with Docker Model Runner:
```
docker model run hf.co/xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M
```

Lemonade

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-128E-Pruned-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.6-35B-A3B 128-Expert Pruned GGUF

Experimental research artifact. This is a 50% expert-pruned model, not an official Qwen release. The pruning calibration is small and coding/agent biased. Test the Q8 artifact on your workload before trusting a lower quant.

This repository contains Q8 through one-bit-class GGUF variants of Qwen/Qwen3.6-35B-A3B. The routed MoE was reduced from 256 to 128 experts per layer while retaining the original top-k 8 active experts per token. Pruning reduces stored parameters and memory, but it does not reduce the eight active expert matmuls per token.

Quant files

Quant	Size GiB	SHA-256	QA	Notes
Q8_0	18.392	`b85b3ab29aabd631…`	8/8 (pass)	Reference pruned artifact; retained weights preserve their original Q8 bytes.
Q6_K	14.212	`7fa673151e2f653f…`	6/8 (fail)	Near-Q8 PPL, but missed 2/8 deterministic acceptance checks.
Q5_K_M	12.347	`495cc9fe01f33bea…`	8/8 (pass)	Best measured local balance: 8/8 checks and only +1.05% PPL.
Q4_K_M	10.592	`b8fec3c109c561b2…`	7/8 (pass)	Smaller viable option: 7/8 checks and +2.38% PPL.
Q3_K_M	8.434	`0d4dfa3c58100e69…`	6/8 (fail)	Aggressive; verify on your workload.
Q2_K	6.582	`602bab193dd82587…`	7/8 (pass)	Passed 7/8 smoke checks, but held-out PPL rose 26.50%.
IQ2_XS	5.396	`9b631cca3ae23cf7…`	6/8 (fail)	Extreme ~2-bit-class build; imatrix required and workload QA strongly recommended.
IQ1_M	4.312	`4064257359d35174…`	2/8 (fail)	Experimental one-bit-class endpoint (1.75 bpw nominal for quantized tensors).
IQ1_S	3.952	`c5caaa158eb73055…`	1/8 (fail)	Smallest experimental endpoint (1.56 bpw nominal for quantized tensors).

GGUF has no conventional Q7_K family, so the ladder follows the supported practical families: Q8, Q6, Q5, Q4, Q3, Q2/IQ2, then IQ1. IQ1_M and IQ1_S are both included because they are distinct one-bit-class formats.

Requantization warning: every file below Q8_0 was produced from the pruned Q8_0 artifact with --allow-requantize, not from F16/BF16. The quantizer itself warns that this can reduce quality more than direct quantization from 16/32-bit weights. A remapped 128-expert importance matrix was used for every lower quant.

What was pruned

This checkpoint is not a direct REAP implementation. REAP motivated the experiment, but the Qwen selection used low router-use frequency because that criterion performed better than REAP in the preceding local Granite-MoE control experiment. See the REAP paper and official Cerebras implementation for the router-weighted activation-norm method that inspired this work.

The selection procedure was:

Run 512 coding/agent calibration tokens through all 40 MoE layers.
Record 20,480 layer-token observations and each token's top-8 routed expert IDs.
In each layer, prune the 128 experts with the lowest routed top-k frequency; break ties deterministically by original expert ID.
Copy retained expert rows and corresponding router rows in ascending original-ID order. Update expert_count from 256 to 128.
Preserve the raw quantized bytes of retained expert tensors. Shared experts and all other tensors remain byte-identical to the source Q8.
Re-open the output and fully hash-check all 733 tensors before publication.

The exact retained IDs, per-expert route counts, source trace hash, and transformation report are included under method/.

Evaluation

The first table compares the unpruned Q8 baseline, the 50%-pruned Q8, and the existing all-expert mixed-Q2 deployment reference on the same eight deterministic coding, JSON, math, tool-call, and context checks.

Artifact	Size GiB	Passed	Exact outputs vs baseline	Median decode tok/s	Gate
full-q8	34.369	7/8	8/8	33.84	PASS
pruned-50-q8	18.392	8/8	7/8	104.38	PASS
full-mixed-q2	16.930	5/8	4/8	118.01	FAIL

Exact-output equality is reported separately from task correctness. Runtime numbers are specific to a local RTX 3060 Ti + RTX 2080 SUPER + RTX 3080 setup; the unpruned Q8 baseline used CPU expert offload, while the smaller models fit across all three GPUs.

The complete per-prompt outputs and timing records are in evaluation/. These checks are a smoke/acceptance suite, not a replacement for HumanEval, BFCL, MMLU, long-context, or safety evaluation.

All-quant functional and speed benchmark

Artifact	Size GiB	Passed	Exact outputs vs baseline	Median decode tok/s	Gate
Q8_0	18.392	8/8	8/8	104.11	PASS
Q6_K	14.212	6/8	6/8	108.44	FAIL
Q5_K_M	12.347	8/8	8/8	118.28	PASS
Q4_K_M	10.592	7/8	7/8	125.22	PASS
Q3_K_M	8.434	6/8	5/8	112.63	FAIL
Q2_K	6.582	7/8	6/8	124.36	PASS
IQ2_XS	5.396	6/8	4/8	109.60	FAIL
IQ1_M	4.312	2/8	1/8	100.48	FAIL
IQ1_S	3.952	1/8	1/8	114.08	FAIL

All functional-speed rows except IQ1_M used the local ik_llama CUDA build. That build aborts on IQ1_M with Unhandled type iq1_m (29), so IQ1_M was bench-tested with the local TurboQuant llama.cpp build instead. Its speed is therefore not directly comparable to the other rows; the engine and full command are preserved in evaluation/quant-ladder-qa-results.json.

Held-out perplexity by quant

Perplexity used the included evaluation/qwen36_ppl_heldout.txt, which is separate from both the pruning router trace and the importance-matrix calibration. Values are most useful as relative degradation within this table, not as a cross-model leaderboard.

Quant	Size GiB	Held-out PPL	Delta vs Q8	Estimate error
Q8_0	18.392	9.7356	+0.00%	0.81899
Q6_K	14.212	9.7435	+0.08%	0.81747
Q5_K_M	12.347	9.8377	+1.05%	0.82701
Q4_K_M	10.592	9.9673	+2.38%	0.84195
Q3_K_M	8.434	10.9941	+12.93%	0.96516
Q2_K	6.582	12.3158	+26.50%	1.09987
IQ2_XS	5.396	14.3985	+47.90%	1.27363
IQ1_M	4.312	21.4644	+120.47%	1.99295
IQ1_S	3.952	29.1534	+199.45%	2.88727

Practical selection from this sweep

Q8_0 is the reference for judging pruning independently of further quantization.
Q5_K_M is the measured recommendation for this local coding/agent workload: 8/8 checks, +1.05% held-out PPL, and 118.28 median decode tok/s.
Q4_K_M is the smaller viable choice: 7/8 checks, +2.38% PPL, and 125.22 tok/s.
Q3 and below are research/space-first builds, not general recommendations; their perplexity degradation is material even when a tiny smoke suite passes.

Usage

llama-server \
  -m Qwen3.6-35B-A3B-128E-Pruned-Q4_K_M.gguf \
  --jinja --n-gpu-layers 99 --ctx-size 32768

Use a recent llama.cpp-compatible runtime with qwen35moe support. The Q8 file is the best reference for deciding whether pruning itself is acceptable; move down the quant ladder only after that comparison.

Limitations

The 512-token selection trace is small and strongly biased toward coding/agent work.
Experts rarely used by that trace may be important for languages, domains, knowledge, reasoning styles, or safety behavior absent from calibration.
There was no recovery fine-tuning after pruning.
Frequency pruning measures router use, not expert activation magnitude or REAP saliency.
Halving stored experts does not halve decode compute because top-k remains 8.
Low-bit requants compound pruning risk with Q8-to-lower-bit requantization error.
Do not use this experiment for high-stakes decisions without independent evaluation.

Reproducibility files

quant-manifest.json: source, imatrix, quantizer hashes, commands, sizes, and hashes.
SHA256SUMS: complete model-file checksums.
method/expert-selection.json: retained/pruned IDs and route counts for every layer.
method/pruning-report.json: Q8 rewrite and verification report.
method/imatrix-remap-report.json: 256-to-128 imatrix transformation report.
evaluation/: deterministic QA outputs and timing evidence.

License and attribution

The GGUF metadata identifies the base license as Apache-2.0. This derivative keeps that license metadata. The source Q8 metadata credits Unsloth as its quantizer. Review the upstream model card and license before redistribution or use.

Downloads last month: 4,222

GGUF

Model size

19B params

Architecture

qwen35moe

Hardware compatibility

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Model tree for xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(777)

this model

Paper for xero0000/Qwen3.6-35B-A3B-128E-Pruned-GGUF

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

Paper • 2510.13999 • Published Oct 15, 2025 • 21