Instructions to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16
# Run inference directly in the terminal:
llama cli -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16
# Run inference directly in the terminal:
llama cli -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16
# Run inference directly in the terminal:
./llama-cli -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Use Docker

docker model run hf.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

LM Studio
Jan
Ollama
How to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with Ollama:
```
ollama run hf.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16
```
Unsloth Desktop

How to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with Docker Model Runner:
```
docker model run hf.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16
```

Lemonade

How to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF-F16

List all available models

lemonade list

Hermes Agent

How to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

this thing is a beast

by Rxnmark - opened Apr 28

Discussion

Rxnmark

Apr 28

I'm sitting with two 1080 ti's (22gb vram). Was daily driving Qwen_Qwen3.6-27B-IQ3_XS (bartowski, 12.7gb) and Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS (lordx64, 18.5gb) until today. Like i was not sure what is better. Just tried this one - Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Compact (16.8gb). And i must say - both MoE mentioned is not less capable for my fullstack project, than 27b model! But speed difference is like 3x (10 t/s and 30 t/s). PP is also night and day. Also while on IQ4_XS i was able to only utilize 131k context - on I-Compact it's possible to go for 196k (and probably more, just didn't test yet). So it's my new daily driver now.

sjorsb

May 4

•

edited May 4

yeah, for real. i used the others too but this one, the i-quality model full context (llama.cpp) used by hermes agent with continuous 50 t/s (compared to 24), excellent precision, excellent reasoning, zero tool call misses. Uses around 60GB of unified RAM, compared to 90+ GB of the others (q8_0) but with same or better quality.

This is the best Qwen 3.6 model by far.

Congrats mudler and many thanks!

mudler

Owner May 4

Glad you enjoy! It's indeed my daily driver all around too. I'll keep pushing other MoE models quants as new releases pop up :)

OrisisCM

May 5

which version would you recommend for a 16gb GPU?

danuzca

May 5

I'm sitting with two 1080 ti's (22gb vram). Was daily driving Qwen_Qwen3.6-27B-IQ3_XS (bartowski, 12.7gb) and Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS (lordx64, 18.5gb) until today. Like i was not sure what is better. Just tried this one - Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Compact (16.8gb). And i must say - both MoE mentioned is not less capable for my fullstack project, than 27b model! But speed difference is like 3x (10 t/s and 30 t/s). PP is also night and day. Also while on IQ4_XS i was able to only utilize 131k context - on I-Compact it's possible to go for 196k (and probably more, just didn't test yet). So it's my new daily driver now.

Care to share what llama branch are you running it with or are you running the official build? I have a multig 1080ti rig too so I am curious

millerlogic

May 5

After all this praise, I tested the balanced version of this on my strix halo. It flies but I don't get consistent quality responses. I'm using llama.cpp image tag: server-vulkan-b8643. On cold start the responses seem fine, but if I resubmit the same query it can go off the rails. I often get different answers like "yes" vs "no" being complete opposite and sometimes the thinking gets stuck in infinite loop. I wonder if there's a kv cache issue, which might be llama specific. My test prompt was "what is the die in something like die cast, and the die in computer chips?" Everything works great on this setup with gemma 4 and many others.

Rxnmark

May 5

Care to share what llama branch are you running it with or are you running the official build? I have a multig 1080ti rig too so I am curious

Im on the main branch. Here is my regular script:

$MODEL = "Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Compact.gguf"
#$MMPROJ = "mmproj-Qwen_Qwen3.6-35B-A3B-f16.gguf"

$PORT = 1234
$CTX = 196608
$NP = 1

$TEMP = 0.6
$TOP_K = 20
$MIN_P = 0.0
$PRES = 0.0

$batContent = ".\llama-server " +
"-m $MODEL " +
#"-mm $MMPROJ " +
"--port $PORT " +
"-ngl 99 " +
"--no-mmap " +
"-c $CTX " +
"-np $NP " +
"-b 1024 " +
"-ub 512 " +
"--cache-type-k q8_0 " +
"--cache-type-v q8_0 " +
"-fa on " +
"--jinja " +
"--reasoning on " +
"--temp $TEMP " +
"--top-k $TOP_K " +
"--min-p $MIN_P " +
"--repeat-penalty 1.0 " +
"--presence-penalty $PRES " +
"--chat-template-kwargs "{\"preserve_thinking`": true}" " + "-cpent 4096 " + "-cram 24576 " + "-ctxcp 48 " + "--reasoning-budget 4096 " + "--reasoning-budget-message "[SYSTEM ALERT: Reasoning budget exceeded. I am stuck in a loop or overcomplicating. I must stop IMMEDIATELY and use the ask_followup_question tool to notify the user and ask for guidance.]`" "

$batPath = ".\temp_runner.bat"
Set-Content -Path $batPath -Value $batContent -Encoding ASCII

Write-Host "Starting llama-server via BAT file on port $PORT..." -ForegroundColor Green
Write-Host "Generated command: $batContent" -ForegroundColor DarkGray

& $batPath

danuzca

May 8

This comment has been hidden (marked as Off-Topic)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment