ubergarm

/

Qwen3.6-27B-GGUF

Like 35

@phakio

Thanks as always for your report and details! Yes, I agree with you the 27b dense is a bit slower but usually worth it to leave it chugging along running in a harness. i've been having good luck with it in pi.dev after using opencode to write an extension to dynamically detect loaded models from llama-server and toggle on/off thinking dynamically!

your command looks good for single 24GB VRAM GPU which is exactly what i was planning with these quants (one for vulkan and two for CUDA depending on how much kv-cache you want).

you could possibly get a touch more speed by adding in --merge-qkv -muge to your command, but not sure how well it would help.

also you can add mmproj support with something like the following (the --no-mmproj-offload should run it on CPU/RAM so save VRAM for precious kv-cache)

# wget https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF/resolve/main/mmproj-Qwen_Qwen3.6-27B-bf16.gguf
  --mmproj "$mmproj" \
  --image-min-tokens 1024 \
  --image-max-tokens 4096 \
  --no-mmproj-offload \

Make sure to enable multimodal support in your client config then too.

Finally you can add some caching to it e.g.

  --ctx-checkpoints 16 \
  -cram 8192

But i'm exprimenting with leaving --ctx-checkpoints 0 as i'm not 100% confident but have anecdotally seen some looping at higher context depth when using it.. cram is showing here use 8GB of system RAM for prompt caching which can speed up things espeically that initial repetative system prompt.

Cheers!

getting an error after using the model for a bit, the ik_llama.cpp process fails with

ggml_cuda_compute_forward: FUSED_RMS_NORM failed
CUDA error: unspecified launch failure
  current device: 0, in function ggml_cuda_compute_forward at /var/home/phone/ik_llama.cpp/ggml/src/ggml-cuda.cu:3885
  err
/var/home/phone/ik_llama.cpp/ggml/src/ggml-cuda.cu:132: CUDA error

haven't had a crash on mainline ever since I rebuilt my fedora CUDA toolbox. I'm going to keep playing around and if the issue persists open an issue on ik_llama.cpp github

Oh interesting, I've not hit that FUSED_RMS_NORM before. I'm using this IQ5_KS model pretty extensively now on 2x RTX A6000's (sm86 arch) with -sm graph and Driver Version: 580.105.08 CUDA Driver Version: 13.0.

I'll keep an eye out on ik's github for the issue, thanks for the heads up!

Ran a quick llama-sweep-bench of the IQ5_KS on my home gaming rig full offload on single 3090TI FE (tuned with LACT).

I use bartowski's bf16 mmproj for actual llama-server usage as shown below. I can fit more kv-cache or better quality e.g. -ctk f16 -ctv q8_0 as you mention when I'm running "headless" without browser/discord and other stuff open. The iq4_ks is nice if i want a little faster or more head-room to run everything on my desktop at once. I'm still experimenting with --ctx-checkpoints but have them disabled (0) at the moment and relying on -cram. When your client harness is setup correctly with chat template stuff all working it doesn't need to re-process kv-cache ever which is great!

llama-server

#!/usr/bin/env bash

model=/mnt/ai/models/ubergarm/Qwen3.6-27B-GGUF/Qwen3.6-27B-IQ5_KS.gguf
mmproj=/mnt/ai/models/ubergarm/Qwen3.6-27B-GGUF/mmproj-Qwen_Qwen3.6-27B-bf16.gguf

CUDA_VISIBLE_DEVICES="0" \
./build/bin/llama-server \
  --model "$model" \
  --alias Qwen3.6-27B \
  -c 65536 \
  -ctk q8_0 -ctv q8_0 \
  --merge-qkv \
  -muge \
  -ngl 99 \
  --threads 1 \
  --host 127.0.0.1 \
  --port 8880 \
  --parallel 1 \
  --jinja \
  --no-mmap \
  --mmproj "$mmproj" \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --image-max-tokens 4096 \
  --ctx-checkpoints 0 \
  -cram 8192

llama-sweep-bench

#!/usr/bin/env bash

model=/mnt/ai/models/ubergarm/Qwen3.6-27B-GGUF/Qwen3.6-27B-IQ5_KS.gguf

CUDA_VISIBLE_DEVICES="0" \
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 65536 \
  -ctk q8_0 -ctv q8_0 \
  --merge-qkv \
  -muge \
  -ngl 99 \
  --threads 1 \
  --parallel 1 \
  --no-mmap \
  --warmup-batch \
  -n 64

It's a very handy model when paired with pi or similar harness for sure!

It's a very handy model when paired with pi or similar harness for sure!

Yes, I settled on this one as my “daily driver,” too. Kimi/GLM are just too slow for regular use.
Speaking of which, @ubergarm , have you by any chance heard about XiaomiMiMo/MiMo-V2.5-Pro? 😀

@sousekd

Yeah, I have Qwen3.6-27B for the "fast" one and Kimi-K2.6-Q4_X for my "slow" one. both have mmproj which is nice to setup a feedback loop with a headless playright browser to iterate and fix html code etc.

MiMo-V2.5-Pro

I can't get any vibe coding done if they keep releasing models every two days now haha... I believe I saw @AesSedai working on it already, is it supposed to be good? I'll check on it soon!

@ubergarm Is Qwen3.6-27B an upgrade from ubergarm/Qwen3.5-122B-A10B-GGUF (my current daily) for agentic coding in your opinion?

@gghfez I'm not the man in question but I can say that I feel like if you're okay with waiting for the thinking process, the new 3.6 is much better. I've used both and they really upped the quality with the 3.6 release, even compared to 122b. The main advantage of the 27b being you can fit a q5 quant with good context within a single 24gb consumer gpu. I think if they release a 122b 3.6 that will be even better for multi gpu, but for now this one is really solid for me!

Yeah, I have Qwen3.6-27B for the "fast" one and Kimi-K2.6-Q4_X for my "slow" one.

Yes, same here. Thank you for the playwright tip. Do you use it with OpenCode or something else?

@gghfez

Is Qwen3.6-27B an upgrade from ubergarm/Qwen3.5-122B-A10B-GGUF (my current daily) for agentic coding in your opinion?

I agree with @phakio . I've stopped loading Qwen3.5-122B-A10B to run Qwen3.6-27B dense. Even though it is slower, it feels more consistent and happily makes tons of tool calls and keeps going even when making mistakes along the way.

I've heard its better to praise your kids as "hard workers" rather than "smart" so I wonder if perhaps I should setup my prompts to say "You are a resilient AI assistant who perseveres through mistakes and flows like water around obstacles." haha....

Apparently some of my Qwen3.6-27B quants were recently benchmarked behind a paywall haha: https://localbench.substack.com/p/qwen-3-6-27b-gguf-quality-benchmark - I haven't seen the results. 🤞

@sousekd

Yes, same here. Thank you for the playright tip. Do you use it with OpenCode or something else?

The last time I used opencode for was last week, when I had it bootstrap my pi.dev environment.

A few reasons I prefer pi over opencode:

• its not a TUI but actual text console mode which i prefer in my tiling windows manager console based "desktop"
• it is much leaner so all models are faster given i start out in the fast <2k context speed
• easy to vibe code custom extensions quickly and /reload to use them immedeately

My only complaint is it required vibe coding an entire typescript extension to connect to localhost:8080/v1 llama-server endpoint.

But now it dynamically queries my ssh tunnel endpoints, detects the loaded models, tracks context use and cache hits, toggle thinking on and off. and most importantly it is now handling preserve_thinking correctly so i don't ever wait on kv-cache re-processing if i hit ESC to cancel or take my user turn.

I just pushed my pi.dev config folder if anyone is interested: https://github.com/ubergarm/dotpi

Cheers!

I hadn't noticed these benchmarks before, but the IQ5_KS looking good on previous Qwen3.5-27B

• https://localbench.substack.com/p/qwen-3-5-27b-gguf-quality-benchmark

I haven't been able to see behind the paywall for the new ones:

• https://localbench.substack.com/p/qwen-3-6-27b-gguf-quality-benchmark

@oobabooga does a pretty good job describing their methodology, but seems to use the unsloth bf16 (should be fine assuming they do it like everyone else using official mainline llama.cpp convert_hf_to_gguf.py with the official released bf16 safetensors). i always bake the official model repo chat template into my GGUF and people can use --chat-template-file ... if they want anything custom.

Lot's of stuff to read in there, and cool to see some of my quants! Thanks ooba!

cheers!

Happy to share the results on this one here (I'm using the substack profits to fund more benchmarks, as I'm running these on the cloud, 627 quants tested so far :)

IQ5_KS in particular seems to have a clear edge, and IQ4_KS is also on the pareto frontier. smol-IQ4_NL gave me an error and I gave up on it.

Thanks for publishing ik quants @ubergarm , I have personally used your step 3.5 flash one. My project has ik_llama.cpp builds now by the way https://github.com/oobabooga/textgen/releases, with binaries compiled at https://github.com/oobabooga/llama-cpp-binaries.

@oobabooga Do you use ik_llama.cpp for all of these? Or is there a mix of llama.cpp and ik_llama.cpp (for the ik quants) ?

@ubergarm Thanks, I'll try dropping it in. It would be nice to free up a couple of GPUs so I can have Gemma running at the same time.

its not a TUI but actual text console mode which i prefer in my tiling windows manager console based "desktop"

Another thing for me to try then. Claude Code within tmux has been giving me the shits recently.

@oobabooga Do you use ik_llama.cpp for all of these? Or is there a mix of llama.cpp and ik_llama.cpp (for the ik quants) ?

Pure ik_llama.cpp for ik quants, with a small patch to extract logprobs from input prompts: https://github.com/oobabooga/ik_llama.cpp/commit/b50c409ca3ae4266562cc796d4b7580f1f201e82

@oobabooga thanks for going the extra mile and including the ik_llama.cpp patches and benchmarks! Really appreciate that!

What is the best way to support you e.g. subscribe on substack, or your github sponser link or you take paypal/venmo? haha...

I'm out of office today, but will look into it later this week. I don't want to influence any findings of course and I assume your reports mention any potential conflict of interest ;p haha...

Cheers y'all!