JASouth/Qwen3.8-27B-IQ-Hybrid

TL; DR Updated (14/09/2026)

Currently, the standout quant is IQ3_E_S_V2, offering the best balance of overall quality and quality-to-size ratio. In the ultra-low bitrate tier (~7.5 GB / sub-8 GB), I am actively benchmarking IQ2_E_S_V2 and IQ2_E_S, which proved to be the most viable variants:

• IQ3_E_S_V2 — Best Overall: Highly capable in long-context coding and complex multi-step reasoning. While it experiences occasional minor attention slips or hallucinations, it remains significantly more reliable and consistent than the lower-bit variants.
• IQ2_E_S_V2 — Best for Code & Agentic Workflows (~2.2 BPW): Remarkably capable in long-context tasks, code synthesis, and architectural understanding (e.g., capable of generating fully playable interactive scripts in a single prompt). However, it is prone to repetition loops in standard conversational text, though it typically self-terminates without freezing. (Tip: Setting repeat_penalty: 1.15–1.20 and presence_penalty: 0.2 effectively eliminates these loops).
• IQ2_E_S — Best Stability for Direct Q&A (~2.2 BPW): Shows strong conversational stability, strict output formatting, and high resistance to counterfactual traps on isolated prompts. However, it suffers from severe attention degradation during complex logic execution, making it unsuitable for long-context generation or functional coding tasks.

Qwen3.8-27B GGUF Quantizations (IQ2_E_S & IQ3_E_S)

This repository provides experimental, low-bitrate GGUF quantizations for Qwen3.8-27B, focused on extreme VRAM efficiency without catastrophic quality degradation.

Motivation & Overview

The primary goal of this project was to explore whether Qwen3.8-27B could be loaded and executed on consumer GPUs with strictly 8 GB of VRAM.

Quantizing a 27B model down to this range yields two distinct results:

1. IQ2_E_S (2.40 BPW, ~7.51 GiB): Fits tightly into an 8 GB VRAM budget. While low-bit quantization introduces inevitable degradation, perplexity remains bounded, making the model practically usable for lighter or structured generation tasks.
2. IQ3_E_S (3.00 BPW, ~9.41 GiB): A balanced quantization targeting 10–12 GB GPUs, showing theoretical perplexity retention close to the unquantized baseline.

Backend & Hardware Compatibility Notes (Intel Arc / oneAPI)

Important Warning for Intel Arc Users:
The IQ2_E_S build is known to collapse/fail under the SYCL backend (producing gibberish, NaNs, or crashes during execution).
If you are running on Intel Arc hardware, you must use the Vulkan backend instead of SYCL:

• Ensure your llama.cpp binary is compiled with -DGGML_VULKAN=ON.
• Run via Vulkan to avoid kernel execution bugs present in current SYCL implementations for low-bit IQ quantizations.

Limitations & Validation Notes

• IQ3_S_S: Due to physical VRAM constraints (8 GB test system) and limited testing time, IQ3_S_S has not been empirically verified on thinking mode in inference, only evaluated on non-thinking mode one-shoot tasks. Users with 12 GB or 16 GB setups are encouraged to test it.

Community Feedback

If you test either the IQ2_E_S, IQ2_S_S, IQ3_E_S or IQ3_S_S builds, please leave feedback in the Discussions tab:

• Real-world coherence vs. baseline expectations.

• Inference speeds (tokens/sec) across different backends (CUDA, Vulkan, SYCL, Metal).

• Context scaling behavior beyond 8k tokens.

Support

If these quantizations are useful for your local workflows or research, consider supporting further compute and quantization tests: