--- license: apache-2.0 base_model: CCSSNE/trohrbaugh-Qwen3.5-122B-A10B-heretic base_model_relation: quantized library_name: gguf pipeline_tag: text-generation tags: - gguf - qwen3.5 - qwen3.5-moe - heretic - rocmfp4 - imatrix - amd - vulkan - rocm - mtp - speculative-decoding - experimental ---
# Qwen3.5 122B-A10B Heretic · ROCmFP4 iMatrix ### A compact, importance-calibrated ROCmFP4 build for high-memory AMD systems **122B total · 10B active · 60.70 GiB · 28.45 tok/s native · 33.68 tok/s MTP · BF16 KLD 0.100716 · 28.11 Decode t/s ** Decode Speed 34.16% Faster Size 13.47gb Smaller.
> [!IMPORTANT] > This GGUF uses custom ROCmFP4 tensor types. It requires > [ROCmFPX](https://github.com/charlie12345/ROCmFPX) or a runtime with equivalent > support; stock `llama.cpp`, Ollama, LM Studio, and similar stock runtimes > cannot load it. I also suggest usign the template from https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates Froggeric is tha dude. ## Downloads | Artifact | Direct download | |---|---| | **Main iMatrix model — 60.70 GiB** | **[Download `Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf`](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/resolve/main/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf?download=true)** | | **Optional MTP companion — 2.14 GiB** | **[Download `Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf`](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/resolve/main/Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf?download=true)** | | Calibration matrix — 342.28 MiB | [Download `Qwen3.5-122B-A10B-Heretic-code-technical.imatrix`](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/resolve/main/Qwen3.5-122B-A10B-Heretic-code-technical.imatrix?download=true) | [Browse every repository file →](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/tree/main) ## What it is This is an iMatrix-calibrated quantization of the full-weight Qwen3.5 122B-A10B Heretic checkpoint. It was built with ROCmFPX's compact `Q4_0_ROCMFP4_STRIX_LEAN` recipe, but the hardware-oriented preset name has been left out of the public filename. | | Result | |---|---:| | **BF16 mean KLD** | **0.100716 ± 0.003407** | | **Two-sigma KLD upper bound** | **0.107530** | | **Improvement from iMatrix** | **14.01% lower KLD** | | **Greedy decode** | **28.448 tok/s** | | **Greedy decode with MTP** | **33.677 tok/s** | | **4,277-token prefill** | **350.248 tok/s** | ## Files | File | Purpose | Size | SHA-256 | |---|---|---:|---| | [`Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf`](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/resolve/main/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf?download=true) | Main text model | 65,184,265,248 bytes | `f563fba693c65a05431d4beecd11736edf7e54208902cae70d7b725057a7556e` | | [`Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf`](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/resolve/main/Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf?download=true) | Optional external MTP companion | 2,294,290,880 bytes | `5554538448efb668099be898ce7c8da5920e86b321f45e8ac1f7dca3518cc0a8` | | [`Qwen3.5-122B-A10B-Heretic-code-technical.imatrix`](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/resolve/main/Qwen3.5-122B-A10B-Heretic-code-technical.imatrix?download=true) | Calibration matrix | 358,906,432 bytes | `3a2a0e0bea5fe380a27cc938ad02e727dcbd840744ab3ddb3a8e8402a67deb05` | | [`chat_template.jinja`](https://huggingface.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF/resolve/main/chat_template.jinja?download=true) | Pinned Qwen3.5 chat template | 7,756 bytes | `a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715` | The main model is complete and runs independently. The 23-tensor, one-layer MTP file is an optional external companion; no MTP layer is embedded in the main GGUF. Runtimes call this role a *draft model*, but `Draft` is omitted from the public filename because it does not mean unfinished. ## Quality The calibrated model and its uncalibrated control were replayed against the same saved distributions from the exact BF16 parent on ROCm0. | Build | Mean KLD | PPL ratio | Same top | RMS Δp | |---|---:|---:|---:|---:| | **iMatrix** | **0.100716 ± 0.003407** | **1.080217 ± 0.010198** | **87.629% ± 0.622%** | **10.881%** | | Uncalibrated | 0.117119 ± 0.004183 | 1.109251 ± 0.011299 | 86.774% ± 0.640% | 12.077% | iMatrix calibration reduced mean KLD by **14.01%**, reduced the PPL ratio by 0.029034, and improved same-top agreement by 0.855 percentage points. The two-sigma KLD upper bound is **0.107530**, clearing the selected **0.126** performance-oriented target. The paired replay covers 11 complete 512-token chunks and 2,805 evaluated next-token distributions. These were recovered at an exact probability-block boundary from an interrupted 32-chunk BF16 run. This is useful preliminary evidence, not a completed 32-chunk acceptance gate. ## Performance Measured on a 128 GB AMD Strix Halo system with Vulkan/RADV, 131,072 context, parallel 1, batch/ubatch 2048/1024, Q8_0 KV cache, flash attention, and MTP off. Values are medians from repeated runs. | Workload | Repeats | Decode | Prompt processing | |---|---:|---:|---:| | Greedy, 256 generated tokens | 5 | **28.448 tok/s** | 71.929 tok/s | | Sampled, 256 generated tokens | 3 | **28.418 tok/s** | 71.294 tok/s | | 4,277-token prefill + 128 generated | 3 | **28.041 tok/s** | **350.248 tok/s** | Relative to the otherwise matched uncalibrated build, decode improved by 2.43%, 5.23%, and 2.84% respectively. Long-prompt prefill changed by -1.03%. These are single-system measurements, not general performance guarantees. ### Optional MTP acceleration The external MTP companion was tested against this exact iMatrix trunk in a matched MTP-off/on A/B. The model, runtime, context, cache, batch geometry, prompts, and companion identity were held constant; only MTP execution changed. | Workload | Native decode | MTP decode | Decode gain | Acceptance | End-to-end gain | |---|---:|---:|---:|---:|---:| | Greedy, 256 tokens | 28.328 tok/s | **33.677 tok/s** | **+18.89%** | 85.03% | **+16.44%** | | Sampled, 256 tokens | 28.035 tok/s | **30.953 tok/s** | **+10.41%** | 80.00% | **+8.76%** | | 4,277-token prefill + 128 | 27.585 tok/s | **33.752 tok/s** | **+22.36%** | 90.73% | -0.45% | MTP reduced prompt-processing throughput by about 7% because the companion must catch up on the prompt. It therefore helps most when generation is long enough to repay that fixed cost. The exact identity probe was unchanged, and both the MTP-off and MTP-on lanes passed technical prose, executable code semantics, strict JSON, tool-calling, balanced-analysis, and defensive-safety smokes. ## Quantization The model was quantized once from the validated BF16 GGUF using a 339-chunk code/technical importance matrix and the `Q4_0_ROCMFP4_STRIX_LEAN` preset. | Tensor type | Count | |---|---:| | `Q4_0_ROCMFP4_FAST` | 457 | | `Q4_0_ROCMFP4` | 60 | | `F32` | 361 | | `Q5_K` | 1 | In this preset, attention K/V tensors retain dual-scale ROCmFP4 protection, most transformer weights use the compact FAST layout, and token embeddings/output use Q5_K. The artifact contains 879 tensors across 48 blocks and no embedded MTP layer. ## Run ```bash llama-server \ --model Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf \ --host 127.0.0.1 --port 8080 \ -dev Vulkan0 --n-gpu-layers 999 \ --ctx-size 131072 --parallel 1 \ --flash-attn on --batch-size 2048 --ubatch-size 1024 \ --cache-type-k q8_0 --cache-type-v q8_0 \ --jinja --reasoning-format deepseek ``` Adjust context and cache settings for your memory budget. The tested runtime was [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) commit `a6a93765f7ce9779c13f9881164a65f7a9f31198`, built in Release mode for `gfx1151` with Vulkan and HIP enabled, plus a local Qwen3.5 MoE graph correction. Results with newer ROCmFPX revisions have not yet been established. To enable the optional companion, append: ```bash --spec-type draft-mtp \ --spec-draft-model Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf \ --spec-draft-device Vulkan0 --spec-draft-ngl 999 \ --spec-draft-type-k f16 --spec-draft-type-v f16 \ --spec-draft-n-max 2 --spec-draft-p-min 0.6 \ --spec-draft-p-split 0.10 --spec-draft-backend-sampling ``` ## Tested system | | | |---|---| | **Platform** | AMD Strix Halo, 128 GB unified memory | | **GPU target** | Radeon 8060S / `gfx1151` | | **Backend** | Vulkan / RADV | | **Kernel** | Linux 6.17.0-1028-oem | | **Mesa** | 25.2.8 | No vision projector is included; treat this release as text-only. ## Lineage and credits This repository exists because several people published substantial work openly. Their roles are kept distinct so downstream users can find the original sources. - **Official base model:** [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B), Apache 2.0. - **Full-weight Heretic source:** [trohrbaugh/Qwen3.5-122B-A10B-heretic](https://huggingface.co/trohrbaugh/Qwen3.5-122B-A10B-heretic), accessed through the pinned [CCSSNE mirror](https://huggingface.co/CCSSNE/trohrbaugh-Qwen3.5-122B-A10B-heretic) at `fd6d6bc89e9be47577736c952dd4d1d7c830f09f`. - **MTP tensor source:** [OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4](https://huggingface.co/OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4) at `07b7c2101c33f23a84fd3c618d00f17ea12269c3`; its model card reports that the MTP tensors were restored from the official Qwen checkpoint. - **Heretic software and method:** [p-e-w/heretic](https://github.com/p-e-w/heretic), created by Philipp Emanuel Weidmann. The source checkpoint reports Heretic v1.2.0. - **Related full-weight Heretic reference:** [scrunter/Qwen3-VL-235B-A22B-Thinking-heretic](https://huggingface.co/scrunter/Qwen3-VL-235B-A22B-Thinking-heretic). This related 235B model is not the tensor source for this release. - **ROCmFP4 implementation, quantizer, and runtime:** [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX). - **Strix Halo deployment stack:** [hec-ovi/llama-vulkan-strix](https://github.com/hec-ovi/llama-vulkan-strix). - **Release inspiration:** [plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF](https://huggingface.co/plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF). - **Conversion, validation, benchmarking, and packaging:** `vmlinux`. Please preserve this lineage, the Apache 2.0 license, and a description of your changes when redistributing derivatives. ## License The distributed model derivative is provided under the upstream Apache License 2.0. The included `LICENSE` is copied from the pinned full-weight source. Runtime and tooling repositories retain their own licenses: Heretic software is AGPL-3.0, while ROCmFPX and the deployment tooling are separately licensed. No runtime source code is bundled in this model repository.