--- base_model: Kwaipilot/KAT-Coder-V2.5-Dev library_name: gguf tags: - gguf - apex - custom-quantization - unsloth-studio - moe - coding - agentic - code - llama.cpp - qwen35moe - quantized - quantization license: apache-2.0 language: - en - zh - es - fr - de - pt - it - ru - ja - ko - vi - th - ar pipeline_tag: text-generation quantized_by: IsValorum --- ## Quick Navigation Index 1. [Independent Benchmark of the APEX-I-MiniPlus Family (Occamy V2 Reference)](#toc-01) 2. [Model Files & Technical Specifications](#toc-02) 3. [Comparative Quantization Analysis (vs. Flat Quants & Generic APEX)](#toc-03) 4. [Everyday Laptop Benchmarks (DDR4 / DDR5 RAM)](#toc-04) 5. [Estimated Projections on Consumer Hardware](#toc-05) 6. [The 24GB Miracle: Full 256K Context Runs In VRAM!](#toc-06) 7. [Hardware Throughput Projections (RTX 30 / 40 / 50)](#toc-07) 8. [Recommended Generation Parameters (Kwaipilot Official)](#toc-08) 9. [Optional Support](#toc-09) > [!NOTE] > ### ARCHITECTURE SELECTION GUIDE — MINIPLUS V2 & V2.1 EDITIONS > This repository hosts the **MiniPlus V2** edition of **KAT-Coder-V2.5-Dev**. Our releases are precision-engineered for specific hardware budgets and memory topologies. **V2 is NOT obsolete or "worse"; each edition serves distinct inference requirements:** > > - **MiniPlus V2 (High Theoretical Layer Protection):** On paper, V2 provides extra protective envelopes on edge layers (10 layers in `IQ3_S` + `IQ4_NL` shared experts + `Q8_0` attention gates). However, **in practical inference benchmarks—even across extreme long-context windows exceeding +160K tokens—there is virtually NO perceptible difference in quality or reasoning compared to V2.1.** > - **MiniPlus V2.1 (System RAM Streaming Specialist with Deep Context):** Specially prepared to run **totally or partially in system RAM (DDR4/DDR5)** across large codebase contexts (up to 256k tokens). By replacing non-linear codebooks with linear `Q3_K` edge experts, keeping `Q8_0` attention gates, and upgrading shared foundation experts to `Q5_K` across all 40 layers, **it completely eliminates AVX2 CPU dequantization stalls (+24 to 28+ tok/s streaming)**. Depending on your processor and memory bandwidth (DDR4/DDR5), **streaming generation in system RAM can be almost as fast as having everything in VRAM**, while supporting deep context reserving GPU VRAM for the codebase KV cache while model weights stream from system RAM. It provides this massive RAM streaming acceleration for **only approx. 100 MB more**, which is completely negligible in system RAM. > > **Which one should you choose? (Official Recommendation: V2.1)** > - **⭐ PRIMARY RECOMMENDATION — [KAT-Coder-V2.5-Dev APEX-I-MiniPlus V2.1](https://huggingface.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF):** For virtually all users and deployments, **V2.1 is the strictly recommended release**. Empirically verified on WikiText-2, V2.1 achieves an outstanding **Perplexity of 5.5045 ± 0.1330** (ΔPPL ≈ +0.06 from unquantized baseline (approx. 5.44)), matching the token fidelity of **Q5_K / Q6_K** class quantizations while weighing only **approx. 14.7 GB** (same footprint as Q3_K_M). Furthermore, it completely eliminates AVX2 CPU stalls, providing blistering **+24 to 28+ tok/s streaming** under system RAM offload. > - **MiniPlus V2 Legacy:** Maintained for architectural transparency and users seeking specialized configurations for their workflow. > > *Both editions are handcrafted and vastly outperform flat 3-bit quants and generic community APEX-I-Mini releases.* > To explore or download the **V2.1** edition of KAT-Coder-V2.5-Dev optimized for system RAM streaming, visit: > **[IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF](https://huggingface.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF)** > [!WARNING] > ### DO NOT CONFUSE APEX-I-MINIPLUS WITH GENERIC COMMUNITY APEX-I-MINI! > **Regardless of release version (whether V1, V2, or V2.1), NEVER confuse handcrafted APEX-I-MiniPlus builds with generic community APEX-I-Mini releases:** > - **Generic Community APEX-I-Mini:** Uniformly compresses all core MoE experts down to aggressive 2-bit `IQ2_S` (dropping below the critical quality floor), leaves the sensitive token output head unarmored at 3-bit `Q3_K_M`, and compresses attention projections down to `Q3_K`. In deep reasoning models, this triggers severe perplexity spikes, syntax errors, and broken code brackets. > - **Handcrafted APEX-I-MiniPlus (All Editions by IsValorum):** Every single MiniPlus release—from V1 and V2 to V2.1—is a custom tensor-by-tensor architecture that preserves uncompressed `F32` router gates, armors the token output head in high-precision `Q6_K`, safeguards attention gates in `Q8_0`, and keeps core reasoning experts at or above calibrated 3-bit (`IQ3_XXS`/`IQ3_S`). Even our earlier builds vastly outperform generic community APEX recipes and flat 3-bit quants. > [!TIP] > ### SYSTEM RAM INFERENCE: FULL OR PARTIAL > This APEX-I-MiniPlus release is designed for **full or partial system-RAM inference**. Depending on the processor, memory bandwidth, and DDR4/DDR5 configuration, generation can range from **20 to 45 tok/s**. With partial GPU offload, systems that cannot fit **128K or more context** entirely in VRAM can place the remaining model and context load in system RAM, maintaining stable, responsive generation at longer context lengths. --- > [!IMPORTANT] > ### EXPLORE THE ESTABLISHED 35B MoE MINIPLUS LINEUP > These are complementary APEX-I-MiniPlus V2.1 releases, not alternate downloads of the same model. Each receives the same tensor-by-tensor approach, integrated MTP where supported, and a design suitable for full or partial system-RAM inference. Choose the model whose native strengths best fit the work you want to do: > > - **[Qwen3.6-35B-A3B MTP APEX-I-MiniPlus-V2.1](https://huggingface.co/IsValorum/Qwen3.6-35B-A3B-MTP-APEX-I-MiniPlus-V2.1-GGUF)** — a versatile frontier MoE for broad reasoning, multilingual work, agents, tool use, and multimodal tasks. > - **Best for:** General reasoning, agent workflows, tool calling, and flexible multimodal use. > - **[Qwen3.6-35B-A3B MTP APEX-I-MiniPlus-V2.1 Abliterated](https://huggingface.co/IsValorum/Qwen3.6-35B-A3B-MTP-APEX-I-MiniPlus-V2.1-Abliterated-GGUF)** — the V2.1 refusal-ablated Qwen3.6 edition for users who deliberately prefer reduced refusal behavior. > - **Best for:** Workflows where an abliterated Qwen3.6 variant is explicitly desired. > - **[Ornith 1.5 APEX-I-MiniPlus-V2.1](https://huggingface.co/IsValorum/Ornith-1.5-35B-A3B-APEX-I-MiniPlus-V2.1-GGUF)** — a software-engineering-focused MoE designed for repository-scale coding and autonomous engineering agents. > - **Best for:** Repository-scale development, multi-file code changes, and software-engineering agents. > - **[Tiel Coder APEX-I-MiniPlus-V2.1](https://huggingface.co/IsValorum/Tiel-Coder-35B-A3B-APEX-I-MiniPlus-V2.1-GGUF)** — a specialist coding MoE tuned for agentic programming, iterative tool use, and implementation-heavy work. > - **Best for:** Focused coding sessions, iterative debugging, and tool-driven implementation. > > These remain distinct model families and editions with their own behavior and empirical results. Pick by workload and intended alignment behavior rather than treating them as interchangeable quantization variants. --- ### 🏅 Independent Benchmark of the APEX-I-MiniPlus Family (Occamy V2 Reference) > [!NOTE] > **External report:** [zephel01 independently benchmarked Occamy V2](https://note.com/zephel01/n/n71d3d7e6b70c?hl=en). The benchmark below was performed on **Occamy-1.0 APEX-I-MiniPlus V2**, not on this specific model. It is included as independent evidence of the broader MiniPlus quantization approach. The **APEX-I-MiniPlus** quantization architecture powering this model was subjected to an extensive independent evaluation by Japanese AI researcher and evaluator [zephel01 (CoolZero)](https://note.com/zephel01/n/n71d3d7e6b70c?hl=en) on an **NVIDIA RTX 5090 (32GB)** workstation running `llama.cpp` CUDA `b11027` with FlashAttention (`-fa on -ctk q8_0 -ctv q8_0 -ngl 99`). The evaluation tested the APEX-I hybrid MoE engine across **348 unseeded trials** on SWE-bench style multi-file Python bug-fixing tasks with hidden `pytest` suites (`llmbench`): - **L6 Multi-File Code Generation (60 tasks):** - **Context 32,768 (32K):** **93.3% Resolved** (46/60 tasks passed 5/5 consecutive trials; 20/20 on Easy–Hard). - **Context 65,536 (65K):** **90.0% Resolved** (45/60 tasks passed 5/5 consecutive trials). - **Match with 25–28 GB Models:** Matches or exceeds the resolution rate of full 25–28 GB models (such as `Ornith-1.5` and `Tiel-Coder` 35B-A3B) while consuming **over 10 GB less VRAM** (14.6 GB vs approx. 26 GB). - **Extreme Context VRAM Scaling (The Hybrid DeltaNet SSM Advantage):** - **32K Context:** **14.6 GB** total VRAM allocation. - **65K Context:** **15.1 GB** total VRAM allocation (only **+0.5 GB VRAM** added when doubling context!). - *Architectural Explanation:* Because 30 of the 40 layers utilize Linear Attention / DeltaNet SSM ($O(1)$ constant recurrence memory), only the 10 full-attention anchor layers expand the KV cache. This proves empirically that **65,536 context runs 100% in VRAM on consumer 16GB GPUs (RTX 4080 / RTX 5080)** without offloading to system RAM. - **Measured Real-World Throughput:** Sustained single-stream generation of **approx. 247 – 251 tok/s** on NVIDIA RTX 5090. --- ## Model Files & Technical Specifications | File Name | File Size | Memory Footprint | BPW | Description | | :--- | :--- | :--- | :--- | :--- | | **`KAT-Coder-V2.5-Dev.APEX-I-MiniPlus-V2.gguf`** | **`14.64 GB` (`13.64 GiB`)** | `13.64 GiB` | **3.38 BPW** | Core agentic code synthesis, syntax verification, refactoring & logic | - **Base Architecture:** `Qwen3_5MoeForConditionalGeneration` (40 layers, 256 fine-grained micro-experts with intermediate dimension 512, 8 active per token). - **Active Parameters:** **approx. 3.2B active parameters per token** (delivering small-model throughput with 35B-scale reasoning). - **Quantization Profile:** Armored boundary layers (`IQ3_S` / `IQ4_NL`), deep core expert compression (`IQ3_XXS` + `imatrix`), uncompressed router gates (`F32`), and high-precision syntax output head (`Q6_K`). - **Memory Footprint:** Ultracompact 13.64 GiB footprint engineered specifically to avoid Out-Of-Memory (OOM) crashes on 16GB and 24GB VRAM hardware. --- ## Comparative Quantization Analysis (vs. Flat Quants & Generic APEX) Also, don't confuse **APEX-I-MiniPlus-V2** with a generic baseline APEX-I-Mini. Traditional APEX-I-Mini drops core experts aggressively to 2-bit `IQ2_S` and leaves `output.weight` at 3-bit `Q3_K_M`, which creates a noticeable perplexity hit on complex reasoning tasks. V2 was specifically re-engineered to avoid that quality floor (keeping core experts at calibrated `IQ3_XXS`, output in `Q6_K`, shared expert in non-linear `IQ4_NL`, and routers in `F32`). To put the numbers in perspective: this cuts nearly **2 GB off a flat 3-bit quant** (approx. 15.6 GB), and weighs only about **approx. 1 GB more than a generic APEX-I-Mini** (approx. 12.5 GB). For that single extra gigabyte of VRAM, you get a massive jump in reasoning and syntactic stability. Take a look at the tensor-by-tensor comparison table below to inspect the exact architectural differences and see why this specific allocation is optimal. That's specifically what this was built for: | Architectural Component | Generic Automated Quants (Flat `Q3_K_S` / `IQ3_S`) | Generic APEX-I-Mini (Baseline Recipe) | Our Handcrafted APEX-I-MiniPlus-V2 (IsValorum) | Perceived Quality & Real-World Impact | | :--- | :--- | :--- | :--- | :--- | | **Output Head (`output.weight`)** | Flat **`IQ3_S` / `Q3_K_S`** (approx. 3.44 BPW) | Inherits base type **`Q3_K_M`** (approx. 3.44 BPW unarmored) | **`Q6_K`** (approx. 6.56 BPW uncompromised) | **Eliminates Syntax & Vocabulary Hallucinations:** Low-bit output heads cause tokenizer classification noise, breaking code indentation, brackets (`{}`, `[]`), math symbols, and domain terms. `Q6_K` preserves near-FP16 output classification. | | **Expert Routers (`ffn_gate_inp.weight`)** | Blindly quantized to 3-bit / unoptimized | Inherits base type **`Q3_K_M`** (approx. 3.44 BPW compressed) | **`F32` uncompressed** (32.0 BPW, 2 MB/layer) | **Zero Router Drift:** In micro-expert models, even minuscule quantization errors in router logits misdirect tokens to wrong experts. Retaining uncompressed `F32` guarantees 100% routing fidelity with virtually zero memory overhead (approx. 80 MB total). | | **Attention & Language (`attn_output`, `attn_qkv`)** | Flat **`IQ3_S` / `Q3_K_S`** | **`Q3_K`** on 34 middle layers (L3–36), **`Q4_K`** on 6 edge layers | **`Q6_K` for `attn_output`**, **`IQ3_S` for `attn_qkv`** | **Contextual Retrieval Precision:** Generic APEX reduces attention and language projections to `Q3_K` across 85% of layers. Our V2 build protects attention output in high-precision `Q6_K` and uses calibrated non-linear `IQ3_S`, ensuring flawless needle-in-a-haystack retrieval across deep 128k–256k context windows. | | **Attention Gates (`attn_gate.weight`)** | Blindly compressed to 3-bit | Compressed to **`Q3_K`** (middle) / **`Q4_K`** (edges) | **`Q8_0`** (8.50 BPW) | **Attention Head Stability:** Attention gates modulate query-key routing across hybrid attention layers. Keeping them in 8-bit prevents attention crosstalk and hallucination over long contexts. | | **Shared Foundation Expert (`ffn_*_shexp`)** | Flat **`IQ3_S` / `Q3_K_S`** (3.44 BPW) | Linear **`Q4_K`** (middle) / **`Q5_K`** (edges) | **`IQ4_NL`** (4.50 BPW non-linear codebook) | **Foundational Knowledge Armor:** The shared expert executes for 100% of tokens. In 256 micro-expert models, `IQ4_NL` non-linear codebooks preserve heavy-tailed outlier representations far better than standard linear quantization. | | **Core MoE Layers (Middle: 10–29)** | Flat **`IQ3_S` / `Q3_K_S`** (uniform bit-rate across all layers) | Aggressive **`IQ2_S` (2.50 BPW)** | **`IQ3_XXS` (3.06 BPW) + calibrated `imatrix`** | **Above the Quality Threshold:** Generic 2-bit `IQ2_S` baselines drop below the critical quality floor for 35B MoEs, resulting in perplexity spikes on reasoning tasks. Our `IQ3_XXS` with imatrix achieves deep compression (272 MiB → 98 MiB per block) without sacrificing logic. | | **Edge MoE Layers (Layers 0–9 & 30–39)** | Flat **`IQ3_S` / `Q3_K_S`** (no layer-wise gradient) | `Q3_K` (limited to first/last 5 layers only: L0–4, L35–39) | **`IQ3_S` (expanded to 10 input & 10 output layers)** | **Protected Ingestion & Synthesis:** Half of the model's layers (10 at input, 10 at output) form a non-linear armored envelope, preventing prompt misunderstanding and token degeneration across 256 micro-experts. | | **Normalization & Biases** | Often degraded | Standard | **`F32` uncompressed** | **Numerical Stability:** Prevents cumulative floating-point underflow/overflow across deep 40-layer computation. | --- ## Everyday Laptop Benchmarks (DDR4 / DDR5 RAM) ### *Estimated Projections on Consumer Hardware* You do not need an expensive workstation to run a cutting-edge 35B Mixture-of-Experts coding model. Estimated throughput projections on a standard consumer laptop (Intel Core i5 / AMD Ryzen, 4GB/6GB Laptop GPU, 32GB DDR4/DDR5 RAM): - **GPU VRAM Allocation:** Uses only **approx. 3.8 GB VRAM** (fits effortlessly on budget 4GB/6GB laptop GPUs such as RTX 3050, 4050, or 2060). - **System Memory Offload:** Standard 32GB system RAM accommodates the remaining layers. - **Estimated Document / Code Ingestion (Prefill):** **300 to 450+ tokens/second** sustained across long prompt files. - **Estimated Streaming Generation:** **20 to 24+ tokens/second** sustained output across system RAM! > [!TIP] > **Pro Tip for Consumer Laptop Users:** > Because the bulk of the model runs from system memory in partial offload mode, standard autoregressive generation streams seamlessly at **20 to 24+ tokens/second** across everyday DDR4/DDR5 memory buses, perfectly sufficient for real-time IDE pair programming! --- ## The 24GB Miracle: Full 256K Context Runs In VRAM! For developers running 24GB GPUs (RTX 3090, RTX 4090, or professional workstations), standard community 3-bit or 4-bit quants weigh 15.8 to 19.5 GiB in weights alone. When combined with KV cache and compute buffers for large codebases, they trigger immediate CUDA Out-Of-Memory crashes. **KAT-Coder APEX-I-MiniPlus-V2 fits massive contexts entirely within 24GB VRAM:** | Context Length | Model Weights (Est.) | KV Cache (q8_0, 4 slots) | Compute Buffers | **Total GPU VRAM (Est.)** | Hardware Feasibility | | :--- | :--- | :--- | :--- | :--- | :--- | | **32,768 (32k)** | `13.64 GiB` | `0.58 GiB` | `1.80 GiB` | **`16.02 GiB`** | Full offload on 24GB; partial on 16GB | | **65,536 (64k)** | `13.64 GiB` | `0.92 GiB` | `1.95 GiB` | **`16.51 GiB`** | Effortless fit on 24GB GPUs | | **131,072 (128k)**| `13.64 GiB` | `1.58 GiB` | `2.22 GiB` | **`17.44 GiB`** | Effortless fit on 24GB GPUs | | **262,144 (256k)**| `13.64 GiB` | `2.92 GiB` | `2.80 GiB` | **`19.36 GiB`** | **FULL 256K CODE REPO IN VRAM!** | *Note: Projections estimate approx. 4.64 GiB of headroom remaining on 24GB cards for system display buffers and tooling.* --- ## Hardware Throughput Projections (RTX 30 / 40 / 50) When running with full GPU offload (`-ngl 99`), KAT-Coder's fine-grained MoE architecture (approx. 3.2B active parameters) unlocks extraordinary generation throughput: | Hardware Target | Offload Mode | Generation Speed (Est.) | Prompt Prefill Speed (Est.) | Engineering Highlights | | :--- | :--- | :---: | :---: | :--- | | **NVIDIA RTX 5080 / 5090 (Blackwell)** | Full GPU (`-ngl 99`) | **approx. 247 – 251 tok/s** | **2,800 – 3,900+ tok/s** | Empirically verified on RTX 5090 by zephel01 (Occamy V2 Reference) | | **NVIDIA RTX 4090 (24GB GDDR6X)** | Full GPU (`-ngl 99`) | **80 – 105+ tok/s** | **1,800 – 2,600+ tok/s** | Near-instantaneous code completion & refactoring | | **NVIDIA RTX 3090 (24GB GDDR6)** | Full GPU (`-ngl 99`) | **65 – 80+ tok/s** | **1,400 – 2,000+ tok/s** | Full 256k repository context in dedicated VRAM | | **NVIDIA RTX 4080 / 5070 (16GB)** | Partial offload (approx. 30 layers) | **35 – 45+ tok/s** | **800 – 1,200+ tok/s** | High-efficiency local coding assistant | | **Consumer Laptop (4GB GPU + 32GB RAM)**| Hybrid Offload | **20 – 24+ tok/s** | **300 – 450+ tok/s** | Smooth streaming from system DDR4/DDR5 RAM | ### ⚙️ Recommended Generation Parameters (Kwaipilot Official) Official sampling hyperparameters specified by [Kwaipilot](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) across benchmark evaluation tracks: | Evaluation Track / Workload | Temperature | Top-P | Top-K | Presence Penalty | Max Tokens | Thinking Mode | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **SWE-bench & Agent Coding (Official)** | `1.00` | `0.95` | `20` | `1.50` | `81,920` | `enable_thinking: true` | | **Terminal-Bench & Direct Code Execution** | `0.70` | `1.00` | `20` | `1.50` | `32,768` | `enable_thinking: false` | | **PinchBench & SciCode Scientific Logic** | `0.60 – 0.70` | `1.00` | `20` | `1.50` | `32,768` | `preserve_thinking: true` | > [!IMPORTANT] > > ### 🔍 Model Inherent Behavior vs. Quantization Fidelity Notice > Any behavioral nuances, stylistic tendencies, domain-specific habits, or zero-shot edge-case oversights **stem entirely from the original unquantized checkpoint weights and fine-tuning distribution, NOT from the APEX-I quantization process.** > Handcrafted APEX-I-MiniPlus strictly preserves mathematical tensor fidelity—keeping 100% of expert routing matrices (`gate_inp`) in uncompressed `F32` (zero router drift), armoring the token output head in `Q6_K`, and safeguarding attention gates in `Q8_0`. Empirical verification confirms near-zero perplexity loss (ΔPPL ≈ +0.06), ensuring that token logits, routing decisions, and reasoning trajectories are mathematically faithful to the original base model. > [!TIP] > ### 💡 Developer Tip for Autonomous Coding & CI Agents (Import Discipline) > In independent evaluations of the APEX-I-MiniPlus architecture (Occamy V2 reference), reasoning and code generation scored a remarkable 90%–93.3% resolution rate on multi-file SWE benchmarks. When deploying autonomous coding agents in production, best practices include specifying in your system prompt: *"Always declare complete, explicit import statements at the beginning of the file"* or pairing with an automated linter (`ruff`) to guarantee clean, zero-shot execution. ## Optional Support Gold Ship dancing If these MiniPlus or NanoPlus releases have been useful to you and you would like to support the work, you can do so voluntarily through https://ko-fi.com/isvalorum. Your contribution helps with evaluation, hosting, and future handcrafted quantizations. Every release will always remain free to download and use; there are no paywalled files, updates, or features.