--- language: - en - vi - zh license: apache-2.0 tags: - moe - qwen - code - coding-agent - int8 - selective-quantization - moe-slice - text-generation-inference - transformers - vllm base_model: Jab1718/qwen3.8-flash-coder-85gb-bf16 pipeline_tag: text-generation --- # ⚡ Qwen3.8-Flash-Coder-44GB-Selective-INT8 (160 Experts Hardware-Aligned Subnet) [![GitHub Toolkit](https://img.shields.io/badge/GitHub-Jab1718%2FMoe--slices-blue?logo=github)](https://github.com/Jab1718/Moe-slices) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0) [![BF16 Parent](https://img.shields.io/badge/BF16%20Parent-Jab1718%2Fqwen3.8--flash--coder--85gb--bf16-orange)](https://huggingface.co/Jab1718/qwen3.8-flash-coder-85gb-bf16) **`Qwen3.8-Flash-Coder-44GB-Selective-INT8`** is a high-performance, selective-quantized Mixture-of-Experts (MoE) coding model. Sliced down from the monolithic **`Qwen/Qwen3.8-Flash-Next` (335GB)** and quantized from the **`Qwen3.8-Flash-Coder-85GB-BF16`** parent checkpoint, this model reduces disk and VRAM footprint to exactly **44.29 GB** (a **44.2% VRAM reduction** and **86.8% reduction from base**), enabling full zero-offload deployment on **only 2x 32GB GPUs** (e.g. 2x NVIDIA RTX 5000 Ada, 2x RTX 4090/3090, or 1x A100/H100 80GB). --- ## 🔬 Selective Quantization Architecture Traditional MoE post-training quantization often quantizes all layers uniformly, which severely degrades the Router Gate and causes **Routing Collapse** (routing tokens to sub-optimal experts). This checkpoint introduces **Selective MoE Quantization**: 1. **Critical High-Precision Modules (Kept in 100% Native BF16):** - **Router Gates:** Retain 100% floating-point routing fidelity across all 48 layers. - **Multi-Head Self-Attention & Linear Attention:** `q_proj`, `k_proj`, `v_proj`, `o_proj`. - **Shared Expert, RMSNorms, Embeddings & LM Head:** Zero quantization loss in embedding projections. 2. **High-Capacity Sparse Experts (Quantized to Symmetric Per-Channel INT8):** - 160 MoE Experts across 48 layers (`gate_up_proj`, `down_proj`). - Symmetrically quantized per-channel with dynamic scaling vectors (`gate_up_proj_scale`, `down_proj_scale`). --- ## 📊 Technical Specifications | Parameter | Original Monolith (`Qwen3.8-Flash-Next`) | BF16 Parent Checkpoint | **Selective MoE INT8 (This Checkpoint)** | | :--- | :--- | :--- | :--- | | **Disk / VRAM Size** | ~335 GB (131 Shards) | 85.24 GB (2 Shards) | **44.29 GB (2 Shards: 25.3GB + 19.0GB)** | | **Numerical Format** | Bfloat16 | Bfloat16 | **Selective INT8 (Router BF16 + Experts INT8)** | | **Layers / Total Experts** | 48 Layers / 512 Experts | 48 Layers / 160 Experts | **48 Layers / 160 Experts** | | **Active Experts / Token**| 10 Experts | 8–10 Experts | **8 Active Experts** | | **Required Hardware** | 8x H100 (80GB) Cluster | 3x RTX 5000 Ada (32GB) | **2x RTX 5000 Ada (32GB) or 2x RTX 4090 (24GB)** | | **Per-GPU Memory Usage** | >45 GB / GPU (8x GPUs) | ~27.3 GB / GPU (3x GPUs) | **~22.1 GB / GPU (2x GPUs)** | | **Toolkit** | — | [**`moe-slice`**](https://github.com/Jab1718/Moe-slices) | [**`moe-slice`**](https://github.com/Jab1718/Moe-slices) | --- ## 🏆 Empirical Sandbox Benchmark Results (100 Real-World Tasks) The model was rigorously tested across an isolated execution-based sandbox benchmark covering 100 challenging tasks in systems engineering, algorithms, and autonomous coding agents: | Language / Domain | Tested Suite | Pass@1 Accuracy | Verified Engineering Competencies | | :--- | :---: | :---: | :--- | | ⚡ **C++ (Modern C++20)** | 10 Tasks | **100.0% (10/10)** | Concurrency (`ThreadSafeQueue`, `AtomicCounter`), Smart Pointers, C++20 Concepts, Templates | | 🦀 **Rust (Systems)** | 10 Tasks | **100.0% (10/10)** | Tokio Async MPSC, Safe Mutex, Iterators, Borrow Checker, Pattern Matching, Traits | | 🐹 **Go (Golang Systems)** | 5 Tasks | **80.0% (4/5)** | Worker Pools, Channel Synchronization, Struct JSON Marshal, Binary Search Slice | | 🌐 **TypeScript (Fullstack)**| 5 Tasks | **80.0% (4/5)** | Generic Debounce, Promise Retry, Generic Deep Clone, Zod-like Schema Validator | | 🤖 **Coding Agent** | 20 Tasks | **80.0% (16/20)** | Automated Debugging (100%), Code Refactoring & Diff Patches (100%), Fill-in-the-Middle (FIM) | | 🐍 **Python Algorithms** | 50 Tasks | **78.0% (39/50)** | Dynamic Programming, Tree Structures (BST, LCA, Trie), Binary Search, Sorting | | 📊 **TOTAL BENCHMARK** | **100 Tasks** | **83.0% (83/100)** | **Real Multi-Language Isolated Sandbox Code Execution** | > [!NOTE] > Compared to the original un-tuned base model (67.0%), this 44.3GB Selective INT8 checkpoint achieves a **+16.0% absolute Pass@1 increase** while slashing memory consumption by nearly half. --- ## 🚀 Quickstart & Inference To achieve high-throughput inference with on-demand vectorized dequantization across 2 GPUs: ```python import os os.environ["CUDA_VISIBLE_DEVICES"] = "0,1" # 2x GPUs import torch from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM model_id = "Jab1718/qwen3.8-flash-coder-44gb-selective-int8" config = AutoConfig.from_pretrained(model_id, trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) # Device mapping across 2 GPUs (Embeddings + Layers 0..23 on GPU 0; Layers 24..47 + Head on GPU 1) device_map = { "model.embed_tokens": "cuda:0", "model.rotary_emb": "cuda:0", "model.hyper_connection_mixer": "cuda:1", "model.norm": "cuda:1", "lm_head": "cuda:1" } for i in range(24): device_map[f"model.layers.{i}"] = "cuda:0" for i in range(24, 48): device_map[f"model.layers.{i}"] = "cuda:1" # Load model weights model = AutoModelForCausalLM.from_pretrained( model_id, device_map=device_map, trust_remote_code=True ) prompt = "Write a lock-free thread-safe queue in C++20 using atomic operations." messages = [ {"role": "system", "content": "You are an expert modern C++20 systems engineer."}, {"role": "user", "content": prompt} ] formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tokenizer(formatted, return_tensors="pt").to("cuda:0") with torch.inference_mode(): outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False) print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)) ``` --- ## 📜 Citation & Acknowledgements ```bibtex @software{moe_slices_qwen38_int8, author = {Thai Nguyen}, title = {Qwen3.8-Flash-Coder-44GB-Selective-INT8: 44.3GB Hardware-Aligned Coding Subnet}, url = {https://github.com/Jab1718/Moe-slices}, year = {2026} } ```