Hugging Face's logo

Vishva007
/
Qwen3.8-9B-Distill-W4A16-AutoRound-LLM-Compressor

Image-Text-to-Text
Safetensors
qwen3_5
quantization
auto-round
gptq
llm-compressor
compressed-tensors
4bit
text-generation
reasoning
distillation
conversational
Model card Files
xet
 Community
Qwen3.8-9B-Distill-W4A16-AutoRound-LLM-Compressor / README.md
Vishva007's picture
Vishva007
Update README.md
d64fda2 verified
preview code
|
Raw History Blame Contribute Delete
6.12 kB
metadata 
license: apache-2.0
base_model: empero-ai/Qwen3.8-9B-Distill
tags:
  - quantization
  - auto-round
  - gptq
  - llm-compressor
  - compressed-tensors
  - 4bit
  - text-generation
  - image-text-to-text
  - reasoning
  - distillation
pipeline_tag: image-text-to-text

Qwen3.8-9B-Distill (W4A16 Quantized via AutoRound)

This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of empero-ai/Qwen3.8-9B-Distill, quantized using Intel's AutoRound algorithm.

⚡ Quantization Details

Calibrated and quantized with fine-grained group sizes and high iteration depth to preserve reasoning traces (<think> blocks) and multimodal capabilities:

  • Algorithm: Intel AutoRound
  • Precision / Scheme: W4A16 (4-bit weights, 16-bit activations)
  • Group Size: 32 (fine-grained reconstruction fidelity)
  • Symmetric (sym): True
  • Calibration Samples (nsamples): 512
  • Sequence Length (seqlen): 4096
  • Tuning Iterations (iters): 1000 (Production-grade accuracy)
  • Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
  • Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

📦 Available Formats

Depending on your inference engine, choose the appropriate repository:

🚀 Usage & Quickstart

1. High-Throughput Serving via vLLM 

# Using the LLM-Compressor / Compressed-Tensors build
vllm serve Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-LLM-Compressor \
    --dtype bfloat16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

📊 VRAM & Performance Benefits

  • Original Model (BF16): ~8–10 GB VRAM required for full context inference
  • Quantized Model (W4A16 Group 32): ~2.5–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs, laptops, and edge devices)
  • Throughput: Lowers memory bandwidth pressure, accelerating token generation speeds during extended chain-of-thought (<think>) reasoning.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

Template CUDA Version Docker Image Template ID Deploy
PyTorch 2.14 (CUDA 12.6) 12.6 vishva123/cuda-12.6-pytorch-2.14-runpod d7lxsa4w9m Deploy to RunPod
PyTorch 2.14 (CUDA 13.0) 13.0 vishva123/cuda-13.0-pytorch-2.14-runpod yk0y6j6rpg Deploy to RunPod
PyTorch 2.14 (CUDA 13.2) 13.2 vishva123/cuda-13.2-pytorch-2.14-runpod gsp4gwx0nw Deploy to RunPod

PyTorch 2.13

Template CUDA Version Docker Image Template ID Deploy
PyTorch 2.13 (CUDA 12.6) 12.6 vishva123/cuda-12.6-pytorch-2.13-runpod gmlupxnxfk Deploy to RunPod
PyTorch 2.13 (CUDA 13.0) 13.0 vishva123/cuda-13.0-pytorch-2.13-runpod y3j8xvk4f4 Deploy to RunPod
PyTorch 2.13 (CUDA 13.2) 13.2 vishva123/cuda-13.2-pytorch-2.13-runpod vigpissn5w Deploy to RunPod

PyTorch 2.12

Template CUDA Version Docker Image Template ID Deploy
PyTorch 2.12 (CUDA 12.6) 12.6 vishva123/cuda-12.6-pytorch-2.12-runpod ctmz86zmf0 Deploy to RunPod
PyTorch 2.12 (CUDA 13.0) 13.0 vishva123/cuda-13.0-pytorch-2.12-runpod qjko5yiwzi Deploy to RunPod
PyTorch 2.12 (CUDA 13.2) 13.2 vishva123/cuda-13.2-pytorch-2.12-runpod ifg6xmye0f Deploy to RunPod

📚 Acknowledgments