JC1DA/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-INT4-Autoround

Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16 (W4, group 128)

4-bit quantized version of DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1, produced with AutoRound.

Quantization details

Scheme	W4A16 (4-bit weights, 16-bit activations)
Group size	128
Symmetric	yes
Format	auto_round:auto_gptq (GPTQ-compatible packing)
Calibration data	NeelNanda/pile-10k
Samples / seq len	256 × 4096
Iters / batch size	512 / 4
Seed	42
AutoRound version	0.12.3

Layers kept in 16-bit (not quantized): lm_head, embed_tokens, the vision tower (model.visual.*), the MTP module (mtp.*), and the linear_attn.in_proj_a / in_proj_b projections in every language-model layer.

Model size

• Total (all shards): ~18.7 GB (17.4 GiB)
• Shards: 10 × model-*.safetensors + model_extra_tensors.safetensors

Usage

The weights are GPTQ-compatible, so any loader supporting that format (vLLM, llama.cpp, transformers with optimum, etc.) should work:

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Hello!"}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
).to(model.device)

out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

Usage with vLLM

Verified with vLLM 0.27.1 (loads the AutoRound/GPTQ-packed weights natively, fuses norm_quant/act_quant ops; first run compiles flashinfer/Triton kernels, which are then cached):

from vllm import LLM, SamplingParams

llm = LLM(
    model="DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16",
    dtype="bfloat16",
    max_model_len=16384,
    enforce_eager=True,  # optional: skip cuda-graph capture for faster startup
)

out = llm.generate(
    ["Prove that the square root of 2 is irrational."],
    SamplingParams(temperature=1.0, top_p=0.95, top_k=20, max_tokens=512),
)
print(out[0].outputs[0].text)

Or serve it as an OpenAI-compatible API:

vllm serve DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16 \
    --dtype bfloat16 --max-model-len 16384 --port 8000

The model fits comfortably on a single 80 GB GPU (~21 GB weights + KV cache).

Quantization command

auto-round \
     --model /path/to/DavidAU_Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 \
     --scheme W4A16 \
     --format auto_round \
     --dataset NeelNanda/pile-10k \
     --nsamples 256 \
     --seqlen 4096 \
     --batch_size 4 \
     --iters 512 \
     --device_map 0 \
     --seed 42 \
     --ignore_layers "model.language_model.layers.*.linear_attn.in_proj_a,model.language_model.layers.*.linear_attn.in_proj_b,lm_head,model.language_model.embed_tokens,model.visual.*,mtp.*" \
     --output_dir /path/to/DavidAU_Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-W4A16

Benchmarks

Evaluated against the bf16 base model on 2× A100 80 GB (one model per GPU, in parallel) using lm-evaluation-harness on vLLM 0.27.1 (dtype bfloat16). Sampling for generative tasks follows the model's generation_config.json (temperature 1.0, top_p 0.95, top_k 20); single run.

Benchmark	Setup	Base (bf16)	This model (W4A16)	Δ
WikiText2 PPL ↓	full test set	8.4621	8.6318	+0.17
MMLU	full test set (14,042 questions), 5-shot log-likelihood	83.78%	83.44%	−0.34 pt
HumanEval	full test set (164 problems), 0-shot, pass@1, sampled	80.49%	78.66%	−1.83 pt
ARC-Easy	full validation (2,376 questions), 5-shot log-likelihood	84.60 ± 0.74%	84.47 ± 0.74%	−0.13 pt
BBH (macro avg, 27 subjects)	full set (6,511 questions), 3-shot CoT	89.69%	88.98%	−0.72 pt

Observations:

• HumanEval: the −1.8 pt loss is within the ±3.1 pt standard error — flat.
• WikiText2 PPL: +0.17 absolute, the expected small increase for 4-bit weights.
• MMLU: the −0.34 pt loss is within the standard error (~±0.38 pt at n=14,042) — effectively flat.
• ARC-Easy: the W4A16 loss is within the margin of error.
• BBH: the loss is concentrated in hard multi-step reasoning — logical deduction seven objects drops the most (65.2% → 49.2%); most other subjects are within ±1 pt, and several (e.g. disambiguation QA, salient translation error detection) are flat or slightly better.

Notes

• Benchmark results above are close to the bf16 base model (PPL +0.17, MMLU −0.34 pt, HumanEval −1.8 pt, ARC −0.13 pt, BBH −0.72 pt macro); please verify on your own workloads for your use case.
• Inference is the intended use; the checkpoint is not fine-tunable in the usual way (4-bit packed weights).