amaan784/Qwen2.5-VL-7B-AWQ-W4A16-substation

Qwen2.5-VL-7B-AWQ-W4A16 (substation-domain calibrated)

A domain-calibrated INT4 quantization of Qwen/Qwen2.5-VL-7B-Instruct for industrial visual inspection (substation, pump impeller, turbine blade, motor thermal). The text tower is W4A16-quantized via AWQ/GPTQ; the native Qwen2.5-VL vision tower and visual merger are kept in FP16.

Built by HPML Team 23 (Columbia, Spring 2026). Companion to the AssetOpsBench-VLM-22 benchmark.

Headline result on AssetOpsBench-VLM-22

Variant	E2E mean (ms)	E2E p50 (ms)	GPU used (MiB)
FP16 baseline	9437	5915	18268
W4A16 domain (this model)	4752 (1.99x)	4312	18264
W4A16 generic calibration	4708 (2.00x)	4356	18264

Single L4 (24 GiB), vLLM 0.19, sequential single-batch traffic, 22 scenarios. The GPU column is the per-run vram_used_mib metric from results/hpml_metrics.csv; vLLM can reuse freed model-weight memory for the KV-cache pool, so this is not the packed checkpoint size.

The interesting result is not the speedup (both calibrations get 2x). It is the failure-mode difference: on the Llama-LLaVA-NeXT family with the same recipe, generic calibration produced a runaway-generation failure mode in the final 22-scenario sweep: two Llama L1g transformer scenarios timed out around the token cap (122 s), dropping its judged denominator from 44 to 40. Domain calibration did not trigger that failure. The Qwen family does not show this on this benchmark, but the calibration-data risk is real and we recommend using the domain-calibrated weights for inspection workloads. See the dataset card for the full discussion.

Quantization recipe

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head", "re:visual.*", "re:model.visual.*"],
)
oneshot(
    model, tokenizer,
    dataset=domain_calibration_set,    # 64 prompts sampled from the domain corpus
    recipe=recipe,
    max_seq_length=512,
    num_calibration_samples=64,
    sequential_targets=["Qwen2_5_VLDecoderLayer"],  # resolved dynamically in script
    pipeline="basic",
)
model.save_pretrained(out, save_compressed=True)  # pack-quantized format

The native Qwen2.5-VL vision encoder and visual-text merger stay FP16. INT4 on the vision tower is well-documented to degrade Qwen2.5-VL output quality.

Calibration data

The project calibration corpus contains 128 hand-authored substation-inspection text prompts spanning equipment identification, defect descriptions, condition assessments, and gauge readings. This checkpoint used 64 sampled prompts with max_seq_length=512, matching the submitted reproduction command for models/qwen2.5-vl-7b-awq-domain. Prompts were authored by the team; no real customer data is included.

Usage

vLLM (recommended)

vllm serve amaan784/Qwen2.5-VL-7B-AWQ-W4A16-substation \
  --quantization compressed-tensors \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --limit-mm-per-prompt image=1

transformers

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "amaan784/Qwen2.5-VL-7B-AWQ-W4A16-substation",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "amaan784/Qwen2.5-VL-7B-AWQ-W4A16-substation"
)

Pinned stack

These weights were validated on the following exact versions only:

• vLLM 0.19.0
• llmcompressor 0.10.0.2
• compressed-tensors 0.14.0.1
• transformers 4.57.6
• torch 2.10.0+cu129

Older llmcompressor (<=0.3) can produce fake-quant FP16 outputs without save_compressed=True. For Qwen2.5-VL, the submitted script uses the pipeline="basic" path to avoid the FX tracing failure around multimodal RoPE, then verifies the packed compressed-tensors checkpoint before benchmarking.

Limitations

• Validated only on a 22-scenario rubric-graded benchmark; accuracy on general VQA / TextVQA / MMBench is not measured here and may regress.
• Vision tower stays FP16; total VRAM win is from the LLM tower only.
• Single-batch sequential traffic. Throughput at concurrency was not measured.
• Calibration set is substation/inspection-themed. On out-of-domain prompts (e.g. open-domain VQA) we expect quality to degrade more than the generic-calibrated baseline would.

License

Apache-2.0, inherited from the base Qwen/Qwen2.5-VL-7B-Instruct.

Citation

@misc{qwen25vl-awq-substation,
  title  = {Qwen2.5-VL-7B-AWQ-W4A16 (Substation-Domain Calibrated)},
  author = {HPML Team 23 (Columbia University)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/amaan784/Qwen2.5-VL-7B-AWQ-W4A16-substation}}
}