EXAONE-4.5-33B-W4A16

Int4 weight-only quantization of LGAI-EXAONE/EXAONE-4.5-33B, in compressed-tensors format for vLLM. 22.75 GB, down from 68.70 GB.

Unofficial and unaffiliated with LG AI Research. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

⚠️ Non-commercial license

The base model is released under the EXAONE AI Model License Agreement 1.2 – NC, and this derivative inherits it in full. That license permits use solely for research and educational purposes. Commercial use of the model, of derivatives, or of their outputs is expressly prohibited without a separate written agreement with LG Management Development Institute. It also forbids using the model or its outputs to develop or improve competing models.

This is not the permissive licensing you may expect from other quantized checkpoints. Read LICENSE in full before you download, and do not deploy this in a product.

What was changed

Weights were quantized from bfloat16 to int4, group size 128, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer, chat template and processor configs are the vendor's, unmodified.

448 Linear modules were converted, covering 70.1% of the output's bytes:

component	precision	source	quantized
MLP linears (64 layers × 3)	int4 g128	53.86 GB	13.88 GB
attention projections (64 layers × 4)	int4 g128	8.05 GB	2.08 GB
vision tower (visual.*, 28 layers)	bfloat16	2.57 GB	2.57 GB
embed_tokens	bfloat16	1.57 GB	1.57 GB
lm_head (untied)	bfloat16	1.57 GB	1.57 GB
MTP head (mtp.*)	bfloat16	1.07 GB	1.07 GB
norms, biases	bfloat16	0.001 GB	0.001 GB
total		68.70 GB	22.75 GB

The model is unusually MLP-heavy — intermediate_size is 27392 against a hidden_size of 5120 — so 78% of the source checkpoint is MLP weight and quantizing it does most of the work.

Left at bfloat16:

• visual.* — vLLM builds multimodal towers with quant_config=None, so a checkpoint carrying quantized vision weights cannot be loaded.
• mtp.* — the multi-token-prediction speculator head (num_nextn_predict_layers: 1), loaded through vLLM's speculative-decoding path rather than the main stack. The base model card's own serving command enables it, so leaving it at full precision keeps that path working as the vendor intends.
• lm_head + embed_tokens — precision-sensitive, and untied here (both tensors are present, 1.57 GB each).

Usage

Requires vLLM >= 0.25.1, where Exaone4_5_ForConditionalGeneration is registered. No nightly build needed.

vllm serve GotoAI-Inc/EXAONE-4.5-33B-W4A16 \
  --served-model-name EXAONE-4.5-33B-W4A16 \
  --max-model-len 131072 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser hermes \
  --limit-mm-per-prompt '{"image": 64}' \
  --speculative_config '{"method": "mtp", "num_speculative_tokens": 3}'

Do not pass --quantization; compressed-tensors is detected from config.json. The int4 W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

The parsers are borrowed, and that is deliberate. vLLM ships no exaone tool-call or reasoning parser. The base model card specifies --reasoning-parser qwen3 (EXAONE frames thinking with <think> tags, which that parser handles) and --tool-call-parser hermes. Use those exact names.

• MTP speculative decoding works off the head already in this checkpoint — no draft model to download. The base card uses num_speculative_tokens: 3. Not smoke-tested here.
• --language-model-only frees the 2.57 GB vision tower (vLLM skips tower weights when every modality limit is zero) plus the multimodal profiling headroom, at the cost of image and video input.

Thinking

The chat template defaults enable_thinking to true, so this model thinks unless told otherwise:

{"chat_template_kwargs": {"enable_thinking": false}}   // emit an empty <think></think> and answer directly

With thinking on, --reasoning-parser qwen3 splits the <think>…</think> block into reasoning_content.

Fitting the card

64 layers: 48 sliding-attention (window 4096) and 16 full attention, all with 8 KV heads at head_dim 128. The sliding layers are bounded by the window at 0.8 GB per sequence regardless of context; the 16 full layers cost **64 KB/token**:

context	KV cache	+ weights
32k	~2.9 GB	~25.6 GB
128k	~9.2 GB	~32.0 GB
256k (max)	~17.6 GB	~40.3 GB

So this is a 48 GB card for the full 262144-token context, and a 32 GB card at up to roughly 64k. A 24 GB card cannot hold it even at short context — the weights alone are 22.75 GB. Context beyond 8192 uses llama3 RoPE scaling (factor: 16.0), configured by the vendor and left untouched here. This is arithmetic from config.json, not a measured deployment.

Reproducing this checkpoint

Built with llm-quantizer:

./llmq.py run --profile exaone-4.5-33b

which re-shards the source — it ships as 2 shards, the larger 49.74 GB, which no consumer GPU can hold — into 18 pieces of ~4 GB, then:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="EXAONE-4.5-33B-resharded",
    save_directory="EXAONE-4.5-33B-W4A16",
    scheme="W4A16",
    ignore=["re:.*visual.*", "re:.*mtp.*", "lm_head", "re:.*embed_tokens.*"],
    device="cuda:0",
)

A job holds one shard at a time, so after re-sharding the build peaks at a few GB of VRAM rather than the model size — the quantization step itself took about a minute on one consumer GPU.

Evaluation

No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Treat the published EXAONE 4.5 numbers as describing the bfloat16 model, not this one.

EXAONE 4.5 is a multilingual model, and this build quantizes every MLP and attention projection without calibration data. Data-free quantization tends to cost most where the weight distribution is least like the majority of the training signal, so if you evaluate anything, evaluate the non-English languages you actually care about rather than assuming English results carry over.

License

EXAONE AI Model License Agreement 1.2 – NC, inherited from the base model — the vendor's LICENSE is included unmodified and applies in full to this derivative. The terms that matter most:

• Research and educational use only (§2.1a). Commercial use of the model, derivatives, or their outputs is prohibited without a separate written license from LG Management Development Institute (§3.1), as is using them to develop or improve competing models.
• Derivatives must keep the "EXAONE" prefix in their name (§2.1c) — which is why this repository is EXAONE-4.5-33B-W4A16.
• Redistribution must carry a copy of the Agreement (§2.1d); it travels in LICENSE.
• Attribution is required when publishing results obtained with the model (§4.3).
• LG claims no rights in model outputs (§4.2), but §3.1 still restricts what you may do with them commercially.

This summary is not legal advice and is not a substitute for reading LICENSE. "EXAONE" is LG's mark; this repository is not endorsed by or affiliated with LG AI Research.