Instructions to use Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2")
model = AutoModelForCausalLM.from_pretrained("Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2

SGLang

How to use Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2 with Docker Model Runner:
```
docker model run hf.co/Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2
```

K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2

English

This is the weight-only NVFP4A16 (W4A16) release of Baekpica/K-EXAONE-152B-A23B-REAP-80E-BF16-v2. The parent has 151,924,654,336 parameters before quantization and approximately 23B parameters active per token. Each MoE layer selects 8 of 80 routed experts.

Repository history. This repository previously hosted an NVFP4 W4A4 checkpoint with calibrated activation global scales. That checkpoint remains available at the tag w4a4-v2 (commit ac55f559d060fcc0a7b0d09c9da1f96c87c59f6f). The current main is a true weight-only W4A16 rebuild from the same BF16 parent: activations stay BF16, and the tensors contain no input_global_scale.

Quantization

Scheme: compressed-tensors NVFP4A16
Algorithm: data-free round-to-nearest with QuantizationModifier; not GPTQ
Format: nvfp4-pack-quantized
Weights: FP4 E2M1, group size 16, FP8 E4M3 block scales
Activations: not quantized; BF16 at runtime
Activation calibration: none required or used
Quantized modules: 10,980 Linear modules in transformer layers 1–45
Preserved in BF16: layer 0, layers 46–47, routers, shared experts, lm_head, and MTP
Tooling: llmcompressor 0.12.0, compressed-tensors 0.17.1

This is a true weight-only checkpoint. Its quantization config has no input activation scheme and its tensors contain no input_global_scale. The explicit re:^mtp\..* exclusion keeps the separately restored MTP modules in BF16.

On A100, vLLM already executed the previous W4A4 files through its Marlin W4A16-compatible fallback, so the degradation observed there was attributable to the 4-bit weights rather than activation FP4. This rebuild makes the weight-only contract explicit in both config and tensors, so native-FP4 hardware such as Blackwell will not enable activation quantization either.

Lineage

236B source → recovered 80E-v2 BF16 → this NVFP4A16 checkpoint

The BF16-v2 parent is a modest post-pruning recovery of the 80-expert REAP model. This checkpoint uses data-free PTQ rather than quantization-aware training. On the 60-chat holdout used throughout this series, the BF16-v2 parent scored CE 1.18012 (perplexity 3.25477) with 0 dead routed experts.

Checkpoint integrity

The strict release audit verifies the exact replacement set against the BF16 parent:

Item	Result
Packed NVFP4 Linear modules	10,980
Weight global scales	10,980 valid / 0 invalid
Input global scales	0
Indexed tensors	33,880
BF16-preserved tensors	940 exact matches
BF16-preserved elements	10,933,129,136 exact matches
Serialized tensor payload	101,174,044,752 bytes
MTP	15 tensors, BF16, explicitly excluded

Machine-readable results are included in checkpoint_audit.json.

Runtime smoke test

The release was loaded with vLLM 0.24.0 using tensor parallel 4 on four A100 80GB GPUs. A100 uses the Marlin W4A16 implementation, so this checks the actual weight-only execution path rather than an activation-FP4 path. The engine selected MarlinNvFp4LinearKernel for dense layers and the MARLIN NvFp4 MoE backend, and model weights took 23.48 GiB per GPU.

Six Korean/English reasoning prompts were sampled with seed 42, temperature=1.0, top_p=0.95, and max_tokens=1024. The model loaded in 79.13 seconds and generated 6,144 tokens in 90.94 seconds. Within the 1,024-token cap none of the six outputs closed </think> or emitted EOT, so this run is an execution check, not a termination or quality verdict; the 3,072-token comparison recorded in the w4a4-v2 tag remains the more informative termination measurement.

MTP compatibility was checked separately with num_speculative_tokens=2. vLLM resolved ExaoneMoeMTP, loaded the restored BF16 draft tensors, shared the target embedding and lm_head, and completed a 64-token speculative generation without an error.

Raw records are included as evaluation_vllm024_w4a16_smoke.json and evaluation_vllm024_w4a16_mtp_smoke.json.

Serving with vLLM

Use a vLLM build with EXAONE-MoE and compressed-tensors NVFP4A16 support:

vllm serve Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2 \
  --tensor-parallel-size 4 \
  --reasoning-parser deepseek_v3

MTP speculative decoding can be enabled separately:

vllm serve Baekpica/K-EXAONE-152B-A23B-REAP-80E-NVFP4-v2 \
  --tensor-parallel-size 4 \
  --reasoning-parser deepseek_v3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

The serialized tensors occupy about 94.2 GiB. Actual serving memory is higher because of KV cache, CUDA graphs, workspaces, and runtime buffers; size the single node for those allocations as well as the weights.

Limitations

Weight-only W4A16 avoids activation FP4 but still quantizes most Linear weights to four bits, so quality can remain below BF16. The W4A4 smoke tests recorded in the w4a4-v2 tag already reflected the A100 W4A16 execution path, so a large quality recovery from this rebuild alone should not be assumed.
Layer 0, layers 46–47, routers, shared experts, lm_head, and MTP remain BF16; other attention and routed-expert Linear weights are NVFP4.
The included generation checks are limited execution smoke tests.
Reasoning termination, Korean, tool use, code, long context, factuality, and safety require workload-specific validation.
Runtime kernel support and memory use depend on the vLLM, CUDA, and GPU versions.

한국어

이 저장소는 Baekpica/K-EXAONE-152B-A23B-REAP-80E-BF16-v2의 weight-only NVFP4A16(W4A16) 체크포인트입니다. 이전에 있던 NVFP4 W4A4 체크포인트는 tag w4a4-v2 (commit ac55f559d060fcc0a7b0d09c9da1f96c87c59f6f)에 보존되어 있습니다.

레이어 1–45의 Linear 10,980개만 FP4 E2M1(group 16, FP8 E4M3 scale)로 변환했습니다. activation은 양자화하지 않고 BF16으로 유지하며, 따라서 activation calibration과 input_global_scale이 없습니다. 첫 레이어, 마지막 두 레이어, router, shared expert, lm_head, MTP는 BF16으로 보존했습니다. 양자화는 GPTQ나 QAT가 아닌 data-free RTN QuantizationModifier로 수행했습니다.

기존 W4A4 파일도 A100에서는 vLLM Marlin의 W4A16 호환 경로로 실행됐으므로, 당시 관측된 품질 저하는 activation FP4보다 4-bit 가중치 자체의 영향으로 보는 것이 타당합니다. 이번 재빌드는 config와 tensor 모두에서 weight-only 계약을 명시하므로 Blackwell 같은 native FP4 환경에서도 activation 양자화가 켜지지 않습니다.

엄격 감사에서 packed module 10,980개, input global scale 0개, 전체 tensor 33,880개, payload 101,174,044,752바이트를 확인했습니다. BF16으로 남긴 tensor 940개와 10,933,129,136개 원소는 parent와 정확히 일치합니다. 상세 결과는 checkpoint_audit.json에 포함됩니다.

직렬화된 tensor는 약 94.2GiB이며 실제 서빙에는 KV cache, CUDA graph, workspace 등의 추가 메모리가 필요합니다. 배포 전 대상 single-node 환경에서 reasoning 종료, 한국어, tool use, 코드, 장문 문맥과 메모리 사용량을 별도로 검증하세요.