Instructions to use Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4")
model = AutoModelForCausalLM.from_pretrained("Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4

SGLang

How to use Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 with Docker Model Runner:
```
docker model run hf.co/Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4
```

K-EXAONE-145B-A23B-REAP-76E-NVFP4

English

This is the weight-only NVFP4A16 (W4A16) release of Baekpica/K-EXAONE-145B-A23B-REAP-76E-BF16. The parent has 144,826,736,896 parameters before quantization, rounded to 145B in the repository name, and approximately 23B parameters active per token. Each MoE layer selects 8 of 76 routed experts.

Quantization

Scheme: compressed-tensors NVFP4A16
Algorithm: data-free round-to-nearest with QuantizationModifier; not GPTQ
Format: nvfp4-pack-quantized
Weights: FP4 E2M1, group size 16, FP8 E4M3 block scales
Activations: not quantized; BF16 at runtime
Activation calibration: none required or used
Quantized modules: 10,440 Linear modules in transformer layers 1–45
Preserved in BF16: layer 0, layers 46–47, routers, shared experts, lm_head, and MTP
Tooling: llmcompressor 0.12.0, compressed-tensors 0.17.1

This is a true weight-only checkpoint. Its quantization config has no input activation scheme and its tensors contain no input_global_scale. The explicit re:^mtp\..* exclusion keeps the separately restored MTP modules in BF16.

Lineage

236B source → recovered 80E BF16 → 76E BF16 → this NVFP4A16 checkpoint

The 76E BF16 parent directly retains the selected tensors from the recovered 80E-v2 model. No additional training was performed after the 80E-to-76E pruning, and this checkpoint uses PTQ rather than quantization-aware training.

Before quantization, the 76E parent was checked on the same 60-chat holdout as the recovered 80E-v2 parent:

Checkpoint	Holdout CE	Perplexity	Dead routed experts
Recovered 80E-v2 BF16	1.18012	3.25477	0
145B-76E BF16	1.18186	3.26042	0

The CE change was +0.00173, or +0.147%. This is a regression check, not a general benchmark.

Checkpoint integrity

The strict release audit verifies the exact replacement set against the BF16 parent:

Item	Result
Packed NVFP4 Linear modules	10,440
Weight global scales	10,440 valid / 0 invalid
Input global scales	0
Indexed tensors	32,236
BF16-preserved tensors	916 exact matches
BF16-preserved elements	10,629,983,988 exact matches
Serialized tensor payload	96,745,692,400 bytes
MTP	15 tensors, BF16, explicitly excluded

Machine-readable results are included in checkpoint_audit.json.

Runtime smoke test

The release was loaded with vLLM 0.24.0 using tensor parallel 4 on four A100 80GB GPUs. A100 uses the Marlin W4A16 implementation, so this checks the actual weight-only execution path rather than an activation-FP4 path.

Six Korean/English reasoning prompts were sampled with seed 42, temperature=1.0, top_p=0.95, and max_tokens=1024. The model loaded in 35.61 seconds and generated 6,144 tokens in 91.57 seconds. One of six outputs closed </think>, none emitted EOT, and all six reached the 1,024-token cap. The short cap makes this useful as an execution check, not as a termination or quality verdict; the BF16 comparison above used a 3,072-token cap.

MTP compatibility was checked separately with num_speculative_tokens=2. vLLM resolved ExaoneMoeMTP, loaded the restored BF16 draft tensors, shared the target embedding and lm_head, and completed a 64-token speculative generation without an error.

This is a deployment smoke test, not a broad quality benchmark. The earlier W4A4 checkpoint also fell back to W4A16 on A100; therefore an A100-only comparison cannot measure the quality difference caused by removing activation FP4 on hardware that would otherwise execute it natively.

Serving with vLLM

Use a vLLM build with EXAONE-MoE and compressed-tensors NVFP4A16 support:

vllm serve Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 \
  --tensor-parallel-size 4 \
  --reasoning-parser deepseek_v3

MTP speculative decoding can be enabled separately:

vllm serve Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 \
  --tensor-parallel-size 4 \
  --reasoning-parser deepseek_v3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

The serialized tensors occupy about 90.1 GiB. Actual serving memory is higher because of KV cache, CUDA graphs, workspaces, and runtime buffers; size the single node for those allocations as well as the weights.

Limitations

Weight-only W4A16 avoids activation FP4 but still quantizes most Linear weights to four bits, so quality can remain below BF16.
Layer 0, layers 46–47, routers, shared experts, lm_head, and MTP remain BF16; other attention and routed-expert Linear weights are NVFP4.
The included holdout and generation checks are limited regression tests.
Reasoning termination, Korean, tool use, code, long context, factuality, and safety require workload-specific validation.
Runtime kernel support and memory use depend on the vLLM, CUDA, and GPU versions.

한국어

이 모델은 Baekpica/K-EXAONE-145B-A23B-REAP-76E-BF16의 weight-only NVFP4A16(W4A16) 체크포인트입니다. 양자화 전 파라미터는 144,826,736,896개이며 저장소 이름에서는 145B로 반올림했습니다. 각 MoE 레이어는 76개 routed expert 중 토큰당 8개를 선택합니다.

레이어 1–45의 Linear 10,440개만 FP4 E2M1(group 16, FP8 E4M3 scale)로 변환했습니다. activation은 양자화하지 않고 BF16으로 유지하며, 따라서 activation calibration과 input_global_scale이 없습니다. 첫 레이어, 마지막 두 레이어, router, shared expert, lm_head, MTP는 BF16으로 보존했습니다. 양자화는 GPTQ나 QAT가 아닌 data-free RTN QuantizationModifier로 수행했습니다.

엄격 감사에서 packed module 10,440개, input global scale 0개, 전체 tensor 32,236개, payload 96,745,692,400바이트를 확인했습니다. BF16으로 남긴 tensor 916개와 10,629,983,988개 원소는 parent와 정확히 일치합니다. 상세 결과는 checkpoint_audit.json에 포함됩니다.

4×A100에서는 vLLM Marlin의 실제 W4A16 경로로 로드·생성을 검증합니다. 다만 기존 W4A4 파일도 A100에서는 W4A16으로 fallback되므로, A100 결과만으로 native activation-FP4 제거 효과를 측정할 수는 없습니다. W4A16도 대부분의 가중치를 4-bit로 줄이므로 BF16 대비 성능 감소가 남을 수 있습니다.

직렬화된 tensor는 약 90.1GiB이며 실제 서빙에는 KV cache, CUDA graph, workspace 등의 추가 메모리가 필요합니다. 배포 전 대상 single-node 환경에서 reasoning 종료, 한국어, tool use, 코드, 장문 문맥과 메모리 사용량을 별도로 검증하세요.