---
license: other
license_name: k-exaone
license_link: LICENSE
language:
- en
- ko
base_model:
- Baekpica/K-EXAONE-145B-A23B-REAP-76E-BF16
pipeline_tag: text-generation
library_name: transformers
tags:
- lg-ai
- exaone
- k-exaone
- moe
- reap
- nvfp4
- nvfp4a16
- w4a16
- compressed-tensors
---

# K-EXAONE-145B-A23B-REAP-76E-NVFP4

## English

This is the weight-only **NVFP4A16 (W4A16)** release of
[Baekpica/K-EXAONE-145B-A23B-REAP-76E-BF16](https://huggingface.co/Baekpica/K-EXAONE-145B-A23B-REAP-76E-BF16).
The parent has 144,826,736,896 parameters before quantization, rounded to 145B
in the repository name, and approximately 23B parameters active per token.
Each MoE layer selects 8 of 76 routed experts.

### Quantization

- Scheme: compressed-tensors `NVFP4A16`
- Algorithm: data-free round-to-nearest with `QuantizationModifier`; not GPTQ
- Format: `nvfp4-pack-quantized`
- Weights: FP4 E2M1, group size 16, FP8 E4M3 block scales
- Activations: **not quantized**; BF16 at runtime
- Activation calibration: none required or used
- Quantized modules: 10,440 Linear modules in transformer layers 1–45
- Preserved in BF16: layer 0, layers 46–47, routers, shared experts,
  `lm_head`, and MTP
- Tooling: `llmcompressor 0.12.0`, `compressed-tensors 0.17.1`

This is a true weight-only checkpoint. Its quantization config has no input
activation scheme and its tensors contain no `input_global_scale`. The explicit
`re:^mtp\..*` exclusion keeps the separately restored MTP modules in BF16.

### Lineage

`236B source → recovered 80E BF16 → 76E BF16 → this NVFP4A16 checkpoint`

The 76E BF16 parent directly retains the selected tensors from the recovered
80E-v2 model. No additional training was performed after the 80E-to-76E
pruning, and this checkpoint uses PTQ rather than quantization-aware training.

Before quantization, the 76E parent was checked on the same 60-chat holdout as
the recovered 80E-v2 parent:

| Checkpoint | Holdout CE | Perplexity | Dead routed experts |
|---|---:|---:|---:|
| Recovered 80E-v2 BF16 | 1.18012 | 3.25477 | 0 |
| 145B-76E BF16 | 1.18186 | 3.26042 | 0 |

The CE change was +0.00173, or +0.147%. This is a regression check, not a
general benchmark.

### Checkpoint integrity

The strict release audit verifies the exact replacement set against the BF16
parent:

| Item | Result |
|---|---:|
| Packed NVFP4 Linear modules | 10,440 |
| Weight global scales | 10,440 valid / 0 invalid |
| Input global scales | 0 |
| Indexed tensors | 32,236 |
| BF16-preserved tensors | 916 exact matches |
| BF16-preserved elements | 10,629,983,988 exact matches |
| Serialized tensor payload | 96,745,692,400 bytes |
| MTP | 15 tensors, BF16, explicitly excluded |

Machine-readable results are included in `checkpoint_audit.json`.

### Runtime smoke test

The release was loaded with vLLM 0.24.0 using tensor parallel 4 on four A100
80GB GPUs. A100 uses the Marlin W4A16 implementation, so this checks the actual
weight-only execution path rather than an activation-FP4 path.

Six Korean/English reasoning prompts were sampled with seed 42,
`temperature=1.0`, `top_p=0.95`, and `max_tokens=1024`. The model loaded in
35.61 seconds and generated 6,144 tokens in 91.57 seconds. One of six outputs
closed `</think>`, none emitted EOT, and all six reached the 1,024-token cap.
The short cap makes this useful as an execution check, not as a termination or
quality verdict; the BF16 comparison above used a 3,072-token cap.

MTP compatibility was checked separately with `num_speculative_tokens=2`.
vLLM resolved `ExaoneMoeMTP`, loaded the restored BF16 draft tensors, shared
the target embedding and `lm_head`, and completed a 64-token speculative
generation without an error.

This is a deployment smoke test, not a broad quality benchmark. The earlier
W4A4 checkpoint also fell back to W4A16 on A100; therefore an A100-only
comparison cannot measure the quality difference caused by removing activation
FP4 on hardware that would otherwise execute it natively.

### Serving with vLLM

Use a vLLM build with EXAONE-MoE and compressed-tensors NVFP4A16 support:

```bash
vllm serve Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 \
  --tensor-parallel-size 4 \
  --reasoning-parser deepseek_v3
```

MTP speculative decoding can be enabled separately:

```bash
vllm serve Baekpica/K-EXAONE-145B-A23B-REAP-76E-NVFP4 \
  --tensor-parallel-size 4 \
  --reasoning-parser deepseek_v3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
```

The serialized tensors occupy about 90.1 GiB. Actual serving memory is higher
because of KV cache, CUDA graphs, workspaces, and runtime buffers; size the
single node for those allocations as well as the weights.

### Limitations

- Weight-only W4A16 avoids activation FP4 but still quantizes most Linear
  weights to four bits, so quality can remain below BF16.
- Layer 0, layers 46–47, routers, shared experts, `lm_head`, and MTP remain
  BF16; other attention and routed-expert Linear weights are NVFP4.
- The included holdout and generation checks are limited regression tests.
- Reasoning termination, Korean, tool use, code, long context, factuality, and
  safety require workload-specific validation.
- Runtime kernel support and memory use depend on the vLLM, CUDA, and GPU
  versions.

## 한국어

이 모델은
[Baekpica/K-EXAONE-145B-A23B-REAP-76E-BF16](https://huggingface.co/Baekpica/K-EXAONE-145B-A23B-REAP-76E-BF16)의
**weight-only NVFP4A16(W4A16)** 체크포인트입니다. 양자화 전 파라미터는
144,826,736,896개이며 저장소 이름에서는 145B로 반올림했습니다. 각 MoE
레이어는 76개 routed expert 중 토큰당 8개를 선택합니다.

레이어 1–45의 Linear 10,440개만 FP4 E2M1(group 16, FP8 E4M3 scale)로
변환했습니다. activation은 양자화하지 않고 BF16으로 유지하며, 따라서
activation calibration과 `input_global_scale`이 없습니다. 첫 레이어, 마지막
두 레이어, router, shared expert, `lm_head`, MTP는 BF16으로 보존했습니다.
양자화는 GPTQ나 QAT가 아닌 data-free RTN `QuantizationModifier`로
수행했습니다.

엄격 감사에서 packed module 10,440개, input global scale 0개, 전체 tensor
32,236개, payload 96,745,692,400바이트를 확인했습니다. BF16으로 남긴 tensor
916개와 10,629,983,988개 원소는 parent와 정확히 일치합니다. 상세 결과는
`checkpoint_audit.json`에 포함됩니다.

4×A100에서는 vLLM Marlin의 실제 W4A16 경로로 로드·생성을 검증합니다.
다만 기존 W4A4 파일도 A100에서는 W4A16으로 fallback되므로, A100 결과만으로
native activation-FP4 제거 효과를 측정할 수는 없습니다. W4A16도 대부분의
가중치를 4-bit로 줄이므로 BF16 대비 성능 감소가 남을 수 있습니다.

직렬화된 tensor는 약 90.1GiB이며 실제 서빙에는 KV cache, CUDA graph,
workspace 등의 추가 메모리가 필요합니다. 배포 전 대상 single-node 환경에서
reasoning 종료, 한국어, tool use, 코드, 장문 문맥과 메모리 사용량을 별도로
검증하세요.

## License

This model is distributed under the same **K-EXAONE AI Model License** as the
original model. See `LICENSE` for the complete terms.