---
language:
- ja
- en
- zh
license: apache-2.0
library_name: optimum-quanto
base_model: openbmb/MiniCPM5-1B
tags:
- quantized
- fp8
- float8
- e4m3fn
- optimum-quanto
- minicpm
- rtx-4070-ti
- edge-ai
- text-generation
- conversational
pipeline_tag: text-generation
---

# MiniCPM5-1B-Quanto-FP8

This repository provides **`openbmb/MiniCPM5-1B` quantized to FP8 (Float8 `e4m3fn`)** using `optimum-quanto`.

It is optimized for 4th Generation Tensor Cores (NVIDIA Ada Lovelace RTX 40xx series, Hopper H100, Blackwell B200) to deliver **ultra-low time-to-first-token (TTFT 76ms)**, **1.78x higher decode throughput (28.3 tok/s)**, and a **compact 1.4GB VRAM footprint** without any loss in reasoning, algebra, or anti-hallucination accuracy compared to native `bfloat16`.

- **Base Model**: [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) (1.16B parameters, 128k context, LlamaForCausalLM)
- **Quantization Method**: `optimum-quanto` FP8 (`weights=qfloat8_e4m3fn`, `activations=None`)
- **Model Weight Size**: 1.22 GB (`model.safetensors`)
- **Benchmark Evaluation Dataset**: [`aoiandroid/minicpm5-1b-quantization-benchmark`](https://huggingface.co/datasets/aoiandroid/minicpm5-1b-quantization-benchmark)

---

## Performance Benchmarks (NVIDIA RTX 4070 Ti 12GB)

Measured across 8 authentic conversational, logical, mathematical, and structural scenarios:

| Metric | bfloat16 (Native) | BitsAndBytes 4bit (NF4) | **Quanto FP8 (This Model)** | Quanto INT4 |
| :--- | :--- | :--- | :--- | :--- |
| **Model Weight VRAM** | 2071.1 MB (2.02 GB) | 1108.3 MB (1.08 GB) | **1425.0 MB (1.39 GB)** | 1128.8 MB (1.10 GB) |
| **Peak VRAM During Inference** | 2109.4 MB | 1148.9 MB | **1478.7 MB** | 1166.3 MB |
| **Time-To-First-Token (TTFT)** | 135.2 ms | 193.3 ms | **76.4 ms (43.6% faster)** | 67.0 ms |
| **Decode Speed (TPS)** | 15.9 tok/s | 11.5 tok/s | **28.3 tok/s (1.78x faster)** | 32.5 tok/s |
| **Overall Speed (TPS)** | 15.8 tok/s | 11.4 tok/s | **28.1 tok/s** | 32.3 tok/s |
| **Reasoning Accuracy (Math/Logic)** | 100% (Full accuracy) | Degraded | **100% (Zero loss vs BF16)** | Severely degraded |
| **Anti-Hallucination Rejection** | Perfect | Unstable | **Perfect** | Infinite loop |

### Why FP8 Outperforms BitsAndBytes 4bit:
1. **Hardware Tensor Core Execution**: On Ada Lovelace GPUs, FP8 GEMM is executed directly on 4th-gen Tensor Cores.
2. **Elimination of Dequantization Overhead**: Traditional 4-bit (BitsAndBytes NF4) suffers from dynamic dequantization kernel overhead before every GEMM operation, causing an inverted slowdown on small (1B) models.
3. **Zero Accuracy Loss**: Float8 `e4m3fn` preserves dynamic range across attention projections, preventing the logic collapse observed in INT4.

---

## Quickstart Inference Code

### Installation
```bash
pip install torch transformers accelerate optimum-quanto
```

### Direct Loading with `optimum.quanto`
```python
import torch
from optimum.quanto import QuantizedModelForCausalLM
from transformers import AutoTokenizer

model_id = "aoiandroid/MiniCPM5-1B-Quanto-FP8"

# 1. Load model and move to CUDA
model = QuantizedModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.to("cuda")

# 2. Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# 3. Format prompt using chat template
prompt = "こんにちは！自己紹介をしてください。"
messages = [{"role": "user", "content": prompt}]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
inputs.pop("token_type_ids", None)

# 4. Generate with high-speed FP8 Tensor Cores
with torch.no_grad():
    output_tokens = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.8
    )

response = tokenizer.decode(output_tokens[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
```

---

## Evaluation Data and Reproducibility

For full raw benchmark output logs (JSON) and evaluation scripts across all 8 test cases, visit the evaluation dataset repository:
- [`aoiandroid/minicpm5-1b-quantization-benchmark`](https://huggingface.co/datasets/aoiandroid/minicpm5-1b-quantization-benchmark)