Instructions to use AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K

SGLang

How to use AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
```

MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K

This version of MiniCPM5-2B-GPTQ-Int4 has been converted to run on the AX650 NPU using w4a16 quantization.

Compatible with Pulsar2 version: 5.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo :

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU HOST LLM Runtime

Support Platform

AX650
- AX650N DEMO Board
- M4N-Dock(爱芯派Pro)
- M.2 Accelerator card

Chips	ttft(128 tokens)	prefill	decode	CMM	Flash
AX650	196 ms	653 tokens/s	12 token/s	3.3GiB	3.8GiB

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

安装 axllm

方式一：克隆仓库后执行安装脚本：

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

方式二：一行命令安装（默认分支 axllm）：

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：

如果没有编译环境，请到： https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm 下载 最新 CI 导出的可执行程序（axllm），然后：

chmod +x axllm
sudo mv axllm /usr/bin/axllm

模型下载（Hugging Face）

mkdir -p AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
cd AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
hf download AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K --local-dir .

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

运行（CLI）

root@ax650:~# axllm run AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K/
11:12:03.898 INF Init:475 | LLM init start
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 17% | #####                            |   8 /  45 [6.94s<39.03s, 1.15 count/s] init 6 axmodel ok,remain_cmm(9453 MB)^C
 20% | ######                           |   9 /  45 [7.87s<39.34s, 1.14 count/s] init 7 axmodel ok,remain_cmm(9380 MB)^C
 97% | ###############################  |  44 /  45 [47.96s<49.05s, 0.92 count/s] init post axmodel ok,remain_cmm(6623 MB)
11:12:51.868 INF Init:798 | max_token_len : 2047
11:12:51.868 INF Init:801 | kv_cache_size : 256, kv_cache_num: 2047
11:12:51.868 INF init_groups_from_model:150 | prefill_token_num : 128
11:12:51.868 INF init_groups_from_model:398 | decode grp: 0, gid: 0, max_token_len : 2047
11:12:51.869 INF init_groups_from_model:398 | decode grp: 1, gid: 1, max_token_len : 4095
11:12:51.869 INF init_groups_from_model:398 | decode grp: 2, gid: 2, max_token_len : 6143
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 0, gid: 3, history_cap: 0, total_cap: 128, symbolic_cap: 1
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 1, gid: 4, history_cap: 128, total_cap: 256, symbolic_cap: 128
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 2, gid: 5, history_cap: 256, total_cap: 384, symbolic_cap: 256
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 3, gid: 6, history_cap: 384, total_cap: 512, symbolic_cap: 384
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 4, gid: 7, history_cap: 512, total_cap: 640, symbolic_cap: 512
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 5, gid: 8, history_cap: 640, total_cap: 768, symbolic_cap: 640
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 6, gid: 9, history_cap: 768, total_cap: 896, symbolic_cap: 768
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 7, gid: 10, history_cap: 896, total_cap: 1024, symbolic_cap: 896
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 8, gid: 11, history_cap: 1024, total_cap: 1152, symbolic_cap: 1024
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 9, gid: 12, history_cap: 1152, total_cap: 1280, symbolic_cap: 1152
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 10, gid: 13, history_cap: 1280, total_cap: 1408, symbolic_cap: 1280
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 11, gid: 14, history_cap: 1408, total_cap: 1536, symbolic_cap: 1408
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 12, gid: 15, history_cap: 1536, total_cap: 1664, symbolic_cap: 1536
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 13, gid: 16, history_cap: 1664, total_cap: 1792, symbolic_cap: 1664
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 14, gid: 17, history_cap: 1792, total_cap: 1920, symbolic_cap: 1792
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 15, gid: 18, history_cap: 1920, total_cap: 2048, symbolic_cap: 1920
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 16, gid: 19, history_cap: 2048, total_cap: 2176, symbolic_cap: 2048
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 17, gid: 20, history_cap: 2176, total_cap: 2304, symbolic_cap: 2176
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 18, gid: 21, history_cap: 2304, total_cap: 2432, symbolic_cap: 2304
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 19, gid: 22, history_cap: 2432, total_cap: 2560, symbolic_cap: 2432
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 20, gid: 23, history_cap: 2560, total_cap: 2688, symbolic_cap: 2560
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 21, gid: 24, history_cap: 2688, total_cap: 2816, symbolic_cap: 2688
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 22, gid: 25, history_cap: 2816, total_cap: 2944, symbolic_cap: 2816
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 23, gid: 26, history_cap: 2944, total_cap: 3072, symbolic_cap: 2944
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 24, gid: 27, history_cap: 3072, total_cap: 3200, symbolic_cap: 3072
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 25, gid: 28, history_cap: 3200, total_cap: 3328, symbolic_cap: 3200
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 26, gid: 29, history_cap: 3328, total_cap: 3456, symbolic_cap: 3328
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 27, gid: 30, history_cap: 3456, total_cap: 3584, symbolic_cap: 3456
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 28, gid: 31, history_cap: 3584, total_cap: 3712, symbolic_cap: 3584
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 29, gid: 32, history_cap: 3712, total_cap: 3840, symbolic_cap: 3712
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 30, gid: 33, history_cap: 3840, total_cap: 3968, symbolic_cap: 3840
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 31, gid: 34, history_cap: 3968, total_cap: 4096, symbolic_cap: 3968
11:12:51.869 INF init_groups_from_model:409 | prefill_max_token_num : 4096
11:12:51.870 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  45 /  45 [47.97s<47.97s, 0.94 count/s] embed_selector init ok
11:12:51.883 INF load_config:554 | load config: 
11:12:51.883 INF load_config:554 | {
11:12:51.883 INF load_config:554 |     "enable_repetition_penalty": false,
11:12:51.883 INF load_config:554 |     "enable_temperature": false,
11:12:51.883 INF load_config:554 |     "enable_top_k_sampling": false,
11:12:51.883 INF load_config:554 |     "enable_top_p_sampling": false,
11:12:51.883 INF load_config:554 |     "penalty_window": 128,
11:12:51.883 INF load_config:554 |     "repetition_penalty": 1.1,
11:12:51.883 INF load_config:554 |     "temperature": 0.7,
11:12:51.883 INF load_config:554 |     "top_k": 20,
11:12:51.883 INF load_config:554 |     "top_p": 0.8
11:12:51.883 INF load_config:554 | }
11:12:51.883 INF Init:913 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
----------------------------------------
prompt >> 
prompt >> 请计算 12345 * 6789，并给出计算过程。
11:14:04.261 INF SetKVCache:2027 | decode_grpid:0 prefill_grpid:3 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:33 prefer_symbolic_group:0
11:14:04.261 INF SetKVCache:2049 | current prefill_max_token_num:4096
11:14:04.514 INF SetKVCache:2065 | first run
11:14:04.578 INF Run:257 | input token num : 33, prefill_split_num : 1
11:14:04.578 INF Run:342 | prefill chunk p=0 history_len=0 grpid=3 kv_cache_num=0 input_tokens=33
11:14:04.777 INF Run:671 | ttft: 199.68 ms
<think>
我们被要求计算12345乘以6789，并给出计算过程。我们需要展示计算步骤。
</think>

我们将计算 12345 × 6789。

**计算过程：**

我们可以使用乘法分配律来简化计算，或者直接进行乘法运算。

**方法一：直接乘法**

12345 × 6789

我们可以将6789分解为6000 + 700 + 80 + 9，然后分别计算：

12345 × 6000 = 12345 × 6 × 1000 = 74070 × 1000 = 74,070,000
12345 × 700 = 12345 × 7 × 100 = 86,415 × 100 = 8,641,500
12345 × 80 = 12345 × 8 × 10 = 98,760 × 10 = 987,600
12345 × 9 = 111,105

现在将所有结果相加：

74,070,000 + 8,641,500 = 82,711,500
82,711,500 + 987,600 = 83,699,100
83,699,100 + 111,105 = 83,810,205

所以，12345 × 6789 = 83,810,205。

**方法二：使用竖式乘法**

我们可以按照竖式计算：

   12345
×  6789
---------

逐步计算：

首先，12345 × 9 = 111,105
然后，12345 × 80 = 987,600
然后，12345 × 700 = 8,641,500
然后，12345 × 6000 = 74,070,000

将所有部分相加：

111,105 + 987,600 = 1,098,705
1,098,705 + 8,641,500 = 9,740,205
9,740,205 + 74,070,000 = 83,810,205

因此，结果相同。

**最终答案：**
12345 × 6789 = 83,810,205。

11:14:43.674 NTC Run:1104 | hit eos,decode avg 11.85 token/s
11:14:43.699 INF GetKVCache:2004 | precompute_len:495, remaining:3601 (tracked)

启动服务（OpenAI 兼容）

root@ax650:~# axllm serve AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
[I][                            Init][ 138]: LLM init start
tokenizer_type = 1
 96% | ███████████████████████████████   |  30 /  31 [4.63s<4.79s, 6.47 count/s] init post axmodel ok,remain_cmm(9563 MB)
[I][                            Init][ 199]: max_token_len : 2047
[I][                            Init][ 202]: kv_cache_size : 1024, kv_cache_num: 2047
[I][                            Init][ 205]: prefill_token_num : 128
[I][                            Init][ 209]: grp: 1, prefill_max_kv_cache_num : 1
[I][                            Init][ 209]: grp: 2, prefill_max_kv_cache_num : 128
[I][                            Init][ 209]: grp: 3, prefill_max_kv_cache_num : 256
[I][                            Init][ 209]: grp: 4, prefill_max_kv_cache_num : 384
[I][                            Init][ 209]: grp: 5, prefill_max_kv_cache_num : 512
[I][                            Init][ 209]: grp: 6, prefill_max_kv_cache_num : 640
[I][                            Init][ 209]: grp: 7, prefill_max_kv_cache_num : 768
[I][                            Init][ 209]: grp: 8, prefill_max_kv_cache_num : 896
[I][                            Init][ 209]: grp: 9, prefill_max_kv_cache_num : 1024
[I][                            Init][ 209]: grp: 10, prefill_max_kv_cache_num : 1152
[I][                            Init][ 214]: prefill_max_token_num : 1152
[I][                            Init][  27]: LLaMaEmbedSelector use mmap
100% | ████████████████████████████████ |  31 /  31 [4.64s<4.64s, 6.69 count/s] embed_selector init ok
[W][                            Init][ 457]: Qwen-VL vision size override: cfg=448x448 bytes=1204224, model_input_bytes=884736 -> 384x384 (square).
[I][                            Init][ 641]: Qwen-VL token ids: vision_start=151652 image_pad=151655 video_pad=151656
[I][                            Init][ 666]: VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
[I][                            Init][ 672]: VisionModule deepstack enabled: layers=3
[I][                     load_config][ 282]: load config:
{
    "enable_repetition_penalty": false,
    "enable_temperature": false,
    "enable_top_k_sampling": false,
    "enable_top_p_sampling": false,
    "penalty_window": 20,
    "repetition_penalty": 1.2,
    "temperature": 0.9,
    "top_k": 10,
    "top_p": 0.8
}

[I][                            Init][ 272]: LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K'...
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K

OpenAI 调用示例

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
completion = client.chat.completions.create(
    model=MODEL,
    messages=messages,
)

print(completion.choices[0].message.content)

OpenAI 流式调用示例

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
stream = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    stream=True,
)

print("assistant:")
for ev in stream:
    delta = getattr(ev.choices[0], "delta", None)
    if delta and getattr(delta, "content", None):
        print(delta.content, end="", flush=True)
print("
")

Downloads last month: 28

Model tree for AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K

Base model

openbmb/MiniCPM5-2B

Finetuned

(40)

this model

Collection including AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K

MiniCPM5

Collection

3 items • Updated 2 days ago