Instructions to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

SGLang

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with Ollama:
```
ollama run hf.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL
```
Unsloth Desktop

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.ERNIE-4.5-21B-A3B-Thinking-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

ERNIE-4.5-21B-A3B-Thinking-GGUF / README.md

danielhanchen

Update README.md

4c24188 verified about 1 year ago

preview code

Raw

History Blame

7.26 kB

	---
	license: apache-2.0
	language:
	- en
	- zh
	pipeline_tag: text-generation
	tags:
	- ERNIE4.5
	library_name: transformers
	base_model: baidu/ERNIE-4.5-21B-A3B-Thinking
	---

	<div align="center" style="line-height: 1;">
	<a href="https://ernie.baidu.com/" target="_blank" style="margin: 2px;">
	<img alt="Chat" src="https://img.shields.io/badge/🤖_Chat-ERNIE_Bot-blue" style="display: inline-block; vertical-align: middle;"/>
	</a>
	<a href="https://huggingface.co/baidu" target="_blank" style="margin: 2px;">
	<img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Baidu-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
	</a>
	<a href="https://github.com/PaddlePaddle/ERNIE" target="_blank" style="margin: 2px;">
	<img alt="Github" src="https://img.shields.io/badge/GitHub-ERNIE-000?logo=github&color=0000FF" style="display: inline-block; vertical-align: middle;"/>
	</a>
	<a href="https://ernie.baidu.com/blog/ernie4.5" target="_blank" style="margin: 2px;">
	<img alt="Blog" src="https://img.shields.io/badge/🖖_Blog-ERNIE4.5-A020A0" style="display: inline-block; vertical-align: middle;"/>
	</a>
	<a href="https://discord.gg/JPmZXDsEEK" target="_blank" style="margin: 2px;">
	<img alt="Discord" src="https://img.shields.io/badge/Discord-ERNIE-5865F2?logo=discord&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
	</a>
	<a href="https://x.com/PaddlePaddle" target="_blank" style="margin: 2px;">
	<img alt="X" src="https://img.shields.io/badge/X-PaddlePaddle-6080F0"?logo=x&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
	</a>
	</div>

	<div align="center" style="line-height: 1;">
	<a href="#license" style="margin: 2px;">
	<img alt="License" src="https://img.shields.io/badge/License-Apache2.0-A5de54" style="display: inline-block; vertical-align: middle;"/>
	</a>
	</div>

	# ERNIE-4.5-21B-A3B-Thinking

	## Model Highlights

	Over the past three months, we have continued to scale the thinking capability of ERNIE-4.5-21B-A3B, improving both the quality and depth of reasoning, thereby advancing the competitiveness of ERNIE lightweight models in complex reasoning tasks. We are pleased to introduce ERNIE-4.5-21B-A3B-Thinking, featuring the following key enhancements:

	* Significantly improved performance on reasoning tasks, including logical reasoning, mathematics, science, coding, text generation, and academic benchmarks that typically require human expertise.
	* Efficient tool usage capabilities.
	* Enhanced 128K long-context understanding capabilities.

	> [!NOTE]
	> Note: This version has an increased thinking length. We strongly recommend its use in highly complex reasoning tasks.

	![benchmark](./benchmark.png)

	## Model Overview

	ERNIE-4.5-21B-A3B-Thinking is a text MoE post-trained model, with 21B total parameters and 3B activated parameters for each token. The following are the model configuration details:

	\|Key\|Value\|
	\|-\|-\|
	\|Modality\|Text\|
	\|Training Stage\|Posttraining\|
	\|Params(Total / Activated)\|21B / 3B\|
	\|Layers\|28\|
	\|Heads(Q/KV)\|20 / 4\|
	\|Text Experts(Total / Activated)\|64 / 6\|
	\|Vision Experts(Total / Activated)\|64 / 6\|
	\|Shared Experts\|2\|
	\|Context Length\|131072\|

	## Quickstart

	> [!NOTE]
	> To align with the wider community, this model releases Transformer-style weights. Both PyTorch and PaddlePaddle ecosystem tools, such as vLLM, transformers, and FastDeploy, are expected to be able to load and run this model.

	### FastDeploy Inference

	Quickly deploy services using FastDeploy as shown below. For more detailed usage, refer to the [FastDeploy GitHub Repository](https://github.com/PaddlePaddle/FastDeploy).

	Note: 80GB x 1 GPU resources are required. Deploying this model requires FastDeploy version 2.2.

	```bash
	python -m fastdeploy.entrypoints.openai.api_server \
	--model baidu/ERNIE-4.5-21B-A3B-Thinking \
	--port 8180 \
	--metrics-port 8181 \
	--engine-worker-queue-port 8182 \
	--load_choices "default_v1" \
	--tensor-parallel-size 1 \
	--max-model-len 131072 \
	--reasoning-parser ernie_x1 \
	--tool-call-parser ernie_x1 \
	--max-num-seqs 32
	```

	The ERNIE-4.5-21B-A3B-Thinking model supports function call.

	```bash
	curl -X POST "http://0.0.0.0:8180/v1/chat/completions" \
	-H "Content-Type: application/json" \
	-d $'{
	"messages": [
	{
	"role": "user",
	"content": "How \'s the weather in Beijing today?"
	}
	],
	"tools": [
	{
	"type": "function",
	"function": {
	"name": "get_weather",
	"description": "Determine weather in my location",
	"parameters": {
	"type": "object",
	"properties": {
	"location": {
	"type": "string",
	"description": "The city and state e.g. San Francisco, CA"
	},
	"unit": {
	"type": "string",
	"enum": [
	"c",
	"f"
	]
	}
	},
	"additionalProperties": false,
	"required": [
	"location",
	"unit"
	]
	},
	"strict": true
	}
	}]
	}'
	```

	### vLLM inference

	```bash
	vllm serve baidu/ERNIE-4.5-21B-A3B-Thinking
	```

	The `reasoning-parser` and `tool-call-parser` for vLLM Ernie are currently under development.

	### Using `transformers` library

	Note: You'll need the`transformers`library (version 4.54.0 or newer) installed to use this model.

	The following contains a code snippet illustrating how to use the model generate content based on given inputs.

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model_name = "baidu/ERNIE-4.5-21B-A3B-Thinking"

	# load the tokenizer and the model
	tokenizer = AutoTokenizer.from_pretrained(model_name)
	model = AutoModelForCausalLM.from_pretrained(
	model_name,
	device_map="auto",
	torch_dtype=torch.bfloat16,
	)

	# prepare the model input
	prompt = "Give me a short introduction to large language model."
	messages = [
	{"role": "user", "content": prompt}
	]
	text = tokenizer.apply_chat_template(
	messages,
	tokenize=False,
	add_generation_prompt=True
	)
	model_inputs = tokenizer([text], add_special_tokens=False, return_tensors="pt").to(model.device)

	# conduct text completion
	generated_ids = model.generate(
	**model_inputs,
	max_new_tokens=1024
	)
	output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()

	# decode the generated ids
	generate_text = tokenizer.decode(output_ids, skip_special_tokens=True)
	print("generate_text:", generate_text)
	```

	## License

	The ERNIE 4.5 models are provided under the Apache License 2.0. This license permits commercial use, subject to its terms and conditions. Copyright (c) 2025 Baidu, Inc. All Rights Reserved.

	## Citation

	If you find ERNIE 4.5 useful or wish to use it in your projects, please kindly cite our technical report:

	```text
	@misc{ernie2025technicalreport,
	title={ERNIE 4.5 Technical Report},
	author={Baidu-ERNIE-Team},
	year={2025},
	primaryClass={cs.CL},
	howpublished={\url{https://ernie.baidu.com/blog/publication/ERNIE_Technical_Report.pdf}}
	}
	```