Instructions to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Use Docker

docker model run hf.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

SGLang

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with Ollama:
```
ollama run hf.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M
```
Unsloth Desktop

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with Docker Model Runner:
```
docker model run hf.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M
```

Lemonade

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen2.5-1.5B-Instruct-BE-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen2.5-1.5B-Instruct-BE-GGUF

File size: 9,116 Bytes

---
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/blob/main/LICENSE
language:
- en
pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-1.5B-Instruct
base_model_relation: quantized
tags:
- chat
- mainframe
- s390x
- z15
- z16
- z17
- big-endian
library_name: transformers
quantized_by: taronaeo
---

# Qwen2.5-1.5B Instruct Big-Endian - GGUF (Verified for IBM Z & LinuxONE Mainframes)
- Model Creator: [Alibaba Cloud Qwen](https://huggingface.co/Qwen)
- Original Model: [Qwen2.5-1.5B-Instruct](https://huggingface.co/qwen/Qwen2.5-1.5B-Instruct)

### Description
This repository contains GGUF format model for [Qwen2.5-1.5B-Instruct](https://huggingface.co/qwen/Qwen2.5-1.5B-Instruct), compiled using Big-Endian. Every model has been verified to work on IBM z16 Mainframe.

### Provided Files
| Name                                                                                                                                                 | Quant Method | Bits | Size | Use Case                                                               |
|------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|------|------|------------------------------------------------------------------------|
| [qwen2.5-1.5b-instruct-be.Q2_K.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q2_K.gguf)     | Q2_K         | 2    | 645M | smallest, significant quality loss - not recommended for most purposes |
| [qwen2.5-1.5b-instruct-be.Q3_K_S.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q3_K_S.gguf) | Q3_K_S       | 3    | 726M | very small, high quality loss                                          |
| [qwen2.5-1.5b-instruct-be.Q3_K_M.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q3_K_M.gguf) | Q3_K_M       | 3    | 786M | very small, high quality loss                                          |
| [qwen2.5-1.5b-instruct-be.Q3_K_L.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q3_K_L.gguf) | Q3_K_L       | 3    | 840M | small, substantial quality loss                                        |
| [qwen2.5-1.5b-instruct-be.Q4_0.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q4_0.gguf)     | Q4_0         | 4    | 892M | legacy; small, very high quality loss - prefer using Q3_K_M            |
| [qwen2.5-1.5b-instruct-be.Q4_K_S.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q4_K_S.gguf) | Q4_K_S       | 4    | 897M | small, greater quality loss                                            |
| [qwen2.5-1.5b-instruct-be.Q4_K_M.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q4_K_M.gguf) | Q4_K_M       | 4    | 941M | medium, balanced quality - recommended                                 |
| [qwen2.5-1.5b-instruct-be.Q5_0.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q5_0.gguf)     | Q5_0         | 5    | 1.1G | legacy; medium, balanced quality - prefer using Q4_K_M                 |
| [qwen2.5-1.5b-instruct-be.Q5_K_S.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q5_K_S.gguf) | Q5_K_S       | 5    | 1.1G | large, low quality loss - recommended                                  |
| [qwen2.5-1.5b-instruct-be.Q5_K_M.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q5_K_M.gguf) | Q5_K_M       | 5    | 1.1G | large, very low quality loss - recommended                             |
| [qwen2.5-1.5b-instruct-be.Q6_K.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q6_K.gguf)     | Q6_K         | 6    | 1.2G | very large, extremely low quality loss                                 |
| [qwen2.5-1.5b-instruct-be.Q8_0.gguf](https://huggingface.co/taronaeo/Qwen2.5-1.5B-Instruct-BE-GGUF/blob/main/qwen2.5-1.5b-instruct-be.Q8_0.gguf)     | Q8_0         | 8    | 1.6G | very large, extremely low quality loss - not recommended               |

# Original Model Card: Qwen2.5-1.5B-Instruct

## Introduction

Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:

- Significantly **more knowledge** and has greatly improved capabilities in **coding** and **mathematics**, thanks to our specialized expert models in these domains.
- Significant improvements in **instruction following**, **generating long texts** (over 8K tokens), **understanding structured data** (e.g, tables), and **generating structured outputs** especially JSON. **More resilient to the diversity of system prompts**, enhancing role-play implementation and condition-setting for chatbots.
- **Long-context Support** up to 128K tokens and can generate up to 8K tokens.
- **Multilingual support** for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. 

**This repo contains the instruction-tuned 1.5B Qwen2.5 model**, which has the following features:
- Type: Causal Language Models
- Training Stage: Pretraining & Post-training
- Architecture: transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias and tied word embeddings
- Number of Parameters: 1.54B
- Number of Paramaters (Non-Embedding): 1.31B
- Number of Layers: 28
- Number of Attention Heads (GQA): 12 for Q and 2 for KV
- Context Length: Full 32,768 tokens and generation 8192 tokens

For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).

## Requirements

The code of Qwen2.5 has been in the latest Hugging face `transformers` and we advise you to use the latest version of `transformers`.

With `transformers<4.37.0`, you will encounter the following error:
```
KeyError: 'qwen2'
```

## Quickstart

Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen2.5-1.5B-Instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```


## Evaluation & Performance

Detailed evaluation results are reported in this [📑 blog](https://qwenlm.github.io/blog/qwen2.5/).

For requirements on GPU memory and the respective throughput, see results [here](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html).

## Citation

If you find our work helpful, feel free to give us a cite.

```
@misc{qwen2.5,
    title = {Qwen2.5: A Party of Foundation Models},
    url = {https://qwenlm.github.io/blog/qwen2.5/},
    author = {Qwen Team},
    month = {September},
    year = {2024}
}

@article{qwen2,
      title={Qwen2 Technical Report}, 
      author={An Yang and Baosong Yang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Zhou and Chengpeng Li and Chengyuan Li and Dayiheng Liu and Fei Huang and Guanting Dong and Haoran Wei and Huan Lin and Jialong Tang and Jialin Wang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Ma and Jin Xu and Jingren Zhou and Jinze Bai and Jinzheng He and Junyang Lin and Kai Dang and Keming Lu and Keqin Chen and Kexin Yang and Mei Li and Mingfeng Xue and Na Ni and Pei Zhang and Peng Wang and Ru Peng and Rui Men and Ruize Gao and Runji Lin and Shijie Wang and Shuai Bai and Sinan Tan and Tianhang Zhu and Tianhao Li and Tianyu Liu and Wenbin Ge and Xiaodong Deng and Xiaohuan Zhou and Xingzhang Ren and Xinyu Zhang and Xipin Wei and Xuancheng Ren and Yang Fan and Yang Yao and Yichang Zhang and Yu Wan and Yunfei Chu and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhihao Fan},
      journal={arXiv preprint arXiv:2407.10671},
      year={2024}
}
```