Instructions to use Qwen/Qwen2-7B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Qwen/Qwen2-7B-Instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Qwen/Qwen2-7B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2-7B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-7B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M

Ollama
How to use Qwen/Qwen2-7B-Instruct-GGUF with Ollama:
```
ollama run hf.co/Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Qwen/Qwen2-7B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M
```

Lemonade

How to use Qwen/Qwen2-7B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Qwen/Qwen2-7B-Instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen2-7B-Instruct-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Issue with q8_0

by sm54 - opened Jun 6, 2024

Discussion

sm54

Jun 6, 2024

•

edited Jun 6, 2024

Hello,

I downloaded the q8_0 model and it is giving me a strange response, shown below. I am using text generation webui and using the chat template of "Custom (obtained from model metadata)". My other parameters are quite standard.

AI
How can I help you today?

You
Hello

AI
Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly Blockly

bartowski

Qwen org Jun 6, 2024

try to not offload to CUDA or enable flash attention (-fa in llama.cpp)

sm54

Jun 6, 2024

With flash attention enabled I get the same result, and if I set gpu layers to zero and tensorcores to off I get this:

AI
How can I help you today?

You
Hello

AI
Blockly is a visual programming language that allows users to create programs using blocks. It is designed to be intuitive and easy to use, making it a popular choice for teaching programming concepts to beginners. Here are some ways Blockly can help you:

Educational Tool: Blockly is often used in educational settings to teach children and adults the basics of programming. It breaks down complex concepts into simple, manageable blocks that are easy to understand. This makes it an excellent tool for learning programming logic, algorithms, and basic syntax.
Interactive Learning: The visual nature of Blockly allows for interactive learning.

sm54

Jun 6, 2024

Okay, if I set it to CPU only mode as well, now it seems to work, it just runs slow.

bartowski

Qwen org Jun 6, 2024

Ah okay. Was hoping fa would work but maybe not. Either way there's a CUDA bug

zhicwu

Jun 6, 2024

Confirmed working after applying this patch.

najomi

Jun 7, 2024

I'm using jan.ai, I get the blocky error when using CUDA, using the CPU works fine but it's super slow. Is there any fix for CUDA?

shishao

Jun 8, 2024

I encountered the same issue. I was running qwen2 on ollama, but qwen2 returned a bunch of repeating
characters. This issue was resolved by adding the environment variable OLLAMA_FLASH_ATTENTION=1. The
corresponding connection is https://github.com/QwenLM/Qwen2?tab=readme-ov-file.

shishao

Jun 8, 2024

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_FLASH_ATTENTION=1"

[Install]
WantedBy=default.target

Above , Ollama serves' setting , I don't know how to add configs in Environment, so I add Environment again. Maybe I was wrong，but it is worked