Instructions to use RemySkye/rwkv7-g1h-7.2b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RemySkye/rwkv7-g1h-7.2b-GGUF with RWKV:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use RemySkye/rwkv7-g1h-7.2b-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use RemySkye/rwkv7-g1h-7.2b-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RemySkye/rwkv7-g1h-7.2b-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RemySkye/rwkv7-g1h-7.2b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M

Ollama
How to use RemySkye/rwkv7-g1h-7.2b-GGUF with Ollama:
```
ollama run hf.co/RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use RemySkye/rwkv7-g1h-7.2b-GGUF with Docker Model Runner:
```
docker model run hf.co/RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M
```

Lemonade

How to use RemySkye/rwkv7-g1h-7.2b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull RemySkye/rwkv7-g1h-7.2b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.rwkv7-g1h-7.2b-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

rwkv7-g1h-7.2b-20260710-ctx10240 GGUF

GGUF conversions of rwkv7-g1h-7.2b-20260710-ctx10240.pth. This repository is one model only; the other parameter sizes are published in separate repositories.

Conversion

The source checkpoint is BF16. It was converted directly from .pth to a BF16 GGUF with RWKV's converter, then the standard release ladder below was produced from that BF16 GGUF using the pinned llama.cpp quantizer. No safetensors staging file and no importance matrix were used.

Source revision: 6d5762253b343eec6cfbf5ed62f872f30a4cd89c
rwkv-mobile revision: ebfb744281c31a07aad5606ec7473f79f837e92a
llama.cpp revision: c92e806d1c81091c9035edce99c35374da1b465e
Context marker in source filename: ctx10240

Files

BF16 master GGUF
Q2_K
Q3_K_S
IQ4_XS
Q4_K_S
Q5_K_S
Q6_K
Q8_0
Q4_0
Q4_1
Q5_0
Q5_1

Quant	PPL	BF16 retained
`BF16`	6.0738	100.00%
`Q8_0`	6.0739	100.00%
`Q6_K`	6.0941	99.67%
`Q5_1`	6.1495	98.77%
`Q5_K_M`	6.143	98.87%
`Q5_K_S`	6.1579	98.63%
`Q5_0`	6.1763	98.34%
`Q4_1`	6.3985	94.93%
`Q4_K_M`	6.2938	96.50%
`Q4_K_S`	6.3748	95.28%
`Q4_0`	7.3582	82.54%
`IQ4_XS`	6.4467	94.22%
`Q3_K_L`	6.4221	94.58%
`Q3_K_M`	6.5279	93.04%
`Q3_K_S`	19.6605	30.89%
`Q2_K`	276.2831	2.20%

RWKV-aware mixed quantizations

The Q3_K_M, Q3_K_L, Q4_K_M, and Q5_K_M files use custom RWKV-aware recipes with explicit tensor assignments. Higher precision is used for the token embeddings and selected value, time-mix output, and channel-mix tensors where it is expected to preserve the most quality.

Earlier automated files with these names were removed after verification showed that llama.cpp's generic mixed recipes did not recognize RWKV's time_mix_* and channel_mix_* tensor roles. Because of that, the automated M and L variants had collapsed to the same effective layouts as the retained S variants.

These replacement files have genuinely different tensor layouts, providing additional size and quality choices between the existing S variants and the larger quantizations.

Prompting

Follow the prompt guide in the upstream model card. Avoid a trailing space at the end of the input.

Downloads last month: 775

GGUF

Model size

7B params

Architecture

rwkv7

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for RemySkye/rwkv7-g1h-7.2b-GGUF

Base model

BlinkDL/rwkv7-g1

Quantized

(36)

this model

Collection including RemySkye/rwkv7-g1h-7.2b-GGUF

RWKV7 G1H GGUF

Collection

8 items • Updated Aug 17 • 1