Instructions to use EleutherAI/Hermes-RWKV-v5-3B-HF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EleutherAI/Hermes-RWKV-v5-3B-HF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="EleutherAI/Hermes-RWKV-v5-3B-HF", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("EleutherAI/Hermes-RWKV-v5-3B-HF", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use EleutherAI/Hermes-RWKV-v5-3B-HF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EleutherAI/Hermes-RWKV-v5-3B-HF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EleutherAI/Hermes-RWKV-v5-3B-HF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/EleutherAI/Hermes-RWKV-v5-3B-HF

SGLang

How to use EleutherAI/Hermes-RWKV-v5-3B-HF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "EleutherAI/Hermes-RWKV-v5-3B-HF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EleutherAI/Hermes-RWKV-v5-3B-HF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "EleutherAI/Hermes-RWKV-v5-3B-HF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EleutherAI/Hermes-RWKV-v5-3B-HF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use EleutherAI/Hermes-RWKV-v5-3B-HF with Docker Model Runner:
```
docker model run hf.co/EleutherAI/Hermes-RWKV-v5-3B-HF
```

Hermes-RWKV-v5-3B-HF / README.md

norabelrose

Update README.md

7d5ec7d verified over 2 years ago

preview code

Raw History Blame Contribute Delete

1.55 kB

	---
	license: apache-2.0
	datasets:
	- teknium/OpenHermes-2.5
	language:
	- en
	---
	This is a 3B parameter RWKV-v5 ("Eagle") RNN language model pretrained on multi-lingual data and fine-tuned on Nous Research's OpenHermes 2.5 chat dataset.

	The chat format is identical to the default HuggingFace format, EXCEPT that we replace the `<\|im_start\|>` and `<\|im_end\|>` special tokens with the unused ASCII codes `"\x16"` and `"\x17"` respectively. This was done to ensure complete compatibility withe standard RWKV World tokenizer.

	An example interaction:
	```
	\x16user
	Write a poem about the beauty of mathematics in the style of Emily Dickinson.\x17
	\x16assistant
	In the heart of Numbers, so precise,
	A Beauty dwells - both cool and nice.
	An elegance that sings in sums,
	And dances to algebraic drums.

	Addition's grace, subtraction's might,
	Multiplication's sheer delight,
	Division's firm yet gentle hand,
	All part of this enchanted land.

	Infinity, a lover bold,
	To whom the stars their secrets told,
	While Zero, cloaked in mystery,
	Unlocks each mathematical decree.

	Fractals bloom like flowers wild,
	Geometry, Nature's favored child,
	Calculus, with steady gaze,
	Maps out life's intricate maze.

	Pi, an endless serenade,
	In circles' arms forever laid,
	And prime numbers,
	standing lone,
	Make the universe their own.

	Each equation, theorem scribed,
	In
	parchment old or computer imbibed,
	Is but a whisper from the skies,
	A cosmic truth in disguise.\x17
	```
	In the above, newlines are printed without escaping, while ASCII codes are escaped for readability.