Instructions to use Reiterate3680/guns-and-roses-r1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Reiterate3680/guns-and-roses-r1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Reiterate3680/guns-and-roses-r1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Reiterate3680/guns-and-roses-r1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Reiterate3680/guns-and-roses-r1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M

Ollama
How to use Reiterate3680/guns-and-roses-r1-GGUF with Ollama:
```
ollama run hf.co/Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Reiterate3680/guns-and-roses-r1-GGUF with Docker Model Runner:
```
docker model run hf.co/Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M
```

Lemonade

How to use Reiterate3680/guns-and-roses-r1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Reiterate3680/guns-and-roses-r1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.guns-and-roses-r1-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

...So what happened here?

by inflatebot - opened Aug 24, 2024

Discussion

inflatebot

Aug 24, 2024

Was having a poke around, noticed that this is the most downloaded model on your page by like... an order of magnitude.
My main repo hasn't gotten much attention, so that fact is interesting to me.
Nobody's really been talking about it where I can see! But that this particular quant has gotten more attention than even mradermacher's quant of Starcannon V3 is intriguing.

Would you happen to have heard anything about this?

Mostly curious because if I'm doing something right, I wanna know, y'know?

Reiterate3680

Owner Aug 24, 2024

Not sure either, it looks like it's backlinked from https://colab.research.google.com/drive/1l_wRGeD-LnRl3VtZHDc7epW_XW0nJvew which in turn links to someone's model review sheet. I guess this one also has more downloads because people are using this model but there's no other quants? I might as well go make the rest of the GGUF then

John6666

Aug 24, 2024

Over 3,000 downloads is amazing.🙄
I just tried it and this model is definitely one of the most Japanese, English, Anime, NSFW, and Danbooru tagging models in the 8B-12B class.
The response is a bit peaky, but I guess that depends on the settings.

inflatebot

Aug 24, 2024

Not sure either, it looks like it's backlinked from https://colab.research.google.com/drive/1l_wRGeD-LnRl3VtZHDc7epW_XW0nJvew which in turn links to someone's model review sheet. I guess this one also has more downloads because people are using this model but there's no other quants? I might as well go make the rest of the GGUF then

Yeah, mradermacher had some up for a while, but they disappeared. Not sure what happened there.
Their comment indicates that the project was more or less successful:
"Developed on top of magnum. deliver good prose and brain, It does very well with character that has unusual anatomy (especially non-human/furry)"
That was basically the point, and that they reached that conclusion without my advertising it as such, to me, indicates some success.

This notebook came from the JanitorAI Discord, which would get quite a few eyes on it. So I guess that tracks?

Not bad for a merge using a cursed method that I did not adequately test. :P

inflatebot

Sep 4, 2024

Over 3,000 downloads is amazing.🙄
I just tried it and this model is definitely one of the most Japanese, English, Anime, NSFW, and Danbooru tagging models in the 8B-12B class.
The response is a bit peaky, but I guess that depends on the settings.

I meant to respond to this but completely forgot -- thank you very much for the kind words!
That's an interesting use case, not one that I had in mind when I designed it, but given the type of data that Rosier was trained on, it does make sense. ^^

John6666

Sep 4, 2024

Thanks for a nice model!😊

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment