Instructions to use pmysl/c4ai-command-r-plus-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use pmysl/c4ai-command-r-plus-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf pmysl/c4ai-command-r-plus-GGUF:Q4_K_M

Use Docker

docker model run hf.co/pmysl/c4ai-command-r-plus-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use pmysl/c4ai-command-r-plus-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pmysl/c4ai-command-r-plus-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pmysl/c4ai-command-r-plus-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pmysl/c4ai-command-r-plus-GGUF:Q4_K_M

Ollama
How to use pmysl/c4ai-command-r-plus-GGUF with Ollama:
```
ollama run hf.co/pmysl/c4ai-command-r-plus-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use pmysl/c4ai-command-r-plus-GGUF with Docker Model Runner:
```
docker model run hf.co/pmysl/c4ai-command-r-plus-GGUF:Q4_K_M
```

Lemonade

How to use pmysl/c4ai-command-r-plus-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull pmysl/c4ai-command-r-plus-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.c4ai-command-r-plus-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

c4ai-command-r-plus-GGUF / README.md

pmysl

Update README.md

7d4fb9d over 2 years ago

preview code

Raw History Blame

1.44 kB

metadata

license: cc-by-nc-4.0
pipeline_tag: text-generation
base_model: CohereForAI/c4ai-command-r-plus

Command R+ GGUF

Description

This repository contains experimental GGUF weights that are currently compatible with pull request #6491 in the llama.cpp. I will update them once support for Command R+ is merged into the llama.cpp repository.

Getting started

Clone the Carolinabanana/llama.cpp repository:

git clone https://github.com/Carolinabanana/llama.cpp.git llama.cpp-fork
cd llama.cpp-fork
git reset --hard 8b6577bd631fec33eeadb4b9dfc5a07ed2118148

Build it using make
Use it in the same way as the regular llama.cpp. If you're unsure of how to start, you can use the following command as a starting point:

./main -p "<|START_OF_TURN_TOKEN|><|USER_TOKEN|>Who are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>" --color -m /path/to/command-r-plus-Q3_K_L-00001-of-00002.gguf

Merging Weights

After commit 8a28d12, weights are split with gguf-split, which means that you don't have to merge weights. Simply pass the first split, as in the example above, and llama.cpp will automatically load all splits. If, for some reason, you want to merge splits, you can use the following command:

./gguf-split --merge /path/to/command-r-plus-f16-00001-of-00005.gguf /path/to/command-r-plus-f16-combined.gguf