Instructions to use bartowski/DeepSeek-V2.5-1210-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use bartowski/DeepSeek-V2.5-1210-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M

Use Docker

docker model run hf.co/bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use bartowski/DeepSeek-V2.5-1210-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bartowski/DeepSeek-V2.5-1210-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bartowski/DeepSeek-V2.5-1210-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M

Ollama
How to use bartowski/DeepSeek-V2.5-1210-GGUF with Ollama:
```
ollama run hf.co/bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use bartowski/DeepSeek-V2.5-1210-GGUF with Docker Model Runner:
```
docker model run hf.co/bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M
```

Lemonade

How to use bartowski/DeepSeek-V2.5-1210-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bartowski/DeepSeek-V2.5-1210-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.DeepSeek-V2.5-1210-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

4k context by default?

by mclassHF2023 - opened Dec 28, 2024

Discussion

mclassHF2023

Dec 28, 2024

Thanks for the GGUFs as always! Just a question:
The original model has 160k context length. I'm not familiar with all the rope_freq_base and compress_pos_emb settings, but something seems off.
I do believe that it's possible to get more than 4k context out of this, but don't really know 100% how?

mclassHF2023

Jan 7, 2025

Sorry for the bump, but I'm still confused by this one...

bartowski

Owner Jan 7, 2025

oh sorry about that, if you look at the original model's config.json file it'll give you some guidance:

https://huggingface.co/deepseek-ai/DeepSeek-V2.5-1210/blob/main/config.json#L39

  "rope_scaling": {
    "beta_fast": 32,
    "beta_slow": 1,
    "factor": 40,
    "mscale": 1.0,
    "mscale_all_dim": 1.0,
    "original_max_position_embeddings": 4096,
    "type": "yarn"
  },

so you'll need to set the ROPE to with yarn

you should be able to see the options if you use --help i think, but also you can find the values in the server README here:

https://github.com/ggerganov/llama.cpp/tree/53ff6b9b9fb25ed0ec0a213e05534fe7c3d0040f/examples/server

should work for llama-cli as well I'm pretty sure, just CTRL+F for 'yarn'

my best guess is you'll want:

--rope-scaling yarn --yarn-orig-ctx 4096 --yarn-attn-factor 40

beta_slow is default 1 and beta_fast is default 32

you might also be able to get away with JUST --rope-scaling yarn, there's an implication that it's able to load from the model's data, but I'm not positive

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment