DeepSeek-V4.1-Flash-GSQ-RCO-GGUF

Instructions to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16
# Run inference directly in the terminal:
./llama-cli -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Use Docker

docker model run hf.co/taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

LM Studio
Jan

vLLM

How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Ollama
How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with Ollama:
```
ollama run hf.co/taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16
```
Unsloth Desktop

How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with Docker Model Runner:
```
docker model run hf.co/taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16
```

Lemonade

How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Run and chat with the model

lemonade run user.DeepSeek-V4.1-Flash-GSQ-RCO-GGUF-BF16

List all available models

lemonade list

Hermes Agent

How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

DeepSeek-V4.1-Flash-GSQ-RCO-GGUF / eval /2026-09-13 /provenance /deepseek-mmlu-runtime.json

taurusduan

Duplicate from pfeifferj/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF

0b10848 4 days ago

Raw

History Blame Contribute Delete

3.64 kB

	{
	"status": "prepared_not_published",
	"base_repository": "https://github.com/vcruz305/llama.cpp",
	"base_revision": "5210c7c5ed61dddaee6ed476623abf4b63093d16",
	"base_source_archive_sha256": "3e0ad09b1f2175a170d87310731ac12d1bf52d8ab0b1f4fbdb19298457ae73e6",
	"patch_sha256": "1a70ed3aacb89fec18ff7fdd263702a767f1e2606b3a57bad425035c1ed29fe6",
	"model": "pfeifferj/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF",
	"model_file": "DeepSeek-V4.1-Flash-GSQ-RCO-3.0bit.gguf",
	"model_sha256": "11f46543370256ef616b6e458b6950e148625b5f8b7545173d124445d3393c53",
	"official_reference_revision": "dba1be0a40aa45a94ad051997016db3960a90277",
	"runtime_artifacts_sha256": {
	"libllama.so.0.4.0": "9abf951448ec289625e27aff805986ea80780afc1c7e9b745162f3fd5cf3d7d8",
	"libggml-cuda.so.0.23.0": "b33b8b786f7c61c1cc10a6699d1f5d778eedfa094dec34f9013c38a82c8f5648",
	"mc-score-deepseek-f32": "e5b8a91ba2547b02a14668335b590ddcac185b6b606c3b10f50b0cf29834fb4c",
	"llama-perplexity": "51197334ee7c016569cbbb893eb20150cc2714ef46a161c34f00ae5175da3632"
	},
	"evaluation_settings": {
	"context": 2048,
	"batch": 2048,
	"microbatch": 2048,
	"sequences": 1,
	"threads": 16,
	"flash_attention": false,
	"cache_k": "f32",
	"cache_v": "f32",
	"CUDA_VISIBLE_DEVICES": "2,3",
	"NVIDIA_TF32_OVERRIDE": "0",
	"GGML_CUDA_MMF_F32_DISABLE": "1",
	"GGML_CUDA_REFERENCE_F32": "1",
	"engram_placement": "lazy CPU mmap",
	"method": "single prompt forward, raw answer-letter single-token log-probabilities, variable option counts from archived task index, no chat template",
	"tokenization": {
	"scorer_add_special": true,
	"gguf_add_bos_token": false,
	"gguf_add_eos_token": false,
	"effective_raw_prompt": "No forced BOS or EOS; preserve original Spark scorer and model metadata behavior",
	"chat_prompt": "Pinned official encoder supplies BOS explicitly"
	}
	},
	"throughput_pilot": {
	"questions": 20,
	"decode_seconds": 32.749756,
	"seconds_per_question": 1.6374878,
	"projected_2000_decode_minutes": 54.582926666666665
	},
	"ppl_sidecar_patch_sha256": "2ed02e1d9c7a77303ef50153792b3a40da19cadc394ae0d79aa76d59c8ba2068",
	"question_state_isolation": "research/question-state-isolation.md",
	"tokenizer_parity": {
	"status": "PASS",
	"vocabulary_size": 129280,
	"compared": "Every native token ID, exact token string, and decoded byte piece including special tokens",
	"token_string_mapping_sha256": "ad4e020abc59468b66b83039c5663753efe861c0af3cda9034bfc283d298059c",
	"token_byte_mapping_sha256": "dced44b89abfc32450c7398912681c477f0da4db310b0caa1bbdcf14ed0e9c7d",
	"tokenizer_sha256": "c90dfa01249db1be4245780a052ede752e1361c612ac6d08e2bdada7d599476b",
	"native_dump_sha256": "c53162bbc06168cdb73b3e69f41c13fb1ed4b8603dd33ac0f85e54e78df9bebc"
	},
	"accepted_mmlu": {
	"correct": 1220,
	"n": 2000,
	"accuracy": 0.61,
	"tsv_sha256": "7ac61bdb80ce8c0c41079c2d2364968442380785bfdb33246d3b0846704254e1",
	"physical_gpu_pairs": [
	[
	2,
	3
	],
	[
	0,
	1
	]
	],
	"cross_pair_parity": "All20 predictions and every printed option log probability match exactly",
	"final_merge": {
	"status": "PASS",
	"rows": 2000,
	"primary": [
	0,
	1500
	],
	"tail": [
	1700,
	2000
	],
	"sha256": "7ac61bdb80ce8c0c41079c2d2364968442380785bfdb33246d3b0846704254e1",
	"reused_initial_rows": 824,
	"second_restart": 1379,
	"middle": [
	1500,
	1700
	],
	"same_binary_and_settings": true
	}
	}
	}