Instructions to use ubergarm/Qwen3.8-27B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ubergarm/Qwen3.8-27B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/Qwen3.8-27B-GGUF
# Run inference directly in the terminal:
llama cli -hf ubergarm/Qwen3.8-27B-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/Qwen3.8-27B-GGUF
# Run inference directly in the terminal:
llama cli -hf ubergarm/Qwen3.8-27B-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ubergarm/Qwen3.8-27B-GGUF
# Run inference directly in the terminal:
./llama-cli -hf ubergarm/Qwen3.8-27B-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ubergarm/Qwen3.8-27B-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ubergarm/Qwen3.8-27B-GGUF

Use Docker

docker model run hf.co/ubergarm/Qwen3.8-27B-GGUF

LM Studio
Jan

vLLM

How to use ubergarm/Qwen3.8-27B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubergarm/Qwen3.8-27B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubergarm/Qwen3.8-27B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubergarm/Qwen3.8-27B-GGUF

Ollama
How to use ubergarm/Qwen3.8-27B-GGUF with Ollama:
```
ollama run hf.co/ubergarm/Qwen3.8-27B-GGUF
```
Unsloth Desktop

How to use ubergarm/Qwen3.8-27B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3.8-27B-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ubergarm/Qwen3.8-27B-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ubergarm/Qwen3.8-27B-GGUF with Docker Model Runner:
```
docker model run hf.co/ubergarm/Qwen3.8-27B-GGUF
```

Lemonade

How to use ubergarm/Qwen3.8-27B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ubergarm/Qwen3.8-27B-GGUF

Run and chat with the model

lemonade run user.Qwen3.8-27B-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use ubergarm/Qwen3.8-27B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3.8-27B-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ubergarm/Qwen3.8-27B-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ubergarm/Qwen3.8-27B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ubergarm/Qwen3.8-27B-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ubergarm/Qwen3.8-27B-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Testing MTP-IQ4_KS

by shewin - opened 21 days ago

Discussion

shewin

21 days ago

Layer 65: 1288.28, 6192.00, 7480.28 MiB (output layer)

Total : 12777.92, 12341.48, 25119.40 MiB
Memory required for model tensors + cache: 26408 MiB
Memory available on all devices - compute: 28264 MiB
llm_load_tensors: ggml ctx size = 0.90 MiB
model has unused tensor output_extra.weight (size = 676423680 bytes) -- ignoring
model has unused tensor blk.64.attn_norm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.64.post_attention_norm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.64.attn_q.weight (size = 66846720 bytes) -- ignoring
model has unused tensor blk.64.attn_k.weight (size = 5570560 bytes) -- ignoring
model has unused tensor blk.64.attn_v.weight (size = 5570560 bytes) -- ignoring
model has unused tensor blk.64.attn_output.weight (size = 33423360 bytes) -- ignoring
model has unused tensor blk.64.attn_q_norm.weight (size = 1024 bytes) -- ignoring
model has unused tensor blk.64.attn_k_norm.weight (size = 1024 bytes) -- ignoring
model has unused tensor blk.64.ffn_gate.weight (size = 94699520 bytes) -- ignoring
model has unused tensor blk.64.ffn_down.weight (size = 94699520 bytes) -- ignoring
model has unused tensor blk.64.ffn_up.weight (size = 94699520 bytes) -- ignoring
model has unused tensor blk.64.nextn.eh_proj.weight (size = 55705600 bytes) -- ignoring
model has unused tensor blk.64.nextn.enorm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.64.nextn.hnorm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.64.nextn.shared_head_norm.weight (size = 20480 bytes) -- ignoring
llm_load_tensors: offloading 65 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 66/66 layers to GPU
llm_load_tensors: CUDA_Host buffer size = 985.16 MiB
llm_load_tensors: CUDA0 buffer size = 14066.22 MiB
........................................................................................
~ggml_backend_cuda_context: have 0 graphs
llama_init_from_model: n_ctx = 170240
llama_init_from_model: n_batch = 8192
llama_init_from_model: n_ubatch = 8192
llama_init_from_model: flash_attn = 1
llama_init_from_model: attn_max_b = 2048
llama_init_from_model: fused_moe = 1
llama_init_from_model: grouped er = 0
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad = 1
llama_init_from_model: rope_cache = 0
llama_init_from_model: graph_reuse = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: v_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 0
llama_init_from_model: reduce_type = f16
llama_init_from_model: sched_async = 0
llama_init_from_model: ser = -1, 0
llama_init_from_model: freq_base = 10000000.0
llama_init_from_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 5802.13 MiB
llama_init_from_model: KV self size = 5652.50 MiB, K (q8_0): 2826.25 MiB, V (q8_0): 2826.25 MiB
llama_init_from_model: CUDA_Host output buffer size = 0.95 MiB
llama_init_from_model: CUDA0 compute buffer size = 8080.00 MiB
llama_init_from_model: CUDA_Host compute buffer size = 2820.22 MiB
llama_init_from_model: graph nodes = 3125
llama_init_from_model: graph splits = 2
llama_init_from_model: enabling only_active_experts scheduling

main: n_kv_max = 170240, n_batch = 8192, n_ubatch = 8192, flash_attn = 1, n_gpu_layers = 99, n_threads = 80, n_threads_batch = 80

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
8192	2048	0	2.571	3186.48	24.220	84.56
8192	2048	8192	2.548	3215.23	26.738	76.59
8192	2048	16384	2.677	3059.88	28.230	72.55
8192	2048	24576	2.677	3060.68	29.434	69.58

shewin

21 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment