Instructions to use YTan2000/Muse-Glimmer-30B-TQ3_4S with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0
# Run inference directly in the terminal:
llama cli -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0
# Run inference directly in the terminal:
llama cli -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Use Docker

docker model run hf.co/YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

LM Studio
Jan

vLLM

How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "YTan2000/Muse-Glimmer-30B-TQ3_4S"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YTan2000/Muse-Glimmer-30B-TQ3_4S",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Ollama
How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with Ollama:
```
ollama run hf.co/YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0
```
Unsloth Desktop

How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with Docker Model Runner:
```
docker model run hf.co/YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0
```

Lemonade

How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Run and chat with the model

lemonade run user.Muse-Glimmer-30B-TQ3_4S-Q8_0

List all available models

lemonade list

Hermes Agent

How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use YTan2000/Muse-Glimmer-30B-TQ3_4S with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "YTan2000/Muse-Glimmer-30B-TQ3_4S:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Model Crashes

by jacquesbezuidenhout - opened 23 days ago

Discussion

jacquesbezuidenhout

23 days ago

Hi there, thank you for the effort with the model - i get a significant jump in tokens with this model - 60 -> 83t/s, however, basic queries work, but as soon as i call from within an agent, such as openclaw, it crashes. :-( this is sad, because there is a clear jump in efficiency, from the base or unsloth versions. If you need anymore info, please shout, and I'll help troubleshoot

Runtime stopped before the model became ready Aug 14 14:01:23 mint runtime-run[2083143]: #6 0x000079bdbbd59217 in ggml_backend_sched_graph_compute_async () from /home/jacques/github/llama-server/tq3/runtime/libggml-base.so.0 Aug 14 14:01:23 mint runtime-run[2083143]: #7 0x000079bdbb0f30f1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/jacques/github/llama-server/tq3/runtime/libllama.so.0 Aug 14 14:01:23 mint runtime-run[2083143]: #8 0x000079bdbb0f738a in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/jacques/github/llama-server/tq3/runtime/libllama.so.0 Aug 14 14:01:23 mint runtime-run[2083143]: #9 0x000079bdbb0fe558 in llama_context::decode(llama_batch const&) () from /home/jacques/github/llama-server/tq3/runtime/libllama.so.0 Aug 14 14:01:23 mint runtime-run[2083143]: #10 0x000079bdbb10026f in llama_decode () from /home/jacques/github/llama-server/tq3/runtime/libllama.so.0 Aug 14 14:01:23 mint runtime-run[2083143]: #11 0x000079bdbc3a95ea in server_context_impl::decode(int&, int, llama_batch&) () from /home/jacques/github/llama-server/tq3/runtime/libllama-server-impl.so Aug 14 14:01:23 mint runtime-run[2083143]: #12 0x000079bdbc3aad68 in server_context_impl::update_slots() () from /home/jacques/github/llama-server/tq3/runtime/libllama-server-impl.so Aug 14 14:01:23 mint runtime-run[2083143]: #13 0x000079bdbc3517e1 in server_queue::start_loop(long) () from /home/jacques/github/llama-server/tq3/runtime/libllama-server-impl.so Aug 14 14:01:23 mint runtime-run[2083143]: #14 0x000079bdbc2f1aaa in llama_server(common_params&, int, char**) () from /home/jacques/github/llama-server/tq3/runtime/libllama-server-impl.so Aug 14 14:01:23 mint runtime-run[2083143]: #15 0x000079bdbc2f37cf in llama_server(int, char**) () from /home/jacques/github/llama-server/tq3/runtime/libllama-server-impl.so Aug 14 14:01:23 mint runtime-run[2083143]: #16 0x000079bdbba2a1ca in __libc_start_call_main (main=main@entry=0x567a2fa8e270

, argc=argc@entry=46, argv=argv@entry=0x7fff5a4f5068) at ../sysdeps/nptl/libc_start_call_main.h:58 Aug 14 14:01:23 mint runtime-run[2083143]: warning: 58 ../sysdeps/nptl/libc_start_call_main.h: No such file or directory Aug 14 14:01:23 mint runtime-run[2083143]: #17 0x000079bdbba2a28b in __libc_start_main_impl (main=0x567a2fa8e270

, argc=46, argv=0x7fff5a4f5068, init=, fini=, rtld_fini=, stack_end=0x7fff5a4f5058) at ../csu/libc-start.c:360 Aug 14 14:01:23 mint runtime-run[2083143]: warning: 360 ../csu/libc-start.c: No such file or directory Aug 14 14:01:23 mint runtime-run[2083143]: #18 0x0000567a2fa8e2a5 in _start () Aug 14 14:01:23 mint runtime-run[2083143]: [Inferior 1 (process 2074938) detached] Aug 14 14:01:28 mint systemd-coredump[2083160]: [🡕] Process 2074938 (llama-server) of user 1000 dumped core. Module libgomp.so.1 from deb gcc-14-14.2.0-4ubuntu2~~24.04.1.amd64 Module libgcc_s.so.1 from deb gcc-14-14.2.0-4ubuntu2~~24.04.1.amd64 Module libstdc++.so.6 from deb gcc-14-14.2.0-4ubuntu2~24.04.1.amd64 Stack trace of thread 2074938: #0 0x000079bdbba9ec0c __pthread_kill_implementation (libc.so.6 + 0x9ec0c) #1 0x000079bdbba4527e __GI_raise (libc.so.6 + 0x4527e) #2 0x000079bdbba288ff __GI_abort (libc.so.6 + 0x288ff) #3 0x000079bdbbd3aae6 n/a (/home/jacques/github/llama-server/tq3/runtime/libggml-base.so.0.18.0 + 0x1dae6) #4 0x69672f7365757163 n/a (n/a + 0x0) ELF object binary architecture: AMD x86-64 Aug 14 14:01:29 mint systemd[1996544]: llama-server.service: Main process exited, code=dumped, status=6/ABRT Aug 14 14:01:29 mint systemd[1996544]: llama-server.service: Failed with result 'core-dump'. Aug 14 14:01:29 mint systemd[1996544]: llama-server.service: Consumed 1min 30.289s CPU time.

YTan2000

Owner 23 days ago

Hi Jacques! Thanks for the report — and glad the new runtime is giving you a real speed jump.

I've reproduced your scenario on our side with Muse-Glimmer-30B-TQ3_4S + the dflash drafter: multi-turn agent traffic with tool schemas, ~30k-token contexts, streaming, 4096-token outputs, and even concurrent requests — all clean, no crash. So it isn't a general bug in the current code; your stack trace (SIGABRT in graph_compute_async) points to a CUDA allocation failure during decode, which is environment-specific.

Most likely cause: VRAM exhaustion under agent load. Agent calls (system prompt + tool schemas + conversation history + large max_tokens) consume context far faster than basic queries. After ~15 GB of weights + drafter, a 24 GB card has little headroom left for KV at depth, and decode aborts the moment an allocation fails — exactly matching "basic queries work, agent crashes".

To pin it down, could you share:

The log line just ABOVE the stack trace — that's the actual CUDA error message (journalctl -u llama-server --since "14:00" | head -60)
Your full launch command (I can see you started with ~46 args — flags like -c, -ctk/-ctv, --spec-draft-n-max are the ones I need)
Your GPU + VRAM state when the agent connects (nvidia-smi)
Roughly when it dies — first agent turn, or after several turns?

Worth trying in the meantime: lower -c (e.g. 16384) if you have it higher, and -ctk q8_0 -ctv tq3_0 for the KV cache — both cut VRAM pressure significantly.

Happy to keep digging once you can capture those!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment