deploy/DEPLOYMENT.md

Muse-Glimmer-30B-Abliterated-GGUF — Deployment Kit

Serve the abliterated Muse Glimmer 30B locally with llama.cpp, optionally with the DFlash block-diffusion drafter for ~1.6–3× faster decoding (lossless — identical output).

---

1. Requirements

• A recent llama.cpp build with CUDA/Metal (DFlash support merged in ggml-org/llama.cpp#26841). Build from master.
• Disk for the quant you pick (Q4_K_M ≈ 16 GB, Q8_0 ≈ 28 GB) + the DFlash drafter (≈ 4.8 GB).
• A GPU with enough VRAM for the quant, or CPU (slower).

2. Get the files

Text quant (pick one) + the DFlash drafter, from this repo:

hf download Blackfrost-Research/Muse-Glimmer-30B-Abliterated-GGUF \
  Muse-Glimmer-30B-Abliterated-Q8_0.gguf \
  dflash-Muse-Glimmer-30B-Abliterated-F16.gguf \
  --local-dir ./muse
# for image input, also grab a projector:
hf download Blackfrost-Research/Muse-Glimmer-30B-Abliterated-GGUF \
  mmproj-Muse-Glimmer-30B-Abliterated-F16.gguf --local-dir ./muse

3. Serve

Plain (no speculation):

llama-server -m ./muse/Muse-Glimmer-30B-Abliterated-Q8_0.gguf \
  -ngl 999 --jinja --host 0.0.0.0 --port 8080 -c 16384 \
  --temp 1.0 --top-p 0.95 --top-k 64

With DFlash speculative decoding (recommended — faster, same output):

llama-server -m ./muse/Muse-Glimmer-30B-Abliterated-Q8_0.gguf \
  -md ./muse/dflash-Muse-Glimmer-30B-Abliterated-F16.gguf \
  --spec-type draft-dflash --spec-draft-n-max 15 \
  -ngl 999 -ngld 999 --jinja --host 0.0.0.0 --port 8080 -c 16384 \
  --temp 1.0 --top-p 0.95 --top-k 64

• --spec-type draft-dflash selects the block-diffusion draft path (block size 16, drafter injects the target's hidden states into its attention). DFlash must run under llama-server (it needs the shared target context) — not llama-cli.
• --spec-draft-n-max 15 is the draft block length (clamped to the trained 16).
• -md = the DFlash drafter file; -ngld 999 offloads it to GPU too.

Multimodal (image input): add the projector to either command:

  --mmproj ./muse/mmproj-Muse-Glimmer-30B-Abliterated-F16.gguf

Or run the ready-made script in this kit: bash serve.sh (see serve.sh).

4. Query (OpenAI-compatible)

curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages":[{"role":"user","content":"Write a binary search in Python."}],
  "max_tokens": 2048, "temperature": 1.0, "top_p": 0.95, "top_k": 64
}'

5. Important notes

• Heavy thinker. Muse spends tokens on a reasoning channel first. Use a generous max_tokens (≥ 1024) or the answer can come back empty (the budget is consumed by reasoning). llama-server returns the reasoning in reasoning_content and the answer in content.
• Sampling (Meta guidance): temperature 1.0, top_p 0.95, top_k 64. Steer depth with a Reasoning strength: low/medium/high/xhigh line in the system prompt.
• Flash attention: -fa on is fastest and lets DFlash reach its peak speedup. If your GPU/CUDA combo hangs at load with -fa on (seen on some brand-new archs paired with an older CUDA toolkit), use -fa off — everything works, at a modest speed cost.
• Persona: ships with an "AI assistant" system template baked in; override with your own system message.

6. Reference performance

Single NVIDIA RTX PRO 6000 (Blackwell), Q8_0, -fa off:

config	decode tok/s	speedup
baseline	~46	1.0×
+ DFlash	~73	1.6×

DFlash speedup rises with structured/code output and with -fa on. Meta reports up to 3.1× on an RTX 5090.

---

Built by Blackfrost · Las Vegas, NV. Not affiliated with Meta.