# Muse-Glimmer-30B-Abliterated-GGUF — Deployment Kit

Serve the abliterated Muse Glimmer 30B locally with **llama.cpp**, optionally with the
**DFlash** block-diffusion drafter for ~1.6–3× faster decoding (lossless — identical output).

---

## 1. Requirements

- A recent **llama.cpp** build with CUDA/Metal (DFlash support merged in [ggml-org/llama.cpp#26841](https://github.com/ggml-org/llama.cpp/pull/26841)). Build from `master`.
- Disk for the quant you pick (Q4_K_M ≈ 16 GB, Q8_0 ≈ 28 GB) + the DFlash drafter (≈ 4.8 GB).
- A GPU with enough VRAM for the quant, or CPU (slower).

## 2. Get the files

Text quant (pick one) + the DFlash drafter, from this repo:

```bash
hf download Blackfrost-Research/Muse-Glimmer-30B-Abliterated-GGUF \
  Muse-Glimmer-30B-Abliterated-Q8_0.gguf \
  dflash-Muse-Glimmer-30B-Abliterated-F16.gguf \
  --local-dir ./muse
# for image input, also grab a projector:
hf download Blackfrost-Research/Muse-Glimmer-30B-Abliterated-GGUF \
  mmproj-Muse-Glimmer-30B-Abliterated-F16.gguf --local-dir ./muse
```

## 3. Serve

**Plain (no speculation):**
```bash
llama-server -m ./muse/Muse-Glimmer-30B-Abliterated-Q8_0.gguf \
  -ngl 999 --jinja --host 0.0.0.0 --port 8080 -c 16384 \
  --temp 1.0 --top-p 0.95 --top-k 64
```

**With DFlash speculative decoding (recommended — faster, same output):**
```bash
llama-server -m ./muse/Muse-Glimmer-30B-Abliterated-Q8_0.gguf \
  -md ./muse/dflash-Muse-Glimmer-30B-Abliterated-F16.gguf \
  --spec-type draft-dflash --spec-draft-n-max 15 \
  -ngl 999 -ngld 999 --jinja --host 0.0.0.0 --port 8080 -c 16384 \
  --temp 1.0 --top-p 0.95 --top-k 64
```

- `--spec-type draft-dflash` selects the block-diffusion draft path (block size 16, drafter injects the target's hidden states into its attention). DFlash must run under `llama-server` (it needs the shared target context) — not `llama-cli`.
- `--spec-draft-n-max 15` is the draft block length (clamped to the trained 16).
- `-md` = the DFlash drafter file; `-ngld 999` offloads it to GPU too.

**Multimodal (image input):** add the projector to either command:
```bash
  --mmproj ./muse/mmproj-Muse-Glimmer-30B-Abliterated-F16.gguf
```

Or run the ready-made script in this kit: `bash serve.sh` (see `serve.sh`).

## 4. Query (OpenAI-compatible)

```bash
curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages":[{"role":"user","content":"Write a binary search in Python."}],
  "max_tokens": 2048, "temperature": 1.0, "top_p": 0.95, "top_k": 64
}'
```

## 5. Important notes

- **Heavy thinker.** Muse spends tokens on a reasoning channel first. Use a **generous `max_tokens` (≥ 1024)** or the answer can come back empty (the budget is consumed by reasoning). llama-server returns the reasoning in `reasoning_content` and the answer in `content`.
- **Sampling (Meta guidance):** `temperature 1.0, top_p 0.95, top_k 64`. Steer depth with a `Reasoning strength: low/medium/high/xhigh` line in the system prompt.
- **Flash attention:** `-fa on` is fastest and lets DFlash reach its peak speedup. If your GPU/CUDA combo hangs at load with `-fa on` (seen on some brand-new archs paired with an older CUDA toolkit), use `-fa off` — everything works, at a modest speed cost.
- **Persona:** ships with an "AI assistant" system template baked in; override with your own system message.

## 6. Reference performance

Single NVIDIA RTX PRO 6000 (Blackwell), Q8_0, `-fa off`:

| config | decode tok/s | speedup |
|---|--:|--:|
| baseline | ~46 | 1.0× |
| + DFlash | ~73 | **1.6×** |

DFlash speedup rises with structured/code output and with `-fa on`. Meta reports up to 3.1× on an RTX 5090.

---

*Built by [Blackfrost](https://x.com/Blackfrost_AI) · Las Vegas, NV. Not affiliated with Meta.*