File size: 3,609 Bytes
bdbceda | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | # Fuse3 llama.cpp Integration
Full llama.cpp support for Fuse3 (fuse-1 Lite) — LFM2 host + Qwen3.6 coding experts.
## Files
| File | Purpose |
|------|---------|
| `fuse3.cpp` | C++ graph builder (place in `src/models/`) |
| `fuse3.h` | C++ header (place in `src/models/`) |
| `fuse3_converter.py` | Python GGUF converter (place in `conversion/`) |
| `export_gguf.py` | Standalone GGUF exporter (no llama.cpp fork needed to create GGUF) |
| `INTEGRATION.md` | Step-by-step patches for llama.cpp source files |
## Quick Start
### 1. Create the GGUF file (no fork needed)
```bash
pip install gguf safetensors torch
python export_gguf.py --model-dir /path/to/fuse-1-Lite --output fuse-1-Lite-f16.gguf --outtype f16
```
### 2. Build llama.cpp with Fuse3 support
```bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
# Apply patches (see INTEGRATION.md)
# Copy fuse3.cpp and fuse3.h to src/models/
# Copy fuse3_converter.py to conversion/
# Modify src/llama-arch.h, src/llama-arch.cpp, src/llama-model.cpp, src/CMakeLists.txt
# Build
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
```
### 3. Run inference
```bash
./build/bin/llama-cli \
-m fuse-1-Lite-f16.gguf \
-p "Write a Python function to check if a number is prime." \
-n 512 --temp 0.1
```
## Architecture
Fuse3 extends llama.cpp's native LFM2 implementation:
- **Host layers**: Reuses LFM2's attention (`Lfm2AttentionDecoderLayer`) and short-conv (`Lfm2ShortConvDecoderLayer`) graph builders
- **Dense FFN**: Standard LFM2 SwiGLU FFN (always dense, not MoE)
- **Expert augmentation**: After each augmented layer's FFN, a custom MoE block runs:
1. Router: `sqrtsoftplus(x @ router_weight)` → top-k selection → normalize
2. Experts: Per-expert SwiGLU FFN (gate, up, down projections)
3. Scale: `softplus(expert_scale)` clamped to 0.1
4. Add: `host_output + scale * expert_output`
## GGUF Tensor Names
### Standard LFM2 tensors (host)
- `token_embd.weight`, `token_embd_norm.weight`, `output.weight`
- `blk.{i}.attn_norm.weight`, `blk.{i}.attn_q/k/v.weight`, `blk.{i}.attn_output.weight`
- `blk.{i}.attn_q_norm.weight`, `blk.{i}.attn_k_norm.weight`
- `blk.{i}.ffn_norm.weight`, `blk.{i}.ffn_gate/up/down.weight`
- `blk.{i}.shortconv_conv/inproj/outproj.weight`
### Fuse3-specific tensors
- `blk.{i}.fuse3_router.weight` — `{n_embd, n_exp}` router gate
- `blk.{i}.fuse3_expert_scale.weight` — `{1}` learned scale parameter
- `blk.{i}.fuse3_experts.gate.weight` — `{n_embd, n_ff_exp, n_exp}` stacked expert gates
- `blk.{i}.fuse3_experts.up.weight` — `{n_embd, n_ff_exp, n_exp}` stacked expert up projections
- `blk.{i}.fuse3_experts.down.weight` — `{n_ff_exp, n_embd, n_exp}` stacked expert down projections
### Custom GGUF metadata keys
- `fuse3.augmented_layers` — array of layer indices with experts
- `fuse3.expert_counts` — per-layer expert count array
- `fuse3.swiglu_limit` — float, expert activation clamp (default 10.0)
- `fuse3.expert_scale_init` — float, initial scale (not used at inference)
## Quantization
The GGUF file can be quantized using llama.cpp's standard quantization tools:
```bash
# Q4_K_M (recommended)
./build/bin/llama-quantize fuse-1-Lite-f16.gguf fuse-1-Lite-q4_k_m.gguf q4_k_m
# Q8_0
./build/bin/llama-quantize fuse-1-Lite-f16.gguf fuse-1-Lite-q8_0.gguf q8_0
```
Note: Only the host LFM2 weights are quantized. The expert weights are kept in f16
for quality (they're small — 512 intermediate × ~30 experts × 30 layers ≈ 200MB).
|