File size: 4,567 Bytes
51cf6c8 98a03d3 51cf6c8 650c9a5 a4d873c 4dffb6f 211d684 a4d873c 9cdffa9 650c9a5 c0edc4f 6dc5c65 c0edc4f 8a18b58 c0edc4f 6dc5c65 650c9a5 00685de 650c9a5 00685de 650c9a5 00685de 650c9a5 560d592 650c9a5 6dc5c65 650c9a5 a4d873c 6df5abc 650c9a5 a4d873c 650c9a5 d19d319 650c9a5 d19d319 650c9a5 d19d319 650c9a5 d19d319 650c9a5 d19d319 6df5abc 650c9a5 2480254 211d684 d19d319 6dc5c65 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | ---
license: mit
base_model:
- inclusionAI/Ling-3.0-flash
---
## Compatibility
⚠️ Ling-3.0-flash uses the new `bailingmoe3` GGUF architecture. While waiting on upstream support, use the following fork:
https://github.com/aetherbird/llama.cpp/tree/bailingmoe3-support
Stock llama.cpp builds without bailingmoe3 support will not load the model.
### ⚠️ Correction in progress
Earlier GGUF revisions omitted Ling 3.0's trained SwiGLU clamp metadata. Please wait for the completion notice before downloading;
a combined metadata and template audit is underway.
Existing files will only need redownloading or [local repair](./add-ling3-clamp-metadata.py), not requantization. llama.cpp fix:
[`c51308d8`](https://github.com/aetherbird/llama.cpp/commit/c51308d8).
## Conversion and Quantization
Taken directly from the released `inclusionAI/Ling-3.0-flash` BF16 safetensors.
Conversion-specific tensor transformations include:
- `A_log` stored as `exp(A_log)`
- MLA `kv_b_proj` split into separate K and V tensors, with the K tensor transposed
- KDA convolution weights reshaped for llama.cpp
- Per-expert tensors stacked into GGUF expert tensors
- KDA and MLA `g_proj` tensors mapped separately
Norms, routing tensors, expert routing bias, KDA state scalars, `dt_bias`, and convolution weights remain F32.
### Importance Matrix
Importance matrix generated from the Q8_0 model:
- `wiki.train.raw`
- 100 chunks
- 512 tokens per chunk
- 51,200 calibration tokens total
- 573 matrix entries
### Quants
`MXFP4_MOE`:
- Quantized using llama.cpp's MXFP4_MOE quantization type
`Q8_0`:
- 8.51 BPW
- 126.3 GiB
- Includes MTP block
`UD-Q2_K_XL`:
- Model-specific Unsloth-style mixed tensor recipe
- Main expert gate/up tensors: IQ2_XS
- Main expert down tensors: IQ3_XXS
- Final target layer experts: IQ3_XXS and IQ4_XS
- Attention, shared experts, and KDA projections retained at higher precision
- MTP experts: Q3_K and Q4_K
`IQ1_S`:
- Expected size: approximately 24.9 GiB
- Preserves MTP functionality
## Notes
The GGUF contains 43 blocks:
- 42 target-model layers
- 35 KDA layers
- 7 gated MLA layers at zero-based indices 5, 11, 17, 23, 29, 35, and 41
- One MTP/NextN block at index 42
The first two target layers use dense FFNs. The remaining target layers use 512 routed experts with top-8 selection plus one shared
expert. Routing uses sigmoid scoring, expert bias, eight expert groups, and four selected groups.
The KDA safe gate is implemented as:
`lower_bound * sigmoid(exp(A_log) * (f_proj(x) + dt_bias))`
The lower bound is `-5.0`. The GGUF stores the positive `exp(A_log)` value, while the sign is supplied by the negative lower bound.
## MTP Support
The MTP block is bundled inside every GGUF.
During ordinary inference, llama.cpp skips the MTP tensors and may report them as unused. They occupy disk space but are not loaded
into the ordinary target-model buffer.
With `--spec-type draft-mtp`, the same GGUF is opened as an MTP draft model and block 42 is loaded and executed. No separate drafter
file is required.
## Validation Completed
- BF16 architecture load and tensor round-trip
- CPU and CUDA execution on a reduced-size BailingMoE3 fixture
- Target next-token parity against the released Hugging Face implementation before the missing trained clamps were identified
- Nonzero SwiGLU clamp execution and GGUF round-trip on the reduced-size BailingMoE3 fixture
- First three recursive MTP proposals matched the Hugging Face implementation
- Full MXFP4_MOE target and MTP graph smoke test
- Q8_0 conversion completed successfully with all 938 tensors
## Build
```bash
git clone --branch bailingmoe3-support https://github.com/aetherbird/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --target llama-cli llama-server
```
## Usage
```
./build/bin/llama-server \
-m Ling-3.0-flash-Q8_0.gguf \
-c 131072 \
-ngl auto \
--flash-attn auto
```
Enable the bundled MTP drafter:
```
./build/bin/llama-server \
-m Ling-3.0-flash-Q8_0.gguf \
-c 131072 \
-ngl auto \
--flash-attn auto \
--spec-type draft-mtp
```
MoE placement can be adjusted for available VRAM with -ncmoe N. Draft-model placement can be controlled separately with -ncmoed N and -ngld N.
Supports up to 256K context. Reasoning is enabled by default.
Upstream PR:
https://github.com/ggml-org/llama.cpp/pull/26608
|