---
license: mit
tags: [inference-engine, moe, fp8, cpu, meituan, longcat, longcat-flash-thinking, systems]
---

# dragon.c: A From-Scratch C Inference Engine for **Meituan LongCat-Flash-Thinking** (560B MoE, FP8) on a 31 GB Consumer Laptop

**bespoke ontology — field engineering record, 2026-07-05**

> **📦 Source code available on GitHub:** https://github.com/bespokeontology/dragon.c

![dragon.c launch card](launch_card.png)
*Built and validated in one night on a Dell laptop (RTX 4070 8GB, 31 GB RAM, Ubuntu 24.04), plus a GPU tier the following session: 50× total speedup, ~6 s/token and falling.*

---

## Abstract

We present dragon.c, a from-scratch C inference engine that runs **[Meituan LongCat-Flash-Thinking-FP8](https://huggingface.co/meituan-longcat/LongCat-Flash-Thinking-FP8)**
(560B parameters, Mixture-of-Experts) on a consumer laptop with 31 GB of RAM. The engine reads the
model's native FP8 safetensors directly via mmap — no format conversion, no GGUF, no llama.cpp, no
Python in the hot path — and streams 531 GB of weights through 31 GB of memory using the OS page
cache as the eviction layer. Every mathematical operation was validated brick-by-brick against a
PyTorch reference to floating-point precision, culminating in a full 28-layer forward pass whose
output logits are IDENTICAL to the reference (same top-5 token IDs, logits within ~0.1). The engine
generates coherent English. Through four measured optimization rounds (f32 LRU cache → native-precision compute → AVX2+LUT
kernels → a GPU-resident attention tier on the RTX 4070) decode latency improved 50× to ~6 s/token,
with output invariant across every optimization. We document every dead-end as a control. The build demonstrates
the thesis this project exists to prove: **the deficit in consumer-hardware AI is drivetrain, not
engine** — organization of computation and data movement, not parameter count.

---

## 1. Hardware and model

| | |
|---|---|
| Machine | Dell laptop, Ubuntu 24.04, 31 GB RAM, RTX 4070 Laptop 8 GB, NVMe SSD |
| Model | **Meituan LongCat-Flash-Thinking-FP8** (`meituan-longcat/LongCat-Flash-Thinking-FP8`), 560B MoE, 531 GB on disk, 75 safetensors shards, 43,756 tensors |
| Architecture | 28 decoder layers, each with **2 sub-blocks** (so 56 attention units); hidden 6144; vocab 131,072 |
| Attention | MLA (Multi-head Latent Attention): 64 heads, qk_nope 128 + qk_rope 64 (head dim 192), v 128, q_lora 1536, kv_lora 512 |
| MoE | 512 real experts + 256 zero-compute (identity) experts, top-12, routed_scaling 6.0, expert FFN 2048 |
| ScMoE | Shortcut-connected MoE: the MoE is computed on sub-block-0's normed hidden state but its output is added at the END of sub-block-1 — a designed overlap window |
| Precision | HYBRID: attention/norms/embeddings bf16, router f32, the ScMoE FFNs fp8 — both the routed experts AND the per-sub-block dense branches (e4m3, 128×128 blockwise f32 scales stored IN the shards, not in the index) |

## 2. Method: validate every op, advance only on proof

The build discipline, applied without exception:

1. Dump ground-truth tensors from a working Python/PyTorch reference (`ref/*.f32`).
2. Write the C implementation of ONE operation.
3. Compare element-wise. Advance only when the diff is at floating-point-noise level.
4. Write the result (and every dead-end) into the permanent record before moving on.

This discipline caught real bugs at every stage and — critically — caught a **corrupt reference**
(§6) that would otherwise have burned days.

## 3. Phase results (every brick, with measured error)

| Phase | Operation | Max abs diff vs reference | Notes |
|---|---|---|---|
| 1 | safetensors reader (mmap, u64-LE header JSON, byte offsets) | byte-exact | embed row verified byte-for-byte |
| 2b | RMSNorm (dim 6144, eps 1e-5) | 2.4e-7 | |
| 2c | FP8 e4m3 dequant + 128×128 block scales + matmul | **0.0 (bit-exact)** / 4.3e-7 | dequant is exactly reproducible |
| 2d | MLA attention (full: lora projections, interleave, rope, softmax, 64 heads) | 6.9e-3 | the hardest op; see §4 |
| 2e | MoE (router softmax + bias + top-12 + SwiGLU experts + identity zeros) | 1.68e-6 | |
| 2f | Full ScMoE decoder layer (2 attn blocks + MoE + 2 dense MLPs + 4 norms + shortcut + residuals) | 1.9e-6 | whole layer in C |
| 3 | Full 28-layer forward, native streaming | logits IDENTICAL (top-5 same IDs, ~0.1) | 74 s cold, pure CPU |
| 4 | Generation (chat/think template, vocab decode, KV cache) | output identical across engines | "The user is asking a simple arithmetic question" |

## 4. The three MLA constants nobody tells you about

Hand-deriving MLA from the paper failed (diff 1.7). Reading Meituan's `modeling_longcat_flash.py`
line-by-line revealed three undocumented-in-practice details, without which the attention is wrong:

1. `mla_scale_q_lora = 2.0` — q_pass AND q_rot are multiplied by 2.0.
2. `mla_scale_kv_lora = 3.4641016151377544` (= 2√3) — applied to the kv-lora vector AFTER its layernorm.
3. **Interleave-before-rope** (`use_mla=True` path): before rotary embedding, the rope dims are
   reshaped `[d/2,2]→transpose→[d]` (even indices first, then odd). Softmax scale = 192^(-0.5) =
   0.07216878. k_rot is computed ONCE per token and SHARED across all 64 heads.

With these three, hand-MLA matches the module to 6.9e-3 (bf16 noise). These constants are now
permanent record; they never have to be rediscovered.

## 5. The engine (Phase 3): manifest + mmap + page-cache eviction

Design (the "game-engine pattern" — assets in a PAK file, OS does the paging):

- **Manifest**: Python pre-parses all 75 shard headers ONCE into `manifest.bin` — 43,756 fixed-width
  records (name, shard id, dtype, absolute byte offset, shape, scale-tensor location). C mmaps it
  and bsearches by name. No JSON parsing in C, ever.
- **mmap everything**: all 75 shards are mmap'd read-only at startup. `get_tensor` returns pointers
  into the maps. Weights are NEVER malloc'd/copied (in the final engine — see §8).
- **Eviction = the kernel's page cache.** 531 GB streams through 31 GB because Linux pages weights
  in on touch and evicts them under pressure. Zero eviction code. The OS is the streaming layer.
- **OpenMP** across output rows of every matvec (14 threads).

Result: full 560B forward, 74 s cold. Layer-0 through the native read path matches Python to 2.2e-3.

## 6. The control that saved the build: the reference was poisoned

While validating the full layer, our composition differed from the saved reference by 3.48 — for an
hour this looked like a C bug. It was not. Root cause, found by checking `weight.is_meta`:

> Meituan's router does `self.classifier.weight` (direct attribute access) instead of calling
> `self.classifier(x)`. Our Python-side lazy-loading hook fires on `__call__` — so it NEVER fired
> for the router. The router weight stayed on the meta device, the module's MoE routing was garbage,
> and the garbage was silently baked into the saved reference tensor.

Force-loading the router and regenerating gave a clean reference — which our C matched to 1.9e-6.
**The C was right all along; the reference was corrupt.** Lessons, now permanent controls:

- Never trust a module-derived reference until 0 meta params is verified.
- Any op that touches `.weight` directly bypasses load hooks.
- dragon.c loads every weight explicitly and is structurally immune to this bug class — the
  strongest single argument for owning the engine instead of trusting a framework.

Other controls earned earlier and re-confirmed tonight: the BOS token (id 1) NaN-poisons layer 0
and must never be added; the model is a THINKING model — raw completion is out-of-distribution
(it predicted "Berlin" in the top-5 for "The capital of France is" — capitals-space, correct
neighborhood; clean Python predicts the identical top-5, proving this is model behavior, not an
engine bug); the chat template + `<longcat_think>` is the in-distribution format.

## 7. Generation: the dragon speaks

Tokenizer stays out of C: Python dumps `vocab.bin` (131,072 length-prefixed decoded strings) once;
C prints text directly. Prompt: the official chat template for "What is 2+2?" →
`<longcat_user>What is 2+2? /think_on <longcat_assistant><longcat_think>`.

First generated words from the from-scratch engine:

> **"The user is asking a simple arithmetic question"**

Fluent, on-topic chain-of-thought. No NaN, no gibberish. This sentence has remained byte-identical
across every subsequent engine optimization — it is the correctness canary.

## 8. The speed war: three rounds, each measured, each a lesson

**Round 1 — f32 LRU weight cache + MLA KV cache (dragon_fast.c): 88 s/token. FAILED to speed up.**
The KV cache itself is correct (reproduces recompute output exactly — "The user…" token-for-token)
and kills the O(n²) attention recompute. But the LRU cached weights dequanted to **f32**, which
QUADRUPLES fp8 and DOUBLES bf16: the dense MLPs alone become 51 GB of f32 — cannot fit 31 GB —
41% hit rate — thrash. **Measured diagnosis: the bottleneck was never attention. It is WEIGHT
BANDWIDTH — each token moves ~50 GB of weights.**

**Round 2 — native-precision compute (dragon_native.c, scalar): 27 s/token (3.3×).**
Delete every f32 weight buffer. The matvec reads the mmap'd bf16/fp8 bytes DIRECTLY, dequanting
inline per element. Bytes moved drop 2–4×; the dense MLPs in native bf16 are 25 GB, which FITS —
so the OS page cache holds the hot working set and "load once" happens for free, in hardware.

**Round 3 — the matmul, written properly (AVX2 + LUT): ~9 s/token (10× total), prefill 2.8 s/token.**
- **fp8 via 256-entry lookup table.** e4m3 has exactly 256 possible values. The scalar path was
  paying a `ldexpf` LIBRARY CALL PER WEIGHT BYTE of a 560B model. Precompute all 256 floats once;
  dequant becomes a table read (AVX2 `vgatherdps`, 8 lanes).
- **bf16 via AVX2+FMA.** bf16→f32 is literally a 16-bit left shift — free inside SIMD:
  load 8×u16 → widen → `<<16` → that IS the f32 → `fmadd`. Two accumulators to hide FMA latency.
  8 weights per instruction. The 131,072×6144 lm_head uses the same kernel.
- 128-column block structure of the fp8 scales folds the scale multiply out of the inner loop.

**Round 4 — the GPU lights up (dragon_hybrid = dragon_native.c + dragon_gpu.cu): ~6 s/token (50× total).**
Profiling showed the 9 s spread across attention 2.6 + dense 2.8 + MoE 3.7 + head 0.3 — and that the
box is fundamentally DISK-bound: the fixed per-token working set (~48 GB) exceeds 31 GB RAM, so every
token re-reads tens of GB from NVMe. The correct GPU move on an 8 GB card is therefore NOT "put the
model on the GPU" (impossible) and NOT "stream weights over PCIe" (16 GB/s — the trap): it is
**VRAM as a permanent third cache tier**. We park the attention weights — 42 of 56 sub-blocks,
7.5 GB bf16 — in VRAM at startup and never move them again. A warp-per-row bf16 matvec kernel
(one warp reduces one output row; `__shfl_down_sync` for the horizontal sum) computes attention
on-card; sub-blocks that don't fit fall back to the CPU path transparently. The win is DOUBLE:
attention drops 2.6 s → 0.5 s (5×), AND 7.5 GB leaves the RAM working set, so the page cache holds
more of the dense/expert tiers and disk reads shrink across the board.

| engine | s/token | change |
|---|---|---|
| recompute (dragon_gen) | ~300+ | correct but O(n²), no caches |
| f32 LRU cache (dragon_fast) | 88 | KV cache ✓; f32 weight cache thrashes |
| native scalar (dragon_native) | 27 | zero f32 buffers; page cache = load-once |
| native AVX2+LUT | ~9 | the matmul |
| **+ GPU attention tier (dragon_hybrid)** | **~6, falling** | **VRAM = 3rd cache tier; attn 5×** |

Output identical at every step — the generated sentence has never changed through a 50× speedup.
Correctness never traded for speed.

## 9. What this proves (the thesis, in working code)

1. **Real answers = the war won.** The existential question — can a 560B run correctly on owned
   consumer hardware with owned code — is closed. Everything remaining is tuning, not truth.
2. **The ScMoE shortcut is an owned moat.** The architecture computes the MoE on block-0's output
   and applies it at block-1's end — a designed overlap window. Generic runtimes don't exploit it;
   an owned engine can compute MoE ∥ attention and hide expert-streaming latency inside attention
   math. This optimization is only accessible to someone who OWNS the engine.
3. **C, not Python — the game-engine discipline.** No game runs Python in the frame loop; inference
   IS a frame loop (tight, memory-bound, latency-critical). Python found the bugs (prototype);
   C runs the frame (engine). The Python stack's own flakiness (silent meta-weights, wedged
   processes, OOM kills) was demonstrated during this build and is documented above.
4. **Bandwidth is the drivetrain.** Every real speedup came from moving fewer bytes or moving them
   smarter (native precision, page cache, SIMD) — never from more FLOPs. The datacenter's answer
   to this problem is to buy more bandwidth. The drivetrain answer is to stop wasting it.

## 10. Remaining work (all de-risked, all speed/usability)

- **fp8-quantize the dense tier**: 25 GB bf16 → 12.7 GB with our own (bit-exact-validated) fp8
  path — brings the fixed working set under RAM so disk only ever reads experts. Largest remaining lever.
- **ScMoE overlap**: thread the MoE against sub-block-1's attention (the moat, §9.2).
- **Compressed-latent KV cache**: store the 576-dim latent instead of expanded K/V for long
  context (matters beyond our current 64-token horizon).
- Sampling (currently greedy argmax), longer rope table, incremental `vocab.bin` mmap.

## Artifacts here — full source included

- `src/dragon_hybrid.c` + `src/dragon_gpu.cu` — **the engine** (CPU AVX2/LUT + GPU-resident attention tier)
- `src/dragon_native.c` — CPU-only engine (no CUDA required)
- `src/dragon_forward.c` — the proven full-forward validator
- `src/dragon_read.c`, `src/phase2*.c` — the six validation bricks (reader, RMSNorm, fp8, MLA, MoE, full layer)
- `tools/build_manifest.py` + `Makefile` — `make dragon` / `make dragon-gpu`
- `launch_card.png` (the launch card) · `dragon_speaks_card.png` (the historic first-words shot) · `manifest.bin` (the 43,756-tensor index)

```
python3 tools/build_manifest.py /path/to/LongCat-Flash-Thinking-FP8 .
make dragon        # CPU-only
make dragon-gpu    # with the RTX 4070 attention tier
./dragon 16
```

*— [kapitoshkina.ai](https://kapitoshkina.ai) · bespoke ontology, 2026-07-05. This is the beginning.*