# llama.cpp Integration Patches for Fuse3

This file describes the changes needed to integrate Fuse3 into llama.cpp.
Apply these patches to a fork of llama.cpp.

## 1. `gguf-py/gguf/constants.py`

Add `FUSE3` to the `MODEL_ARCH` enum and the `MODEL_ARCH_NAMES` / `MODEL_TENSORS` mappings.

```python
# In MODEL_ARCH enum:
class MODEL_ARCH(IntEnum):
    ...
    FUSE3 = auto()  # Add after LFM2MOE

# In MODEL_ARCH_NAMES:
MODEL_ARCH_NAMES = {
    ...
    gguf.MODEL_ARCH.FUSE3: "fuse3",
}

# In MODEL_TENSORS:
MODEL_TENSORS = {
    ...
    gguf.MODEL_ARCH.FUSE3: {
        # Reuse all LFM2 tensors, plus Fuse3-specific ones
        gguf.MODEL_TENSOR.TOKEN_EMBD,
        gguf.MODEL_TENSOR.OUTPUT_NORM_LFM2,
        gguf.MODEL_TENSOR.OUTPUT,
        gguf.MODEL_TENSOR.ATTN_NORM,
        gguf.MODEL_TENSOR.ATTN_Q,
        gguf.MODEL_TENSOR.ATTN_K,
        gguf.MODEL_TENSOR.ATTN_V,
        gguf.MODEL_TENSOR.ATTN_Q_NORM,
        gguf.MODEL_TENSOR.ATTN_K_NORM,
        gguf.MODEL_TENSOR.ATTN_OUT,
        gguf.MODEL_TENSOR.FFN_NORM,
        gguf.MODEL_TENSOR.FFN_GATE,
        gguf.MODEL_TENSOR.FFN_DOWN,
        gguf.MODEL_TENSOR.FFN_UP,
        gguf.MODEL_TENSOR.SHORTCONV_CONV,
        gguf.MODEL_TENSOR.SHORTCONV_INPROJ,
        gguf.MODEL_TENSOR.SHORTCONV_OUTPROJ,
    },
}
```

## 2. `src/llama-arch.h`

Add `LLM_ARCH_FUSE3` to the `llm_arch` enum:

```cpp
enum llm_arch {
    LLM_ARCH_LLAMA,
    ...
    LLM_ARCH_LFM2,
    LLM_ARCH_LFM2MOE,
    LLM_ARCH_FUSE3,  // Add here
    ...
};
```

## 3. `src/llama-arch.cpp`

Register the architecture name and tensor mappings:

```cpp
// In LLM_ARCH_NAMES:
{ LLM_ARCH_FUSE3, "fuse3" },

// In LLM_TENSOR_NAMES for FUSE3:
{
    LLM_ARCH_FUSE3,
    {
        { LLM_TENSOR_TOKEN_EMBD,      "token_embd.weight" },
        { LLM_TENSOR_OUTPUT_NORM_LFM2, "token_embd_norm.weight" },
        { LLM_TENSOR_OUTPUT,          "output.weight" },
        { LLM_TENSOR_ATTN_NORM,       "blk.{bid}.attn_norm.weight" },
        { LLM_TENSOR_ATTN_Q,          "blk.{bid}.attn_q.weight" },
        { LLM_TENSOR_ATTN_K,          "blk.{bid}.attn_k.weight" },
        { LLM_TENSOR_ATTN_V,          "blk.{bid}.attn_v.weight" },
        { LLM_TENSOR_ATTN_Q_NORM,     "blk.{bid}.attn_q_norm.weight" },
        { LLM_TENSOR_ATTN_K_NORM,     "blk.{bid}.attn_k_norm.weight" },
        { LLM_TENSOR_ATTN_OUT,        "blk.{bid}.attn_output.weight" },
        { LLM_TENSOR_FFN_NORM,        "blk.{bid}.ffn_norm.weight" },
        { LLM_TENSOR_FFN_GATE,        "blk.{bid}.ffn_gate.weight" },
        { LLM_TENSOR_FFN_DOWN,        "blk.{bid}.ffn_down.weight" },
        { LLM_TENSOR_FFN_UP,          "blk.{bid}.ffn_up.weight" },
        { LLM_TENSOR_SHORTCONV_CONV,    "blk.{bid}.shortconv_conv.weight" },
        { LLM_TENSOR_SHORTCONV_INPROJ,  "blk.{bid}.shortconv_inproj.weight" },
        { LLM_TENSOR_SHORTCONV_OUTPROJ, "blk.{bid}.shortconv_outproj.weight" },
    }
},
```

## 4. `src/llama-model.cpp`

Add dispatch entries for Fuse3:

```cpp
// In llama_model::load_arch_hparams():
case LLM_ARCH_FUSE3:
    using impl = llama_model_fuse3;
    impl::load_arch_hparams(ml);
    break;

// In llama_model::load_arch_tensors():
case LLM_ARCH_FUSE3:
    using impl = llama_model_fuse3;
    impl::load_arch_tensors(ml);
    break;

// In llama_model::build_graph():
case LLM_ARCH_FUSE3:
    using impl = llama_model_fuse3;
    ctx = impl::build_arch_graph(params);
    break;
```

## 5. `src/CMakeLists.txt`

Add the Fuse3 source file:

```cmake
set(LLAMA_MODEL_SOURCES
    ...
    models/lfm2.cpp
    models/lfm2moe.cpp
    models/fuse3.cpp  # Add here
    ...
)
```

## 6. `src/models/models.h`

Include the Fuse3 header:

```cpp
#include "lfm2.h"
#include "lfm2moe.h"
#include "fuse3.h"  // Add here
```

## 7. Custom GGUF Keys

The Fuse3 converter writes these custom keys that need to be read by the C++ code:

- `fuse3.augmented_layers` — array of layer indices that have experts
- `fuse3.expert_counts` — array of expert counts per layer
- `fuse3.swiglu_limit` — float, clamp value for expert activations (default 10.0)
- `fuse3.expert_scale_init` — float, initial scale value (not used at inference, trained value is in weights)

These are read in `llama_model_fuse3::load_arch_hparams()`.

## 8. Custom Tensor Names

The Fuse3 converter writes these custom tensor names:

- `blk.{i}.fuse3_router.weight` — router gate weights `{n_embd, n_exp}`
- `blk.{i}.fuse3_expert_scale.weight` — expert scale parameter `{1}`
- `blk.{i}.fuse3_experts.gate.weight` — stacked expert gate weights `{n_embd, n_ff_exp, n_exp}`
- `blk.{i}.fuse3_experts.up.weight` — stacked expert up weights `{n_embd, n_ff_exp, n_exp}`
- `blk.{i}.fuse3_experts.down.weight` — stacked expert down weights `{n_ff_exp, n_embd, n_exp}`

These are loaded in `llama_model_fuse3::load_arch_tensors()`.