README.md

---

language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - mixture-of-experts
  - MoE
  - coding
  - python
  - code-generation
  - LFM2
  - Qwen
  - LiquidAI
  - small-language-model
  - SLM
  - agentic
  - fusion
  - expert-routing
  - 5B
  - efficient-inference
base_model:
  - LiquidAI/LFM2.5-2.6B
  - Qwen/Qwen3.6-35B-A3B
pipeline_tag: text-generation
inference: false
model-index:
  - name: fuse-1 Lite
    results:
      - task:
          type: text-generation
          name: Code Generation
        dataset:
          name: HumanEval (style)
          type: openai_humaneval
        metrics:
          - type: pass@1
            value: "TBD"
            name: pass@1
---


# fuse-1 Lite — Coding-Enhanced Mixture-of-Experts (5.72B)

> **A 5.72B parameter Mixture-of-Experts model that fuses LiquidAI's LFM2.5-2.6B host with 960 coding experts extracted from Qwen3.6-35B-A3B. Designed for efficient coding assistance, agentic workflows, and on-device inference.**

## Overview

**fuse-1 Lite** is a novel fusion model that combines the speed and efficiency of a small language model (LFM2.5-2.6B, 2.70B params) with the coding expertise of a large MoE model (Qwen3.6-35B-A3B). Rather than distilling knowledge or fine-tuning from scratch, fuse-1 Lite **transplants actual expert weights** from the donor model and trains a lightweight router to selectively activate them only when coding-related tokens are encountered.

### Key Innovation

Traditional model fusion requires either:
1. **Knowledge distillation** (slow, lossy, requires teacher inference)
2. **Weight merging** (requires compatible architectures)
3. **Full fine-tuning** (expensive, risks catastrophic forgetting)

fuse-1 Lite takes a different approach: **surgical expert transplantation with learned routing**. The 960 coding-specialized experts from Qwen3.6-35B-A3B are extracted, normalized, and integrated as residual augmentations to LFM2.5's decoder layers. A per-layer router learns which experts to activate for each token, and a learned scale factor controls how much each layer's experts contribute.

## Architecture

| Component | Details |
|-----------|---------|
| **Host Model** | LiquidAI/LFM2.5-2.6B (30 layers, hidden_size=2048) |

| **Donor Model** | Qwen/Qwen3.6-35B-A3B (MoE, 35B params) |

| **Total Parameters** | 5.72B |

| **Host Parameters** | 2.70B (frozen) |

| **Expert Parameters** | 3.02B (frozen, from Qwen3.6) |

| **Trainable Parameters** | 2.0M (router + scale only) |

| **Expert Count** | 960 across 30 layers (32 per layer) |

| **Top-K Routing** | 8 experts per token |

| **Expert Intermediate Size** | 512 |

| **Precision** | bfloat16 |



### How It Works



```

Input Token

    │

    ▼

┌─────────────────────────────────────────┐

│  LFM2.5 Decoder Layer (frozen host)     │

│  ├── Attention / ShortConv              │

│  └── SwiGLU FFN                         │

└─────────────┬───────────────────────────┘

              │

              ▼

┌─────────────────────────────────────────┐

│  Fuse3 Augmented Layer                  │

│  1. Router scores token → top-8 experts │

│  2. Selected experts compute SwiGLU     │

│  3. Output normalized to host std       │

│  4. Scaled by learned expert_scale      │
│  5. Added to residual stream            │
└─────────────────────────────────────────┘
```



### Expert Activation Pattern



After training, the router learned to selectively activate experts in **11 of 30 layers**:



| Layer | Scale | Role |

|-------|-------|------|

| 0 | 2.80 | Token-level feature extraction |

| 10-11 | 3.83-4.44 | Mid-level code structure |

| 13-15 | 4.56-4.84 | Algorithmic reasoning |

| 17 | 4.09 | Logic flow |

| 19 | 6.66 | **Peak coding expertise** |

| 21 | 4.94 | Code synthesis |

| 23 | 3.28 | Output formatting |

| 26 | 4.31 | Final code refinement |



The remaining 19 layers have scale ≈ 0 (experts effectively disabled), preserving LFM2's general language capabilities.



## Training Details



| Parameter | Value |

|-----------|-------|

| **Training Steps** | 300 |

| **Training Time** | 8.3 minutes (Modal L4 GPU) |

| **Router Learning Rate** | 1e-3 |

| **Scale Learning Rate** | 1.0 |

| **Loss** | LM loss + 0.01 × Load Balancing loss |

| **Optimizer** | AdamW |

| **Gradient Accumulation** | 4 |

| **Max Sequence Length** | 512 |

| **Training Data** | 55 examples (40 coding + 15 general) |

| **Total Compute Cost** | ~$3 (all phases combined) |



### Training Phases



1. **Phase 1 — Expert Profiling** ($1.90, A100 80GB): Profile Qwen3.6-35B-A3B expert activations on coding vs non-coding prompts. Select 960 coding-specialized experts. Extract weights.



2. **Phase 2 — Assembly** ($0.50, L4): Load LFM2.5-2.6B, wrap decoder layers with Fuse3AugmentedLayer, load extracted expert weights. Verify base model output coherence.



3. **Phase 3 — Router Training** ($0.60, L4): Train router + expert_scale on coding and general examples. Router learns to activate experts only for coding-related tokens.



### Stability Mechanisms



- **Std normalization**: Expert outputs are rescaled to match host layer activation std (ratio clamped to 2.0)

- **Scale clamping**: Expert scale clamped to max=0.1 during forward pass

- **SwiGLU clamping**: Expert intermediate activations clamped to [-10, 10]

- **Frozen host + experts**: Only 2.0M parameters are trainable (router weights + per-layer scale)



## Usage



### Installation



```bash

pip install transformers torch

```

### Quick Start

```python

from transformers import AutoTokenizer, AutoModelForCausalLM

import torch



model_id = "Akahsizrr/fuse-1-Lite"



tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(

    model_id,

    torch_dtype=torch.bfloat16,

    device_map="auto",

    trust_remote_code=True,

)



messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(text, return_tensors="pt").to(model.device)



with torch.no_grad():

    outputs = model.generate(

        **inputs,

        max_new_tokens=1024,

        do_sample=True,

        temperature=0.1,

        top_k=50,

        repetition_penalty=1.1,

        pad_token_id=tokenizer.pad_token_id,

    )



response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)

print(response)

```

## Quantization & Deployment

### Pre-quantized Versions

| Version | Repo | VRAM/Memory | Format |
|---------|------|-------------|--------|
| **4-bit NF4** | [`Akahsizrr/fuse-1-Lite-4bit`](https://huggingface.co/Akahsizrr/fuse-1-Lite-4bit) | 3.36 GB | bitsandbytes |
| **8-bit** | [`Akahsizrr/fuse-1-Lite-8bit`](https://huggingface.co/Akahsizrr/fuse-1-Lite-8bit) | 6.00 GB | bitsandbytes |
| **bfloat16** | This repo | ~12 GB | safetensors |
| **MLX** | [`Akahsizrr/fuse-1-Lite-MLX`](https://huggingface.co/Akahsizrr/fuse-1-Lite-MLX) | ~12 GB | MLX safetensors |
| **GGUF F16** | [`Akahsizrr/fuse-1-Lite-GGUF`](https://huggingface.co/Akahsizrr/fuse-1-Lite-GGUF) | ~11.4 GB | GGUF |
| **vLLM plugin** | [`Akahsizrr/fuse-1-Lite-vLLM`](https://huggingface.co/Akahsizrr/fuse-1-Lite-vLLM) | ~12 GB | vLLM plugin |

### bitsandbytes 4-bit (NF4) — 3.36 GB VRAM

```python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

import torch



bnb_config = BitsAndBytesConfig(

    load_in_4bit=True,

    bnb_4bit_quant_type="nf4",

    bnb_4bit_compute_dtype=torch.bfloat16,

    bnb_4bit_use_double_quant=True,

)



model = AutoModelForCausalLM.from_pretrained(

    "Akahsizrr/fuse-1-Lite",

    quantization_config=bnb_config,

    device_map="auto",

    trust_remote_code=True,

)

tokenizer = AutoTokenizer.from_pretrained("Akahsizrr/fuse-1-Lite")

```

### bitsandbytes 8-bit — 6.00 GB VRAM

```python

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

import torch



bnb_config = BitsAndBytesConfig(load_in_8bit=True)



model = AutoModelForCausalLM.from_pretrained(

    "Akahsizrr/fuse-1-Lite",

    quantization_config=bnb_config,

    device_map="auto",

    trust_remote_code=True,

)

```

### Disable Coding Experts (Pure LFM2 Mode)

```python

# The model includes a coding toggle — disable experts for pure LFM2 inference

model.set_coding_enabled(False)



# Re-enable for coding tasks

model.set_coding_enabled(True)

```

### vLLM — High-Throughput Serving

fuse-1 Lite is supported in vLLM via a plugin that extends vLLM's native LFM2
implementation with expert augmentation layers.

**Plugin repo**: [`Akahsizrr/fuse-1-Lite-vLLM`](https://huggingface.co/Akahsizrr/fuse-1-Lite-vLLM)

```bash

# Install the plugin

pip install git+https://huggingface.co/Akahsizrr/fuse-1-Lite-vLLM



# Serve with vLLM

vllm serve Akahsizrr/fuse-1-Lite \

  --mamba-cache-mode align \

  --max-model-len 4096

```

```python

from vllm import LLM



llm = LLM(

    model="Akahsizrr/fuse-1-Lite",

    mamba_cache_mode="align",

    max_model_len=4096,

)

output = llm.generate("Write a Python function to check if a number is prime.")

```

The plugin registers `Fuse3ForCausalLM` with vLLM's `ModelRegistry` via the
`vllm.general_plugins` entry point. It reuses vLLM's native LFM2 attention
and short-conv layers, adding the expert MoE block after each augmented
layer's FFN.

### MLX (Apple Silicon)

fuse-1 Lite is available in MLX format for Apple Silicon (M1+).

**MLX repo**: [`Akahsizrr/fuse-1-Lite-MLX`](https://huggingface.co/Akahsizrr/fuse-1-Lite-MLX)

```python

from mlx_lm import load, generate



model, tokenizer = load("Akahsizrr/fuse-1-Lite-MLX", trust_remote_code=True)



prompt = tokenizer.apply_chat_template(

    [{"role": "user", "content": "Write a Python function to check if a number is prime."}],

    tokenize=False, add_generation_prompt=True,

)



response = generate(model, tokenizer, prompt=prompt, max_tokens=512)

print(response)

```

```bash

# CLI

mlx_lm.generate --model Akahsizrr/fuse-1-Lite-MLX --trust-remote-code --prompt "Write a Python fizzbuzz"

```

The MLX model file (`fuse3_mlx.py`) extends MLX's native LFM2 implementation
with the same expert MoE augmentation. It uses `model_file` in config.json
with `trust_remote_code=True` for loading.

### GGUF / llama.cpp

fuse-1 Lite is available in GGUF format for llama.cpp.

**GGUF repo**: [`Akahsizrr/fuse-1-Lite-GGUF`](https://huggingface.co/Akahsizrr/fuse-1-Lite-GGUF)

> **Note:** The GGUF uses the custom `fuse3` architecture. Stock llama.cpp
> cannot load it — you need a llama.cpp fork with Fuse3 support. The GGUF
> repo includes the C++ graph builder (`src/models/fuse3.cpp`), Python
> converter (`conversion/fuse3.py`), and integration guide (`INTEGRATION.md`).

```bash

# Build llama.cpp with Fuse3 support (see INTEGRATION.md in the GGUF repo)

./llama-cli -m fuse-1-Lite-f16.gguf \

  -p "Write a Python function to check if a number is prime." \

  -n 512 --temp 0.1

```

The C++ implementation reuses LFM2's attention and short-conv graph builders,
adding the expert MoE block (router → top-k → SwiGLU experts → scale → add)
after each augmented layer's dense FFN.

### Transformers (Universal)

The recommended way to run fuse-1 Lite on any platform:

```bash

pip install transformers torch bitsandbytes accelerate

```

## Performance

### VRAM Requirements

| Backend | Precision | VRAM/Memory | Recommended Hardware |
|---------|-----------|-------------|---------------------|
| Transformers | bfloat16 | ~12 GB | L4, A10G, RTX 4090 |
| Transformers | 8-bit | 6.00 GB | T4, L4, RTX 3060 |
| Transformers | 4-bit | 3.36 GB | T4, RTX 3060, M2 Pro |
| vLLM | bfloat16 | ~12 GB | A10G, A100, H100 |
| MLX | float16 | ~12 GB | M1 Pro+, M2, M3, M4 |
| llama.cpp | F16 | ~11.4 GB | Any CPU/GPU |
| llama.cpp | Q4_K_M | ~4 GB | Any CPU/GPU |

### Sample Outputs

**Prompt**: "Write a Python function to check if a string is a palindrome."

**Output** (excerpt):
```python

def is_palindrome(s: str) -> bool:

    """

    Return True if *s* reads the same forwards and backwards,

    ignoring case and non-alphanumeric characters.

    """

    cleaned = ''.join(ch.lower() for ch in s if ch.isalnum())

    return cleaned == cleaned[::-1]

```

The model produces complete implementations with docstrings, type hints, complexity analysis, and test cases.

## Limitations

1. **Custom architecture**: Requires `trust_remote_code=True` — the model includes custom `Fuse3ForCausalLM` code
2. **No vLLM support**: The custom MoE augmentation is not yet supported by vLLM's optimized inference engine
3. **No GGUF/MLX conversion**: The custom architecture cannot be directly converted to GGUF or MLX format
4. **Training data was small**: Only 55 examples were used for router training — the router may not generalize perfectly to all coding tasks
5. **Expert compatibility**: Qwen3.6 experts operate on LFM2's activation space with std normalization — some expert knowledge may be lost in translation
6. **`use_cache=False` during training**: Augmented layers don't propagate KV cache correctly during training; generation uses the standard cache

7. **bitsandbytes quantization**: 4-bit and 8-bit quantization work at runtime via `BitsAndBytesConfig` — pre-quantized saved versions are not available as separate repos



## Citation



```bibtex

@misc{fuse1lite2026,

  title={fuse-1 Lite: Coding-Enhanced Mixture-of-Experts via Expert Transplantation},

  author={Vasko Djack},

  year={2026},

  publisher={HuggingFace},

  url={https://huggingface.co/Akahsizrr/fuse-1-Lite}

}

```



## Acknowledgments



- **LiquidAI** for the LFM2.5-2.6B host model

- **Qwen Team** for the Qwen3.6-35B-A3B donor model

- **Modal** for compute infrastructure



## License



Apache 2.0 — See [LICENSE](LICENSE) for details.



---

*Built with [Devin](https://devin.ai) — Cognition AI*