README.md

metadata

language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - mixture-of-experts
  - MoE
  - coding
  - python
  - code-generation
  - LFM2
  - Qwen
  - LiquidAI
  - small-language-model
  - SLM
  - agentic
  - fusion
  - expert-routing
  - 5B
  - efficient-inference
base_model:
  - LiquidAI/LFM2.5-2.6B
  - Qwen/Qwen3.6-35B-A3B
pipeline_tag: text-generation
inference: false
model-index:
  - name: fuse-1 Lite
    results:
      - task:
          type: text-generation
          name: Code Generation
        dataset:
          name: HumanEval (style)
          type: openai_humaneval
        metrics:
          - type: pass@1
            value: TBD
            name: pass@1

fuse-1 Lite — Coding-Enhanced Mixture-of-Experts (5.72B)

A 5.72B parameter Mixture-of-Experts model that fuses LiquidAI's LFM2.5-2.6B host with 960 coding experts extracted from Qwen3.6-35B-A3B. Designed for efficient coding assistance, agentic workflows, and on-device inference.

Overview

fuse-1 Lite is a novel fusion model that combines the speed and efficiency of a small language model (LFM2.5-2.6B, 2.70B params) with the coding expertise of a large MoE model (Qwen3.6-35B-A3B). Rather than distilling knowledge or fine-tuning from scratch, fuse-1 Lite transplants actual expert weights from the donor model and trains a lightweight router to selectively activate them only when coding-related tokens are encountered.

Key Innovation

Traditional model fusion requires either:

1. Knowledge distillation (slow, lossy, requires teacher inference)
2. Weight merging (requires compatible architectures)
3. Full fine-tuning (expensive, risks catastrophic forgetting)

fuse-1 Lite takes a different approach: surgical expert transplantation with learned routing. The 960 coding-specialized experts from Qwen3.6-35B-A3B are extracted, normalized, and integrated as residual augmentations to LFM2.5's decoder layers. A per-layer router learns which experts to activate for each token, and a learned scale factor controls how much each layer's experts contribute.

Architecture

Component	Details
Host Model	LiquidAI/LFM2.5-2.6B (30 layers, hidden_size=2048)
Donor Model	Qwen/Qwen3.6-35B-A3B (MoE, 35B params)
Total Parameters	5.72B
Host Parameters	2.70B (frozen)
Expert Parameters	3.02B (frozen, from Qwen3.6)
Trainable Parameters	2.0M (router + scale only)
Expert Count	960 across 30 layers (32 per layer)
Top-K Routing	8 experts per token
Expert Intermediate Size	512
Precision	bfloat16

How It Works

Input Token
    │
    ▼
┌─────────────────────────────────────────┐
│  LFM2.5 Decoder Layer (frozen host)     │
│  ├── Attention / ShortConv              │
│  └── SwiGLU FFN                         │
└─────────────┬───────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────┐
│  Fuse3 Augmented Layer                  │
│  1. Router scores token → top-8 experts │
│  2. Selected experts compute SwiGLU     │
│  3. Output normalized to host std       │
│  4. Scaled by learned expert_scale      │
│  5. Added to residual stream            │
└─────────────────────────────────────────┘

Expert Activation Pattern

After training, the router learned to selectively activate experts in 11 of 30 layers:

Layer	Scale	Role
0	2.80	Token-level feature extraction
10-11	3.83-4.44	Mid-level code structure
13-15	4.56-4.84	Algorithmic reasoning
17	4.09	Logic flow
19	6.66	Peak coding expertise
21	4.94	Code synthesis
23	3.28	Output formatting
26	4.31	Final code refinement

The remaining 19 layers have scale ≈ 0 (experts effectively disabled), preserving LFM2's general language capabilities.

Training Details

Parameter	Value
Training Steps	300
Training Time	8.3 minutes (Modal L4 GPU)
Router Learning Rate	1e-3
Scale Learning Rate	1.0
Loss	LM loss + 0.01 × Load Balancing loss
Optimizer	AdamW
Gradient Accumulation	4
Max Sequence Length	512
Training Data	55 examples (40 coding + 15 general)
Total Compute Cost	~$3 (all phases combined)

Training Phases

1. Phase 1 — Expert Profiling ($1.90, A100 80GB): Profile Qwen3.6-35B-A3B expert activations on coding vs non-coding prompts. Select 960 coding-specialized experts. Extract weights.

2. Phase 2 — Assembly ($0.50, L4): Load LFM2.5-2.6B, wrap decoder layers with Fuse3AugmentedLayer, load extracted expert weights. Verify base model output coherence.

3. Phase 3 — Router Training ($0.60, L4): Train router + expert_scale on coding and general examples. Router learns to activate experts only for coding-related tokens.

Stability Mechanisms

• Std normalization: Expert outputs are rescaled to match host layer activation std (ratio clamped to 2.0)
• Scale clamping: Expert scale clamped to max=0.1 during forward pass
• SwiGLU clamping: Expert intermediate activations clamped to [-10, 10]
• Frozen host + experts: Only 2.0M parameters are trainable (router weights + per-layer scale)

Usage

Installation

pip install transformers torch

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Akahsizrr/fuse-1-Lite"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=True,
        temperature=0.1,
        top_k=50,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
    )

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)

Quantization & Deployment

Pre-quantized Versions

Version	Repo	VRAM/Memory	Format
4-bit NF4	Akahsizrr/fuse-1-Lite-4bit	3.36 GB	bitsandbytes
8-bit	Akahsizrr/fuse-1-Lite-8bit	6.00 GB	bitsandbytes
bfloat16	This repo	~12 GB	safetensors
MLX	Akahsizrr/fuse-1-Lite-MLX	~12 GB	MLX safetensors
GGUF F16	Akahsizrr/fuse-1-Lite-GGUF	~11.4 GB	GGUF
vLLM plugin	Akahsizrr/fuse-1-Lite-vLLM	~12 GB	vLLM plugin

bitsandbytes 4-bit (NF4) — 3.36 GB VRAM

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "Akahsizrr/fuse-1-Lite",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("Akahsizrr/fuse-1-Lite")

bitsandbytes 8-bit — 6.00 GB VRAM

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(load_in_8bit=True)

model = AutoModelForCausalLM.from_pretrained(
    "Akahsizrr/fuse-1-Lite",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

Disable Coding Experts (Pure LFM2 Mode)

# The model includes a coding toggle — disable experts for pure LFM2 inference
model.set_coding_enabled(False)

# Re-enable for coding tasks
model.set_coding_enabled(True)

vLLM — High-Throughput Serving

fuse-1 Lite is supported in vLLM via a plugin that extends vLLM's native LFM2 implementation with expert augmentation layers.

Plugin repo: Akahsizrr/fuse-1-Lite-vLLM

# Install the plugin
pip install git+https://huggingface.co/Akahsizrr/fuse-1-Lite-vLLM

# Serve with vLLM
vllm serve Akahsizrr/fuse-1-Lite \
  --mamba-cache-mode align \
  --max-model-len 4096

from vllm import LLM

llm = LLM(
    model="Akahsizrr/fuse-1-Lite",
    mamba_cache_mode="align",
    max_model_len=4096,
)
output = llm.generate("Write a Python function to check if a number is prime.")

The plugin registers Fuse3ForCausalLM with vLLM's ModelRegistry via the vllm.general_plugins entry point. It reuses vLLM's native LFM2 attention and short-conv layers, adding the expert MoE block after each augmented layer's FFN.

MLX (Apple Silicon)

fuse-1 Lite is available in MLX format for Apple Silicon (M1+).

MLX repo: Akahsizrr/fuse-1-Lite-MLX

from mlx_lm import load, generate

model, tokenizer = load("Akahsizrr/fuse-1-Lite-MLX", trust_remote_code=True)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a Python function to check if a number is prime."}],
    tokenize=False, add_generation_prompt=True,
)

response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

# CLI
mlx_lm.generate --model Akahsizrr/fuse-1-Lite-MLX --trust-remote-code --prompt "Write a Python fizzbuzz"

The MLX model file (fuse3_mlx.py) extends MLX's native LFM2 implementation with the same expert MoE augmentation. It uses model_file in config.json with trust_remote_code=True for loading.

GGUF / llama.cpp

fuse-1 Lite is available in GGUF format for llama.cpp.

GGUF repo: Akahsizrr/fuse-1-Lite-GGUF

Note: The GGUF uses the custom fuse3 architecture. Stock llama.cpp cannot load it — you need a llama.cpp fork with Fuse3 support. The GGUF repo includes the C++ graph builder (src/models/fuse3.cpp), Python converter (conversion/fuse3.py), and integration guide (INTEGRATION.md).

# Build llama.cpp with Fuse3 support (see INTEGRATION.md in the GGUF repo)
./llama-cli -m fuse-1-Lite-f16.gguf \
  -p "Write a Python function to check if a number is prime." \
  -n 512 --temp 0.1

The C++ implementation reuses LFM2's attention and short-conv graph builders, adding the expert MoE block (router → top-k → SwiGLU experts → scale → add) after each augmented layer's dense FFN.

Transformers (Universal)

The recommended way to run fuse-1 Lite on any platform:

pip install transformers torch bitsandbytes accelerate

Performance

VRAM Requirements

Backend	Precision	VRAM/Memory	Recommended Hardware
Transformers	bfloat16	~12 GB	L4, A10G, RTX 4090
Transformers	8-bit	6.00 GB	T4, L4, RTX 3060
Transformers	4-bit	3.36 GB	T4, RTX 3060, M2 Pro
vLLM	bfloat16	~12 GB	A10G, A100, H100
MLX	float16	~12 GB	M1 Pro+, M2, M3, M4
llama.cpp	F16	~11.4 GB	Any CPU/GPU
llama.cpp	Q4_K_M	~4 GB	Any CPU/GPU

Sample Outputs

Prompt: "Write a Python function to check if a string is a palindrome."

Output (excerpt):

def is_palindrome(s: str) -> bool:
    """
    Return True if *s* reads the same forwards and backwards,
    ignoring case and non-alphanumeric characters.
    """
    cleaned = ''.join(ch.lower() for ch in s if ch.isalnum())
    return cleaned == cleaned[::-1]

The model produces complete implementations with docstrings, type hints, complexity analysis, and test cases.

Limitations

1. Custom architecture: Requires trust_remote_code=True — the model includes custom Fuse3ForCausalLM code
2. No vLLM support: The custom MoE augmentation is not yet supported by vLLM's optimized inference engine
3. No GGUF/MLX conversion: The custom architecture cannot be directly converted to GGUF or MLX format
4. Training data was small: Only 55 examples were used for router training — the router may not generalize perfectly to all coding tasks
5. Expert compatibility: Qwen3.6 experts operate on LFM2's activation space with std normalization — some expert knowledge may be lost in translation
6. use_cache=False during training: Augmented layers don't propagate KV cache correctly during training; generation uses the standard cache
7. bitsandbytes quantization: 4-bit and 8-bit quantization work at runtime via BitsAndBytesConfig — pre-quantized saved versions are not available as separate repos

Citation

@misc{fuse1lite2026,
  title={fuse-1 Lite: Coding-Enhanced Mixture-of-Experts via Expert Transplantation},
  author={Vasko Djack},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/Akahsizrr/fuse-1-Lite}
}

Acknowledgments

• LiquidAI for the LFM2.5-2.6B host model
• Qwen Team for the Qwen3.6-35B-A3B donor model
• Modal for compute infrastructure

License

Apache 2.0 — See LICENSE for details.

---

Built with Devin — Cognition AI