---
license: apache-2.0
language:
  - en
  - th
tags:
  - moe
  - test
  - proof-of-concept
  - qwen2.5
  - mergekit
---

# 🧪 FrankenMoE — Proof of Concept (NOT production)

**This is a technical experiment, not a useful model.**

## ⚠️ Important Warning

This repository documents a **proof-of-concept** MoE pipeline. The model quality is **NOT good** — it produces incoherent / random outputs because:

1. The router uses  (no training)
2. Experts were fine-tuned with only ~5K samples each  
3. Base model is only Qwen2.5-1.5B-Instruct

**Do NOT use this model for anything serious.** It exists purely to demonstrate that the FrankenMoE pipeline can be built end-to-end.

## What We Actually Built

A working MoE pipeline from dense LoRA experts → GGUF:

```
Qwen2.5-1.5B-Instruct (base)
  ├── Expert 0: Coding (LoRA fine-tuned)
  ├── Expert 1: Math (LoRA fine-tuned)  
  └── Shared Expert: Base model
```

## Key Technical Discoveries

| Discovery | Detail |
|---|---|
| mergekit 0.1.4 bug |  param incompatible with transformers >= 4.40 — must patch  |
| QwenMoE requirements | Exactly 1 shared expert + 2^n routed experts (2, 4, 8) |
| Tied embeddings fix | Qwen2.5 uses tied embeddings → must clone  →  before GGUF conversion, set  |
| LoRA must be merged | Adapters must be  before MoE assembly |

## Repository Structure

```
📦 frankenmoe_moe_v2-F16.gguf  — MoE GGUF (fixed, has output.weight)
📁 moe_full/                   — Full safetensors model
📁 coding/ math/ chat/         — Individual dense experts (LoRA + GGUF)
📄 FrankenMoE_Academic_Paper.pdf — Research paper
🐍 simple_router.py            — Keyword-based router (functional alternative)
```

## Quick Test

```bash
wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/frankenmoe_moe_v2-F16.gguf
llama-cli -m frankenmoe_moe_v2-F16.gguf -p "Write a Python function"
# Output: Random/incoherent — this is expected! See warning above.
```

## Future: Real Model

The pipeline will be re-run with:
- Larger base model (Qwen2.5-7B/14B)
- Trained router (classification loss)
- More training data per domain
- 4 experts for proper 2^n routing

**Stay tuned — the real model is coming.**

---
Built by UKA 🇹🇭 | May 2026