outlast85/qwen2.5-0.5b-moe-domain-experts

Qwen2.5-0.5B → MoE, domain experts (moe-upcycle demo)

A demonstration artifact, not a production model. It shows the moe-upcycle method: take dense Qwen2.5-0.5B, clone each FFN into 4 routed experts + 1 shared "generalist" expert + a router, then supervise the router by domain so each routed expert becomes a real, named field specialist (Mode B).

What it demonstrates

• Named, individually-loadable domain experts. Held-out routing accuracy 87% mean (code 97.7%, web 97.8%, explanation 88.0%, conversation 64.5% — conversation is honestly the blurriest, its tokens overlap every domain).
• A "secretary": given a prompt it reports the one specialist to load (+ the shared GP) — the basis for selective expert offloading (cf. llama.cpp -ncmoe).

⚠️ Honest limitations (please read)

• This model is larger, slower, and lower-quality than the base Qwen2.5-0.5B it was made from. On held-out perplexity the base wins (9.27 vs ~13–23 depending on the run); this MoE has ~3.5× the params and runs ~0.4× the speed.
• That is expected: upcycling + a few-million-token budget cannot beat a model pretrained on ~18T tokens, and Drop-Upcycling deliberately perturbs pretrained weights. Upcycling pays off only at much larger training scale, and when compared to a same-size dense model — not the small base.
• Use this to study the method and the routing behavior, not as a better small LLM.

How to load (needs the repo's upcycle.py)

This is a custom MoE saved as a state_dict, so loading needs the surgery code:

# pip install torch transformers ; git clone https://github.com/outlast85/moe-upcycle
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from upcycle import upcycle_model            # from the moe-upcycle repo

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
m = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.bfloat16)
upcycle_model(m, num_experts=4, top_k=1, drop_fraction=0.3, num_shared=1)
for l in m.model.layers: l.mlp.router.float()
m.load_state_dict(torch.load("mode_b_domain_experts.pt"))
m.eval()
# expert map: 0=code, 1=conversation, 2=explanation, 3=web

Training

Distilled/trained on ~30k license-clean real examples (wikitext, codeparrot, OpenAssistant/oasst1, Dolly-15k), teacher Qwen2.5-3B, on an RTX 5090. Full method, code, and benchmarks: https://github.com/outlast85/moe-upcycle