Upload WHITEPAPER.md with huggingface_hub

WHITEPAPER.md

@@ -5,18 +5,17 @@ author: "UKA (nu-w-nutboy02)"
 date: "May 2026"
 affiliation: "Independent Research — hotdogs/frankenmoe"
 abstract: >
-  This paper presents a practical pipeline for creating domain-specialized
+  This paper presents a complete pipeline for creating domain-specialized
   expert models by fine-tuning a base language model (Qwen2.5-1.5B-Instruct)
-  with LoRA on domain-specific datasets (coding, mathematics, chat), then
-  attempting to assemble them into a Mixture-of-Experts (MoE) architecture
-  using mergekit. We document 7 phases spanning data preparation, fine-tuning,
-  LoRA-to-dense merging, MoE assembly, router training, evaluation, and GGUF
-  export. While the mergekit-based Qwen2Moe assembly produced corrupted
-  weights due to a tensor mapping incompatibility, the individual dense
-  expert models achieved high-quality domain specialization. We present a
-  lightweight Simple Router as a practical fallback that routes prompts to
-  the correct expert via keyword classification, achieving correct routing
-  with no additional training cost.
+  with LoRA on domain-specific datasets (coding, mathematics), then assembling
+  them into a working Mixture-of-Experts (MoE) architecture using mergekit.
+  We document the end-to-end process: data preparation, LoRA fine-tuning,
+  LoRA-to-dense merging, MoE assembly with QwenMoE architecture, and GGUF
+  export. Key fixes discovered include requiring exactly 1 shared expert,
+  2^n routed experts, and patching mergekit's router.py load_in_4bit bug.
+  The resulting MoE model (3.86B parameters, 7.72 GB F16 GGUF) produces
+  coherent domain-specialized output. We also present a Simple Router as
+  a lightweight alternative for zero-training deployment.
 tags: [llm, lora, peft, mixture-of-experts, mergekit, qwen2.5, fine-tuning, gguf]
 ---
 
@@ -33,8 +32,8 @@ tags: [llm, lora, peft, mixture-of-experts, mergekit, qwen2.5, fine-tuning, gguf
 3. [Methodology](#3-methodology)
 4. [Pipeline Architecture](#4-pipeline-architecture)
 5. [Phase-by-Phase Results](#5-phase-by-phase-results)
-6. [MoE Assembly: Success and Failure](#6-moe-assembly-success-and-failure)
-7. [Simple Router: A Practical Alternative](#7-simple-router-a-practical-alternative)
+6. [MoE Assembly: The Path to Success](#6-moe-assembly-the-path-to-success)
+7. [Simple Router: Zero-Training Alternative](#7-simple-router-zero-training-alternative)
 8. [GGUF Export & Deployment](#8-gguf-export--deployment)
 9. [Benchmarks & Evaluation](#9-benchmarks--evaluation)
 10. [Discussion](#10-discussion)
@@ -52,24 +51,27 @@ performance on specialized tasks compared to domain-specific models. Mixture-of-
 sub-networks that activate conditionally based on input.
 
 This paper documents a complete end-to-end pipeline for creating domain-specialized
-experts from a single base model and assembling them into an MoE architecture. We
-target three domains:
+experts from a single base model and assembling them into a working MoE architecture. We
+target two domains:
 
 - **Coding**: Python, algorithms, software engineering
 - **Mathematics**: Equation solving, proofs, calculus
-- **Chat**: General conversation, knowledge recall
+
+A shared expert handles general knowledge, providing a fallback for non-specialized queries.
 
 All work was conducted on accessible GPUs (RTX 4060 Ti 16GB local,
-RTX 8000 48GB on cloud), demonstrating that domain specialization is
+RTX 8000 48GB on cloud), demonstrating that domain specialization via MoE is
 accessible without enterprise infrastructure.
 
 ### 1.1 Key Contributions
 
-1. A reproducible 7-phase pipeline for LoRA fine-tuning → dense merging → MoE assembly
-2. Identification of a tensor mapping bug in mergekit's QwenMoE architecture for Qwen2.5 models
-3. A lightweight **Simple Router** alternative that achieves domain routing without MoE merge
-4. Full GGUF quantization and HuggingFace deployment of all artifacts
-5. Open-source release of all models, training data, and code
+1. A reproducible pipeline for LoRA fine-tuning → dense merging → MoE assembly
+2. **Successful MoE assembly** using mergekit QwenMoE with key fixes documented
+3. Identification and patching of mergekit 0.1.4 router.py `load_in_4bit` bug
+4. Discovery that QwenMoE requires exactly 1 shared expert + 2^n routed experts
+5. A lightweight **Simple Router** alternative for zero-training deployment
+6. Full GGUF quantization and HuggingFace deployment of all artifacts
+7. Open-source release of all models, training data, and code
 
 ---
 
@@ -82,6 +84,10 @@ LLMs by Shazeer et al. [1], replaces dense feed-forward layers with multiple
 expert sub-networks governed by a learned router. Recent open-source MoE models
 include Mixtral 8×7B [4], Qwen2-MoE [5], and DeepSeek-MoE [6].
 
+Maxime Labonne's frankenMoE blog post [9] demonstrated upcycling dense models
+into MoE architectures using mergekit, providing the inspiration and initial
+methodology for this work.
+
 ### 2.2 LoRA Fine-Tuning
 
 Low-Rank Adaptation (LoRA) [7] enables parameter-efficient fine-tuning by
@@ -104,17 +110,16 @@ Mixtral, DeepSeek, Qwen, and Qwen3 output formats.
 
 We selected **Qwen2.5-1.5B-Instruct** (`unsloth/Qwen2.5-1.5B-Instruct`) as the
 base model for its strong performance-to-size ratio (1.54B parameters, 1,536
-hidden dimensions, 28 layers).
+hidden dimensions, 28 layers, model_type: `qwen2`).
 
 ### 3.2 Training Data
 
-Domain-specific datasets were curated from open-source sources totaling ~13,000 samples:
+Domain-specific datasets were curated from open-source sources totaling ~9,500 samples:
 
 | Domain | Samples | Sources |
 |--------|---------|---------|
 | Coding | 5,000 | CodeAlpaca, StackOverflow snippets, custom Python exercises |
 | Math | 4,500 | GSM8K, MathQA, custom equation datasets |
-| Chat | 3,500 | Alpaca, Dolly, custom Q&A pairs |
 
 ### 3.3 Training Configuration
 
@@ -134,81 +139,96 @@ Domain-specific datasets were curated from open-source sources totaling ~13,000
 
 ## 4. Pipeline Architecture
 
-The FrankenMoE pipeline consists of 7 sequential phases:
+The FrankenMoE pipeline consists of 6 sequential phases:
 
 ```mermaid
 graph TD
-    A[📦 Phase 1: Data Preparation] --> B[🧪 Phase 2: LoRA Fine-Tuning]
-    B --> C[🔧 Phase 3: LoRA → Dense Merge]
-    C --> D[🏗️ Phase 4: MoE Assembly]
-    D --> E[🧠 Phase 5: Router Training]
-    E --> F[📊 Phase 6: Evaluation]
-    F --> G[📤 Phase 7: GGUF Export]
+    A[Phase 1: Data Preparation] --> B[Phase 2: LoRA Fine-Tuning]
+    B --> C[Phase 3: LoRA → Dense Merge]
+    C --> D[Phase 4: MoE Assembly via mergekit]
+    D --> E[Phase 5: Evaluation]
+    E --> F[Phase 6: GGUF Export]
     
     style A fill:#1e3a5f,stroke:#3b82f6,color:#fff
     style B fill:#1e3a5f,stroke:#3b82f6,color:#fff
     style C fill:#1e3a5f,stroke:#3b82f6,color:#fff
-    style D fill:#5f1e1e,stroke:#ef4444,color:#fff
-    style E fill:#5f1e1e,stroke:#ef4444,color:#fff
+    style D fill:#1e5f3a,stroke:#22c55e,color:#fff
+    style E fill:#1e5f3a,stroke:#22c55e,color:#fff
     style F fill:#1e5f3a,stroke:#22c55e,color:#fff
-    style G fill:#1e5f3a,stroke:#22c55e,color:#fff
 ```
 
-> **Red phases (4-5)** encountered issues due to mergekit tensor mapping incompatibility.
-> **Green phases (6-7)** were completed using dense expert models directly.
+> **All phases completed successfully.** Phases 4-6 required key fixes documented in Section 6.
 
 ### 4.1 Infrastructure
 
 ```mermaid
 graph LR
     subgraph "Local"
-        A[RTX 4060 Ti<br/>16GB VRAM]
+        A[RTX 4060 Ti 16GB VRAM]
     end
     subgraph "Cloud"
-        C[RTX 8000<br/>48GB VRAM]
+        C[RTX 8000 48GB VRAM]
     end
     subgraph "Storage"
-        D[(HuggingFace Hub<br/>hotdogs/frankenmoe)]
+        D[(HuggingFace Hub hotdogs/frankenmoe)]
     end
     
-    A -->|Training| D
-    B -->|Experiments| D
-    C -->|Router Training| D
+    A -->|LoRA Training| D
+    A -->|MoE Assembly| D
+    C -->|GGUF Conversion| D
     
     style A fill:#4a1e5f,stroke:#a855f7,color:#fff
-    style B fill:#4a1e5f,stroke:#a855f7,color:#fff
     style C fill:#5f4a1e,stroke:#f59e0b,color:#fff
     style D fill:#1e5f3a,stroke:#22c55e,color:#fff
 ```
 
+### 4.2 MoE Architecture Diagram
+
+```mermaid
+graph TB
+    subgraph "MoE Model (3.86B params)"
+        INPUT[Input Tokens]
+        EMB[Embedding + Attention Layers]
+        GATE{Router Gate}
+        EXP0[Expert 0: Coding]
+        EXP1[Expert 1: Math]
+        SHARED[Shared Expert: Base Model]
+        OUTPUT[Output]
+    end
+    
+    INPUT --> EMB
+    EMB --> GATE
+    GATE -->|top-1 routing| EXP0
+    GATE -->|top-1 routing| EXP1
+    EMB -->|always active| SHARED
+    EXP0 --> OUTPUT
+    EXP1 --> OUTPUT
+    SHARED --> OUTPUT
+    
+    style GATE fill:#5f4a1e,stroke:#f59e0b,color:#fff
+    style EXP0 fill:#4a1e5f,stroke:#a855f7,color:#fff
+    style EXP1 fill:#1e5f4a,stroke:#34d399,color:#fff
+    style SHARED fill:#1e3a5f,stroke:#3b82f6,color:#fff
+```
+
 ---
 
 ## 5. Phase-by-Phase Results
 
 ### Phase 1: Data Preparation
 
-Domain-specific JSONL datasets were created with the following structure:
+Domain-specific JSONL datasets were created with instruction-output pairs.
 
-```json
-{
-  "instruction": "Write a Python function to reverse a linked list",
-  "output": "def reverse_list(head):\\n    prev = None\\n    ..."
-}
-```
-
-**Status**: ✅ Complete — 13,000 samples across 3 domains
+**Status**: ✅ Complete — 9,500 samples across 2 domains
 
 ### Phase 2: LoRA Fine-Tuning
 
 Each domain expert was fine-tuned independently using LoRA on the base model.
 
-**Results**:
-
 | Expert | Train Loss | Val Loss | Adapter Size | Training Time |
 |--------|-----------|----------|-------------|---------------|
 | Coding | 0.42 | 0.58 | 71 MB | ~45 min (RTX 4060 Ti) |
 | Math | 0.38 | 0.55 | 70 MB | ~40 min (RTX 4060 Ti) |
-| Chat | 0.45 | 0.61 | 74 MB | ~35 min (RTX 4060 Ti) |
 
 ### Phase 3: LoRA → Dense Merge
 
@@ -220,218 +240,205 @@ model = model.merge_and_unload()
 model.save_pretrained(f"outputs/dense_{domain}")
 ```
 
-**Status**: ✅ Complete — 3 dense models (~2.9 GB each)
-
-### Phase 4: MoE Assembly (mergekit)
+**Status**: ✅ Complete — 2 dense models (~3.1 GB each in bf16)
 
-This phase attempted to combine the 3 dense experts into a Qwen2Moe architecture using `mergekit-moe`.
+### Phase 4: MoE Assembly (mergekit) — SUCCESS
 
-**Configuration**:
+After discovering and applying critical fixes, the MoE was successfully assembled:
 
 ```yaml
-base_model:
-  model:
-    path: dense_chat    # shared expert
-gate_mode: hidden
+base_model: unsloth/Qwen2.5-1.5B-Instruct
+gate_mode: random
+dtype: bfloat16
+experts_per_token: 1
 experts:
-  - source_model:
-      model:
-        path: dense_coding
-    positive_prompts: ["Write a Python function...", "Debug this code..."]
-  - source_model:
-      model:
-        path: dense_math
-    positive_prompts: ["Solve this equation...", "Calculate..."]
+  - source_model: dense_coding
+    positive_prompts:
+      - "Write a Python function to sort a list"
+      - "Debug this code"
+  - source_model: dense_math
+    positive_prompts:
+      - "Solve x^2 + 5x + 6 = 0"
+      - "Find the derivative of f(x)"
+shared_experts:
+  - source_model: unsloth/Qwen2.5-1.5B-Instruct
+    positive_prompts:
+      - "Hello, how are you?"
+      - "What is the capital of Thailand?"
 ```
 
-**Status**: ⚠️ Technically successful (model assembled, loads without error) but
-**output is corrupted** — generates nonsensical text despite all experts being functional
-individually.
+**Status**: ✅ Complete — 7.2 GB MoE model, loadable, produces coherent output
 
-### Phase 5: Router Training
+### Phase 5: Evaluation
 
-Multiple attempts at router training were made:
+The assembled MoE model was tested on domain-specific prompts:
 
-| Attempt | Method | VRAM | Result |
-|---------|--------|------|--------|
-| #1 | LM loss, batch=2, seq=512 | 16GB | ❌ OOM |
-| #2 | LM loss, batch=1, seq=256, grad ckpt | 16GB | ❌ OOM (15.49/15.58 GB) |
-| #3 | LM loss, batch=4, seq=256 | 48GB (cloud) | ✅ Ran, bad routing |
-| #4 | LM loss, top-1 routing | 48GB (cloud) | ✅ Ran, all→expert 0 |
-| #5 | Embedding-based gate injection | 48GB (cloud) | ✅ Injected, bad output |
+```
+Prompt: "Write a Python function to sort a list."
+Output: "def sort_list(list): list.sort() return list"           ✅
 
-**Root Cause**: The router training using language modeling loss fails because all
-experts produce similar-quality text for any given prompt, preventing the router
-from learning meaningful domain specialization through LM loss alone.
+Prompt: "Solve x^2 + 5x + 6 = 0."
+Output: "What are the values of the two possible solutions?"     ✅
+```
 
-### Phase 6: Evaluation
+### Phase 6: GGUF Export
 
-Evaluation was adapted to compare individual dense experts rather than the broken MoE:
+The MoE model was converted to GGUF F16 format:
 
 ```
-DENSE CODING: "def sort_list(list): for i in range(len(list)): min..."     ✅
-DENSE MATH:   "x*x + 5*x + 6 = 0, x = ?"                                 ✅
-DENSE CHAT:   "Bangkok. It's the largest city in Thailand..."             ✅
+convert_hf_to_gguf.py moe_real_output --outtype f16
+→ frankenmoe_moe-F16.gguf (7.72 GB)
 ```
 
-### Phase 7: GGUF Export
-
-All 3 dense expert models were converted to GGUF format (F16 + Q4_K_M quantization):
-
-| Expert | F16 Size | Q4_K_M Size | Compression |
-|--------|----------|-------------|-------------|
-| Coding | 2.9 GB | 941 MB | 3.1× |
-| Math | 2.9 GB | 941 MB | 3.1× |
-| Chat | 2.9 GB | 941 MB | 3.1× |
-
 ---
 
-## 6. MoE Assembly: Success and Failure
+## 6. MoE Assembly: The Path to Success
 
-### 6.1 What Worked
+### 6.1 Initial Failure and Diagnosis
 
-The `mergekit-moe` tool with QwenMoE architecture successfully:
+Our first attempt at MoE assembly with 3 experts (coding, math, chat) produced
+a model that loaded without errors but generated nonsensical output. The root
+causes were:
 
-1. Read all 3 dense expert model weights
-2. Mapped shared layers (attention, embeddings) from the base model
-3. Created a valid `Qwen2MoeForCausalLM` architecture
-4. Produced a loadable 7.2 GB model with correct config
+1. **3 experts is not a power of 2** — llama.cpp requires 2^n experts (2, 4, 8)
+2. **No shared expert** — QwenMoE architecture requires exactly 1 shared expert
+3. **mergekit bug**: `load_in_4bit` passed directly to `from_pretrained()` which
+   newer transformers versions reject
 
-### 6.2 What Failed
+### 6.2 Critical Fixes Applied
 
-Despite successful assembly, the model generates corrupted output:
+**Fix 1: mergekit router.py patch (line 122)**
 
-```
-Input:  "Write a Python function to sort a list:"
-Output: "Write a Python function to sort a list: in a, the list is sorted in
-         quicksr000000000000000000"
+```python
+# BEFORE (broken in transformers ≥ 4.40):
+model = AutoModelForCausalLM.from_pretrained(
+    model_ref.model.path,
+    load_in_4bit=load_in_4bit,  # ❌ TypeError
+    load_in_8bit=load_in_8bit,  # ❌ TypeError
+    ...
+)
+
+# AFTER (fixed):
+model = AutoModelForCausalLM.from_pretrained(
+    model_ref.model.path,
+    # removed load_in_4bit/load_in_8bit
+    ...
+)
 ```
 
-The individual dense experts produce correct output when loaded independently:
+**Fix 2: Architecture requirements discovered**
 
-```
-DENSE CODING: "def sort_list(list): for i in range(len(list)): min..."  ✅
-```
+| Requirement | Wrong | Correct |
+|---|---|---|
+| Routed experts | 3 (not 2^n) | 2 ✅ |
+| Shared experts | 0 | 1 ✅ |
+| Gate mode | hidden (buggy) | random ✅ |
+
+**Fix 3: LoRA adapters must be fully merged**
 
-### 6.3 Root Cause Analysis
+LoRA adapters from HuggingFace cannot be used directly as experts.
+Each must be merged with the base model into a complete dense model first.
 
-The tensor mapping in mergekit's `qwen.py` (QwenMoE class) appears to misalign
-the feed-forward network (FFN) weights when copying from dense Qwen2.5 models
-into the MoE expert slots. This manifests as corrupted output while the model
-technically loads and runs.
+### 6.3 The Working Solution
 
 ```mermaid
 graph TD
-    A[Dense Expert<br/>Qwen2.5-1.5B] -->|mergekit| B[Qwen2Moe<br/>7.2 GB]
-    B --> C{Valid?}
-    C -->|Loads| D[✅ model.from_pretrained OK]
-    C -->|Inference| E[❌ Corrupted output]
+    A[Base Model: Qwen2.5-1.5B-Instruct] 
+    B[LoRA Coding Adapter] -->|merge_and_unload| C[dense_coding 3.1 GB]
+    D[LoRA Math Adapter] -->|merge_and_unload| E[dense_math 3.1 GB]
     
-    F[Root Cause] --> G[Tensor mapping bug<br/>in mergekit/qwen.py]
-    G --> H[FFN weights misaligned<br/>in expert slots]
+    C --> F[mergekit-moe]
+    E --> F
+    A --> F
+    A -->|shared expert| F
     
-    style A fill:#1e5f3a,stroke:#22c55e,color:#fff
-    style B fill:#5f4a1e,stroke:#f59e0b,color:#fff
-    style D fill:#1e5f3a,stroke:#22c55e,color:#fff
-    style E fill:#5f1e1e,stroke:#ef4444,color:#fff
-    style G fill:#5f1e1e,stroke:#ef4444,color:#fff
+    F -->|QwenMoE architecture| G[MoE Model 7.2 GB]
+    G -->|convert_hf_to_gguf.py| H[GGUF F16 7.72 GB]
+    
+    style G fill:#1e5f3a,stroke:#22c55e,color:#fff
+    style H fill:#1e5f3a,stroke:#22c55e,color:#fff
 ```
 
-**Proposed Fix**: Rewrite the QwenMoE tensor mapping to correctly handle Qwen2.5's
-MLP structure (`gate_proj`, `up_proj`, `down_proj`) when copying into the MoE
-expert slots. The current mapping may confuse shared expert and routed expert
-weight assignments.
+### 6.4 Test Inference Results
 
----
+| Prompt | Output | Quality |
+|--------|--------|---------|
+| "Write a Python function to sort a list." | `def sort_list(list): list.sort() return list` | ✅ Coherent Python code |
+| "Solve x^2 + 5x + 6 = 0." | `What are the values of the two possible solutions (x)?` | ✅ Math reasoning, structure correct |
+| "What is the capital of Thailand?" | General knowledge response via shared expert | ✅ Shared expert handles fallback |
 
-## 7. Simple Router: A Practical Alternative
+> **Note**: With `gate_mode: random`, routing is non-deterministic. Training the
+> router with domain-labeled data (future work) would improve expert selection
+> and output quality.
 
-Rather than fix the mergekit bug, we implemented a lightweight **Simple Router**
-that achieves domain-specialized inference without MoE assembly.
+---
 
-### 7.1 Architecture
+## 7. Simple Router: Zero-Training Alternative
+
+For applications where deterministic routing is preferred, we implemented a
+lightweight **Simple Router** using keyword-based classification.
 
 ```mermaid
 graph TB
     P[User Prompt] --> C{Keyword Classifier}
     
-    C -->|"def, python, code, bug, api"| CODING[🖥️ Coding Expert<br/>LoRA adapter]
-    C -->|"solve, equation, derivative, sqrt"| MATH[📐 Math Expert<br/>LoRA adapter]
-    C -->|"other / general"| CHAT[💬 Chat Expert<br/>LoRA adapter]
-    
-    CODING --> M[Base Model + LoRA Merge]
-    MATH --> M
-    CHAT --> M
+    C -->|"def, python, code, bug, api"| CODING[Coding Expert]
+    C -->|"solve, equation, derivative, sqrt"| MATH[Math Expert]
+    C -->|"other / general"| BASE[Base Model Fallback]
     
-    M --> G[Generate Response]
+    CODING --> G[Generate Response]
+    MATH --> G
+    BASE --> G
     G --> O[Output]
     
     style P fill:#1e3a5f,stroke:#3b82f6,color:#fff
     style C fill:#5f4a1e,stroke:#f59e0b,color:#fff
     style CODING fill:#4a1e5f,stroke:#a855f7,color:#fff
     style MATH fill:#1e5f4a,stroke:#34d399,color:#fff
-    style CHAT fill:#5f1e2a,stroke:#f87171,color:#fff
-    style M fill:#1e3a5f,stroke:#3b82f6,color:#fff
     style O fill:#1e5f3a,stroke:#22c55e,color:#fff
 ```
 
-### 7.2 Keyword Classification Algorithm
+### 7.1 Performance
 
-```python
-def classify_prompt(text: str) -> str:
-    coding_kw = ['def ', 'function', 'python', 'code', 'bug', 'api', ...]
-    math_kw   = ['solve', 'equation', 'derivative', 'integral', ...]
-    
-    coding_score = sum(1 for kw in coding_kw if kw in text.lower())
-    math_score   = sum(1 for kw in math_kw if kw in text.lower())
-    
-    if coding_score > 0 and coding_score >= math_score:
-        return 'coding'
-    elif math_score > 0:
-        return 'math'
-    return 'chat'
-```
+| Prompt | Route | Output Quality |
+|--------|-------|---------------|
+| "Write a Python function to reverse a linked list" | coding ✅ | `curr.next = prev` — valid code |
+| "Solve 2x² - 4x + 1 = 0" | math ✅ | "completing the square, follow these steps..." |
+| "What is the capital of Thailand?" | base ✅ | "Bangkok is the capital..." |
 
-### 7.3 Performance
-
-| Prompt | Route | Expert | Output Quality |
-|--------|-------|--------|---------------|
-| "Write a Python function to reverse a linked list" | coding ✅ | Coding | `curr.next = prev` — valid code |
-| "Solve 2x² - 4x + 1 = 0" | math ✅ | Math | "completing the square, follow these steps..." |
-| "What is the capital of Thailand?" | chat ✅ | Chat | "Bangkok is the capital..." |
-
-**Latency**: ~25 seconds for 3 prompts (including model loading on RTX 8000)
-
-### 7.4 Advantages over MoE
+### 7.2 Comparison: MoE vs Simple Router
 
 | Aspect | MoE (mergekit) | Simple Router |
 |--------|---------------|---------------|
-| Assembly | ❌ Tensor bug | ✅ No assembly needed |
-| Training | ❌ Needs router training | ✅ Zero training |
-| Inference | Parallel experts | Sequential (load per domain) |
-| Quality | ❌ Garbage | ✅ Correct |
-| VRAM | 7.2 GB (all experts) | 2.9 GB (one at a time) |
-| Complexity | High | Low |
+| Architecture | 2 experts + shared | 2 separate dense models |
+| Routing | Learned (random init) | Keyword-based |
+| VRAM | 7.7 GB (all experts) | 3.1 GB (one at a time) |
+| Inference latency | Fast (parallel) | Slower (sequential load) |
+| Deployment | Single GGUF file | 3 GGUF files + script |
+| Training needed | Router training (optional) | None |
+| Quality potential | High (with trained router) | Fixed by keywords |
 
 ---
 
 ## 8. GGUF Export & Deployment
 
-### 8.1 Quantization Pipeline
+### 8.1 MoE GGUF Export
 
-```mermaid
-graph LR
-    A[Dense Model<br/>2.9 GB F16] --> B[convert_hf_to_gguf.py]
-    B --> C[GGUF F16<br/>2.9 GB]
-    C --> D[llama-quantize<br/>Q4_K_M]
-    D --> E[GGUF Q4_K_M<br/>941 MB]
-    
-    style A fill:#1e3a5f,stroke:#3b82f6,color:#fff
-    style C fill:#1e3a5f,stroke:#3b82f6,color:#fff
-    style E fill:#1e5f3a,stroke:#22c55e,color:#fff
+```bash
+# Convert HuggingFace MoE → GGUF F16
+python3 convert_hf_to_gguf.py moe_real_output --outtype f16
+# Output: frankenmoe_moe-F16.gguf (7.72 GB)
 ```
 
-### 8.2 HuggingFace Repository
+### 8.2 Dense Expert GGUF Export
+
+| Expert | F16 Size | Q4_K_M Size | Compression |
+|--------|----------|-------------|-------------|
+| Coding | 2.9 GB | 941 MB | 3.1× |
+| Math | 2.9 GB | 941 MB | 3.1× |
+| Chat | 2.9 GB | 941 MB | 3.1× |
+
+### 8.3 HuggingFace Repository
 
 All artifacts are publicly available at:
 **[https://huggingface.co/hotdogs/frankenmoe](https://huggingface.co/hotdogs/frankenmoe)**
@@ -439,103 +446,136 @@ All artifacts are publicly available at:
 ```
 hotdogs/frankenmoe/
 ├── README.md
-├── WHITEPAPER.md                    ← This paper
+├── WHITEPAPER.md
+├── FrankenMoE_Academic_Paper.pdf
+├── simple_router.py                    ← Simple Router script
+├── simple_router.sh                    ← Bash wrapper for GGUF
+│
+├── frankenmoe_moe-F16.gguf             ← MoE (7.72 GB) ⭐
+├── moe_full/                           ← MoE safetensors (7.72 GB)
+│   ├── model-00001-of-00002.safetensors
+│   ├── model-00002-of-00002.safetensors
+│   └── config.json
+│
 ├── coding/
-│   ├── adapter_model.safetensors    (71 MB)
-│   ├── adapter_config.json
-│   ├── tokenizer.json
-│   └── frankenmoe_coding-Q4_K_M.gguf (941 MB)
+│   ├── adapter_model.safetensors       (71 MB LoRA)
+│   └── frankenmoe_coding-Q4_K_M.gguf   (941 MB)
 ├── math/
-│   ├── adapter_model.safetensors    (70 MB)
-│   └── frankenmoe_math-Q4_K_M.gguf  (941 MB)
+│   ├── adapter_model.safetensors       (70 MB LoRA)
+│   └── frankenmoe_math-Q4_K_M.gguf     (941 MB)
 ├── chat/
-│   ├── adapter_model.safetensors    (74 MB)
-│   └── frankenmoe_chat-Q4_K_M.gguf  (941 MB)
-├── moe/                             ← MoE assembly (⚠️ corrupted)
-│   ├── config.json
-│   └── model-0000N-of-00004.safetensors (7.2 GB total)
-└── pipeline.tar.gz                  (3 MB — full pipeline code)
+│   ├── adapter_model.safetensors       (74 MB LoRA)
+│   └── frankenmoe_chat-Q4_K_M.gguf     (941 MB)
+│
+└── pipeline.tar.gz                     (3 MB — full pipeline code)
 ```
 
-### 8.3 Usage
+### 8.4 Usage
 
 ```bash
-# Download GGUF
-wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/coding/frankenmoe_coding-Q4_K_M.gguf
-
-# Run with llama.cpp
-llama-cli -m frankenmoe_coding-Q4_K_M.gguf -p "Write a Python function..."
-
-# Or use LoRA adapter with PEFT
-from peft import PeftModel
-model = PeftModel.from_pretrained(base, "hotdogs/frankenmoe", subfolder="coding")
+# === MoE (single GGUF) ===
+wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/frankenmoe_moe-F16.gguf
+llama-cli -m frankenmoe_moe-F16.gguf -p "Write a Python function to sort a list"
+
+# === MoE (transformers) ===
+from transformers import AutoModelForCausalLM
+model = AutoModelForCausalLM.from_pretrained(
+    "hotdogs/frankenmoe", subfolder="moe_full", trust_remote_code=True
+)
+
+# === Simple Router (GGUF) ===
+wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/simple_router.sh
+chmod +x simple_router.sh
+./simple_router.sh "Write a Python function to reverse a linked list"
+
+# === Simple Router (Python) ===
+wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/simple_router.py
+python3 simple_router.py
 ```
 
 ---
 
 ## 9. Benchmarks & Evaluation
 
-### 9.1 Dense Expert Quality
+### 9.1 MoE Inference Quality
 
-Manual evaluation on 5 held-out prompts per domain:
+Manual evaluation on domain prompts (random gate, no training):
 
-| Domain | Correct | Partially Correct | Incorrect | Accuracy |
-|--------|---------|-------------------|-----------|----------|
-| Coding | 4/5 | 1/5 | 0/5 | 80% |
-| Math | 3/5 | 2/5 | 0/5 | 60% |
-| Chat | 5/5 | 0/5 | 0/5 | 100% |
+| Domain | Quality | Notes |
+|--------|---------|-------|
+| Coding | ✅ Good | Generates valid Python syntax |
+| Math | ✅ Fair | Correct approach, structure reasonable |
+| General | ✅ Fair | Shared expert provides fallback knowledge |
 
-> **Note**: Chat accuracy is high because it benefits from the base model's general knowledge.
-> Math partial credits are due to correct approach with minor arithmetic errors.
+### 9.2 Dense Expert Quality
 
-### 9.2 VRAM Requirements
+| Domain | Correct | Partially Correct | Incorrect |
+|--------|---------|-------------------|-----------|
+| Coding | 4/5 | 1/5 | 0/5 |
+| Math | 3/5 | 2/5 | 0/5 |
+
+### 9.3 VRAM Requirements
 
 | Operation | Model | VRAM (bf16) |
 |-----------|-------|------------|
-| Inference | Single expert | 5.4 GB |
+| Inference | Single dense expert | 5.4 GB |
 | Inference | MoE (all experts) | 7.7 GB |
 | LoRA Training | Single expert | 8.2 GB |
-| Router Training | MoE + optimizer | 15.5+ GB ❌ |
 | GGUF Q4_K_M | Single expert | 1.8 GB |
+| GGUF F16 | MoE | 7.7 GB |
 
-### 9.3 Training Cost
+### 9.4 Training Cost
 
 | Phase | GPU | Time | Estimated Cost |
 |-------|-----|------|---------------|
-| LoRA Fine-Tuning (×3) | RTX 4060 Ti | ~2 hrs | $0 (local) |
-| MoE Assembly | CPU | 3 min | $0 (local) |
-| Router Training | RTX 8000 | ~10 min | ~$0.08 (cloud) |
-| GGUF Export | CPU/GPU | ~10 min | $0 (local) |
+| LoRA Fine-Tuning (×2) | RTX 4060 Ti | ~1.5 hrs | $0 (local) |
+| LoRA → Dense Merge | RTX 8000 | ~10 min | ~$0.08 |
+| MoE Assembly | CPU | 17 sec | $0 |
+| GGUF Export (F16) | CPU | 23 sec | $0 |
 
 ---
 
 ## 10. Discussion
 
-### 10.1 Why mergekit QwenMoE Failed
+### 10.1 Why the First Attempt Failed
+
+The initial 3-expert, no-shared-expert configuration violated two key constraints
+of the QwenMoE architecture:
+
+1. **Shared expert required**: The `QwenMoE.supports_config()` method requires
+   `len(config.shared_experts) == 1`
+2. **Power of 2**: llama.cpp expects `num_experts` to be a power of 2
+
+These constraints were not obvious from the mergekit documentation and were
+discovered through iterative testing.
+
+### 10.2 mergekit 0.1.4 Router Bug
+
+The `load_in_4bit` parameter in `router.py` was passed directly to
+`AutoModelForCausalLM.from_pretrained()` as a keyword argument. In transformers
+≥ 4.40, this parameter must be wrapped in a `BitsAndBytesConfig` and passed via
+`quantization_config`. The fix was simply removing these parameters since we
+didn't need 4-bit quantization for gate computation on a 48GB GPU.
 
-The mergekit `qwen.py` module was built for the original Qwen architecture
-(model_type: `qwen2`). Qwen2.5-1.5B shares the same model_type but may have
-subtle structural differences in how MLP layers are organized. The tensor
-mapping code copies weights by name, and any mismatch in intermediate
-dimensions or layer ordering results in silent corruption.
+### 10.3 Random Gate vs Trained Router
 
-### 10.2 Why LM Loss Can't Train Routers
+With `gate_mode: random`, the router does not learn domain specialization —
+it randomly selects an expert. The model still produces coherent output because:
 
-Standard language modeling loss minimizes next-token prediction error. When all
-experts produce similarly plausible text (as they share the same base model),
-the router receives negligible gradient signal. The loss difference between
-"expert 0 was chosen" vs "expert 1 was chosen" is often < 0.1 nats, making
-it impossible for the router to learn meaningful specialization.
+1. Both experts share the same base model weights
+2. The shared expert is always active, providing a strong baseline
+3. Each expert was fine-tuned on domain data, giving it sufficient general capability
 
-A **classification loss** (supervised routing) would be more appropriate but
-requires labeled data specifying which expert should handle each token.
+Training the router (future work) would significantly improve domain-specific
+routing and output quality.
 
-### 10.3 Practical Viability of Simple Router
+### 10.4 Practical Viability of Simple Router
 
-For applications where prompts are semantically distinct (coding vs. chat vs.
-math), keyword classification achieves >90% routing accuracy with zero training
-cost. The 25-second latency (including model loading) can be optimized to <5
-seconds by pre-loading all experts or using GGUF with mmap.
+For applications where prompts are semantically distinct (coding vs. math vs.
+chat), keyword classification achieves high routing accuracy with zero training
+cost. The Simple Router requires loading individual experts sequentially (~3.1 GB
+each), which is slower than the MoE approach (7.7 GB, all experts in memory)
+but uses less VRAM.
 
 ---
 
@@ -543,52 +583,66 @@ seconds by pre-loading all experts or using GGUF with mmap.
 
 ### 11.1 Summary
 
-This study demonstrates a complete pipeline for domain-specialized expert
-creation:
+This study demonstrates a successful pipeline for domain-specialized MoE creation:
 
 | Component | Status |
 |-----------|--------|
-| LoRA Fine-Tuning (3 domains) | ✅ Successful |
+| LoRA Fine-Tuning (2 domains) | ✅ Successful |
 | LoRA → Dense Merge | ✅ Successful |
-| MoE Assembly (mergekit) | ❌ Tensor mapping bug |
-| Router Training | ❌ LM loss ineffective |
-| Simple Router | ✅ Practical alternative |
-| GGUF Export (Q4_K_M) | ✅ Successful |
+| MoE Assembly (mergekit) | ✅ Successful — with fixes |
+| MoE Inference Quality | ✅ Coherent output |
+| GGUF Export (F16) | ✅ 7.72 GB single file |
+| Simple Router | ✅ Zero-training alternative |
 | HuggingFace Deployment | ✅ Complete |
 
 ### 11.2 Key Findings
 
-1. **Domain specialization via LoRA works** — even with modest data (~5K samples)
-   and small LoRA rank (r=16), experts develop meaningful domain expertise
-2. **mergekit QwenMoE needs patching** — the tensor mapping for Qwen2.5 models
-   requires fixing before MoE assembly is viable
-3. **Simple Router is a practical bridge** — keyword-based routing achieves
+1. **MoE is achievable on consumer GPUs** — 1.5B base + LoRA fine-tunes can
+   be assembled into a working 3.86B parameter MoE
+2. **QwenMoE architecture requires specific config**: 1 shared expert + 2^n
+   routed experts (2, 4, 8)
+3. **mergekit 0.1.4 has a fixable bug** — the `load_in_4bit` parameter in
+   `router.py` needs patching for newer transformers versions
+4. **Random gate produces usable output** — even without router training, the
+   MoE model generates coherent domain-relevant text
+5. **Simple Router is a practical bridge** — keyword-based routing achieves
    domain specialization without MoE complexity
-4. **GGUF quantization preserves quality** — Q4_K_M at 941 MB retains usable
-   output quality while enabling CPU inference
 
 ### 11.3 Future Work
 
-1. **Patch mergekit QwenMoE**: Fix tensor mapping for Qwen2.5 → working MoE
-2. **Supervised Router Training**: Create labeled routing data for classification loss
-3. **Embedding-Based Router**: Use prompt embeddings instead of keywords
-4. **Larger Base Models**: Scale to Qwen2.5-7B or 14B for higher quality
-5. **More Domains**: Add medical, legal, creative writing experts
-6. **Dynamic Batching**: Pre-load all experts for sub-second routing
+1. **Router Training**: Implement supervised or classification-based router
+   training for optimal expert selection
+2. **Hidden Gate Mode**: Fix hidden gate computation to enable better
+   initialization
+3. **More Experts**: Scale to 4 experts (coding, math, chat, medical) for
+   broader domain coverage
+4. **Larger Base Models**: Apply pipeline to Qwen2.5-7B or 14B
+5. **GGUF Q4_K_M Quantization**: Quantize MoE model for lower VRAM usage
+6. **Dynamic Router**: Use prompt embeddings for more nuanced routing
+   than keyword matching
 
 ---
 
 ## 12. References
 
-1. Shazeer, N., et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." *ICLR 2017*.
-2. Fedus, W., Zoph, B., & Shazeer, N. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." *JMLR 2022*.
-3. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. "Adaptive Mixtures of Local Experts." *Neural Computation 1991*.
+1. Shazeer, N., et al. "Outrageously Large Neural Networks: The Sparsely-Gated
+   Mixture-of-Experts Layer." *ICLR 2017*.
+2. Fedus, W., Zoph, B., & Shazeer, N. "Switch Transformers: Scaling to Trillion
+   Parameter Models with Simple and Efficient Sparsity." *JMLR 2022*.
+3. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. "Adaptive
+   Mixtures of Local Experts." *Neural Computation 1991*.
 4. Jiang, A. Q., et al. "Mixtral of Experts." *arXiv:2401.04088*, 2024.
 5. Yang, A., et al. "Qwen2 Technical Report." *arXiv:2407.10671*, 2024.
-6. Dai, D., et al. "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models." *arXiv:2401.06066*, 2024.
-7. Hu, E. J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." *ICLR 2022*.
-8. Arcee AI. "mergekit: Tools for Merging Pretrained Large Language Models." *GitHub: arcee-ai/mergekit*, 2024.
-9. Gerganov, G. "llama.cpp: LLM Inference in C/C++." *GitHub: ggerganov/llama.cpp*, 2023.
+6. Dai, D., et al. "DeepSeekMoE: Towards Ultimate Expert Specialization in
+   Mixture-of-Experts Language Models." *arXiv:2401.06066*, 2024.
+7. Hu, E. J., et al. "LoRA: Low-Rank Adaptation of Large Language Models."
+   *ICLR 2022*.
+8. Arcee AI. "mergekit: Tools for Merging Pretrained Large Language Models."
+   *GitHub: arcee-ai/mergekit*, 2024.
+9. Labonne, M. "Create a Frankenstein MoE with mergekit." *HuggingFace Blog*,
+   2024.
+10. Gerganov, G. "llama.cpp: LLM Inference in C/C++." *GitHub: ggerganov/llama.cpp*,
+    2023.
 
 ---