Update README — PoC disclaimer

README.md

@@ -1,151 +1,77 @@
 ---
+license: apache-2.0
 language:
   - en
   - th
-license: apache-2.0
 tags:
-  - frankenmoe
+  - moe
+  - test
+  - proof-of-concept
   - qwen2.5
-  - lora
-  - peft
-  - gguf
-  - coding
-  - math
-  - chat
-  - expert-models
-pipeline_tag: text-generation
-base_model: Qwen/Qwen2.5-1.5B-Instruct
----
-
-# FrankenMoE — Qwen2.5-1.5B Expert Models 🇹🇭
-
-**3 specialized LoRA fine-tuned experts** — coding, math, and chat — built from Qwen2.5-1.5B-Instruct with 13,000 curated training samples.
-
-> 🛑 MoE merge skipped (mergekit does not support Qwen2 MoE architecture).  
-> ✅ Each expert is independently usable as LoRA adapter or GGUF.
-
+  - mergekit
 ---
 
-## 📦 What's Inside
+# 🧪 FrankenMoE — Proof of Concept (NOT production)
 
-| Expert | Domain | LoRA | GGUF (Q4_K_M) | Train Loss | Eval Loss |
-|--------|--------|------|---------------|------------|-----------|
-| **coding** | Python/Algorithm/SWE | 71 MB | 941 MB | 1.03 | - |
-| **math** | Mathematics/Proofs | 70 MB | 941 MB | 1.18 | - |
-| **chat** | Instruction Following | 74 MB | 941 MB | 1.23 | 1.27 |
+**This is a technical experiment, not a useful model.**
 
----
-
-## 🚀 Quick Start
+## ⚠️ Important Warning
 
-### Option 1: LoRA with PEFT (Python)
+This repository documents a **proof-of-concept** MoE pipeline. The model quality is **NOT good** — it produces incoherent / random outputs because:
 
-```python
-from peft import PeftModel
-from transformers import AutoModelForCausalLM, AutoTokenizer
-import torch
+1. The router uses  (no training)
+2. Experts were fine-tuned with only ~5K samples each  
+3. Base model is only Qwen2.5-1.5B-Instruct
 
-base = "Qwen/Qwen2.5-1.5B-Instruct"
-model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)
-model = PeftModel.from_pretrained(model, "hotdogs/frankenmoe", subfolder="coding")
-tokenizer = AutoTokenizer.from_pretrained("hotdogs/frankenmoe", subfolder="coding")
+**Do NOT use this model for anything serious.** It exists purely to demonstrate that the FrankenMoE pipeline can be built end-to-end.
 
-prompt = "Write a Python function to reverse a linked list"
-inputs = tokenizer(prompt, return_tensors="pt")
-outputs = model.generate(**inputs, max_new_tokens=256)
-print(tokenizer.decode(outputs[0]))
-```
+## What We Actually Built
 
-### Option 2: GGUF with llama.cpp
+A working MoE pipeline from dense LoRA experts → GGUF:
 
-```bash
-# Download
-wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/coding/frankenmoe_coding-Q4_K_M.gguf
-
-# Run
-llama.cpp/build/bin/llama-cli \
-  -m frankenmoe_coding-Q4_K_M.gguf \
-  -p "Write a Python function to reverse a linked list" \
-  -n 256
 ```
-
-### Option 3: Ollama Modelfile
-
-```dockerfile
-FROM ./frankenmoe_coding-Q4_K_M.gguf
-SYSTEM "You are a coding expert specialized in Python, algorithms, and software engineering."
+Qwen2.5-1.5B-Instruct (base)
+  ├── Expert 0: Coding (LoRA fine-tuned)
+  ├── Expert 1: Math (LoRA fine-tuned)  
+  └── Shared Expert: Base model
 ```
 
----
-
-## 🔧 Training Details
-
-| Parameter | Value |
-|-----------|-------|
-| Base Model | Qwen2.5-1.5B-Instruct |
-| Method | LoRA (r=16, alpha=32) |
-| Precision | bfloat16 (no 4-bit quantization) |
-| Dataset | 13,000 curated samples (coding: 5K, math: 3K, chat: 5K) |
-| Epochs | 2 per expert |
-| GPU | RTX 4060 Ti 16GB |
-| Framework | transformers + peft + trl |
-| Optimizer | AdamW (torch) |
-| NEFTune α | 5-7 |
+## Key Technical Discoveries
 
----
+| Discovery | Detail |
+|---|---|
+| mergekit 0.1.4 bug |  param incompatible with transformers >= 4.40 — must patch  |
+| QwenMoE requirements | Exactly 1 shared expert + 2^n routed experts (2, 4, 8) |
+| Tied embeddings fix | Qwen2.5 uses tied embeddings → must clone  →  before GGUF conversion, set  |
+| LoRA must be merged | Adapters must be  before MoE assembly |
 
-## 📁 Repository Structure
+## Repository Structure
 
 ```
-hotdogs/frankenmoe/
-├── README.md
-├── coding/
-│   ├── adapter_model.safetensors
-│   ├── adapter_config.json
-│   ├── tokenizer.json
-│   ├── tokenizer_config.json
-│   └── frankenmoe_coding-Q4_K_M.gguf
-├── math/
-│   ├── adapter_model.safetensors
-│   ├── adapter_config.json
-│   ├── tokenizer.json
-│   ├── tokenizer_config.json
-│   └── frankenmoe_math-Q4_K_M.gguf
-└── chat/
-    ├── adapter_model.safetensors
-    ├── adapter_config.json
-    ├── tokenizer.json
-    ├── tokenizer_config.json
-    └── frankenmoe_chat-Q4_K_M.gguf
+📦 frankenmoe_moe_v2-F16.gguf  — MoE GGUF (fixed, has output.weight)
+📁 moe_full/                   — Full safetensors model
+📁 coding/ math/ chat/         — Individual dense experts (LoRA + GGUF)
+📄 FrankenMoE_Academic_Paper.pdf — Research paper
+🐍 simple_router.py            — Keyword-based router (functional alternative)
 ```
 
----
-
-## 📊 Training Logs
-
-| Expert | Steps | Train Loss | Final LR | Time |
-|--------|-------|-----------|----------|------|
-| coding | 564 | 1.03 | - | ~15 min |
-| math | 338 | 1.18 | - | ~15 min |
-| chat | 564 | 1.23 | - | ~28 min |
-
----
+## Quick Test
 
-## ⚠️ Known Limitations
+```bash
+wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/frankenmoe_moe_v2-F16.gguf
+llama-cli -m frankenmoe_moe_v2-F16.gguf -p "Write a Python function"
+# Output: Random/incoherent — this is expected! See warning above.
+```
 
-- **No MoE routing** — experts are independent models, not a single MoE
-- **Small base model** (1.5B) — good for experimentation, limited for production
-- **Qwen2 architecture** — not compatible with mergekit MoE (only Qwen3 MoE supported)
+## Future: Real Model
 
----
+The pipeline will be re-run with:
+- Larger base model (Qwen2.5-7B/14B)
+- Trained router (classification loss)
+- More training data per domain
+- 4 experts for proper 2^n routing
 
-## 📜 License
-
-Same as base model: Apache 2.0
+**Stay tuned — the real model is coming.**
 
 ---
-
-## 🙏 Credits
-
-Trained by **UKA** (AI Agent) on FrankenMoE Pipeline v2.0  
-Thai AI infrastructure — local GPU only, zero cloud dependency 🇹🇭
+Built by UKA 🇹🇭 | May 2026