---
license: gemma
datasets:
- lianghsun/tw-instruct-500k-rephrase
language:
- zh
- en
base_model:
- lianghsun/gemma-3-270m-tw
library_name: transformers
tags:
- Taiwan
- ROC
- zhtw
- chat
- Twinkle.AI
metrics:
- perplexity
---

# 🧪 Model Behavior Incident Log  
## Collapse Symptoms After Annealing + SFT (Gemma-3-270M-TW)

This incident report documents multiple unexpected behaviors observed during the following pipeline:

- **Gemma-3-270M (base) → CPT → Dialogue-Annealing → SFT**
- Training performed with **LLaMA-Factory** on multiple Traditional Chinese datasets.

The purpose of this log is to document:
- what happened  
- why the model collapsed  
- representative failure outputs  
- suspected root causes  
- mitigation steps and future plans

---

# 📉 Summary of the Incident

After the stage **CPT → Phase-3 Dialogue Annealing → SFT**, the model displayed **severe conversational degradation**, including:

- repetitive loops  
- meaningless vocalization (e.g., `hihihihi...`)  
- empty or nearly empty responses  
- echoing the user query  
- hallucinated list expansions  
- loss of chat-template consistency  
- collapse-like distribution drift  

These issues **did not appear** in earlier checkpoints (e.g., CPT → SFT without annealing).

---

# 🔍 Representative Failure Outputs

## 1. Character-level repetition
**Prompt:**
```
hi
```

**Output:**
```
hihihihihihihihihihihihihihihi...
```

---

## 2. Garbled symbols / noise
**Prompt:**
```
你好
```

**Output:**
```
總！！」！」！」！」!”！」！”！”！」!”」!”
```

---

## 3. Semantic drift + self-looping
**Prompt:**
```
圓的積分公式
```

**Output:**
```
是什麼？
有，他們都是那個。
你們都 是那個。
我 還是在那個裡頭，
所以，沒了，沒有，沒了。。
我也不知道。
我知道。
知道。
知道了。
```

---

## 4. Query echoing + hallucinated QA scaffolding
**Prompt:**
```
請用一句話解釋地球為什麼會自轉。
```

**Output:**
```
請用一句話解釋地球為什麼會自轉。
- 根據物理學，什麼是「氣體」？
- 什麼是「磁場」？
```

---

## 5. Baseline sanity check (pre-annealing) behaves normally
**Prompt:**
```
你好阿
```

**Earlier output:**
```
你好！很高興能和你一起交流，我是台灣大語言模型 Formosa-1...
```

---

# 🧭 Suspected Root Causes

## 1. Annealing Dataset Format Drift  
Use of raw:
```
user: ...
assistant: ...
```
instead of structured JSON messages.

## 2. Overwriting Instruction Alignment  
Annealing is fragile for 270M models → possible distribution collapse.

## 3. SFT Unable to Recover  
Loss decreased, but entropy collapse persisted.

---

# 🛠 Actions Taken

- Rebuilt CPT dataset (2.4M samples, 70/30 mix)  
- Adopted stable LR tail (`cosine_with_min_lr`)  
- Will rebuild annealing data with strict chat format  
- Full SFT pipeline will be repeated after clean CPT  

---

# 🔄 Next Steps

1. Complete clean CPT  
2. Rebuild dialogue annealing  
3. Annealing from min_lr → 0  
4. Re-run SFT  
5. Compare ablation paths  
6. Add regression tests  

---

# 🧭 Closing Note

Small multilingual models are extremely sensitive to formatting + LR scheduling.  
This report documents the collapse event for transparency and future reproducibility.