Upload folder using huggingface_hub

README.md

@@ -0,0 +1,151 @@
+---
+language:
+- zh
+- nan
+license: apache-2.0
+base_model: yentinglin/Taiwan-LLM-7B-v2.1-chat
+tags:
+- taiwanese
+- taigi
+- reading-comprehension
+- lora
+- peft
+library_name: peft
+---
+
+# Taiwan-LLM Taigi Reading Comprehension LoRA
+
+台語閱讀理解任務的 LoRA adapter，基於 Taiwan-LLM-7B-v2.1-chat 微調。
+
+## 📊 模型效能
+
+- **Test Accuracy**: 73.33%
+- **Validation Loss**: 0.573 (最佳 checkpoint)
+- **Baseline**: 53.7% (2020 競賽最佳成績)
+- **提升**: +19.63 百分點
+
+## 🎯 模型資訊
+
+- **基礎模型**: yentinglin/Taiwan-LLM-7B-v2.1-chat
+- **訓練任務**: 台語閱讀理解多選題
+- **訓練資料**: 13,550 筆台語文章 + 12,195 筆 Alpaca 台語對話
+- **LoRA Rank**: 32
+- **LoRA Alpha**: 32
+- **Target Modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+
+## 💻 使用方法
+
+### 安裝依賴
+
+```bash
+pip install transformers peft torch unsloth
+```
+
+### 載入模型
+
+```python
+from unsloth import FastLanguageModel
+from peft import PeftModel
+
+# 1. 載入基礎模型
+base_model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name="yentinglin/Taiwan-LLM-7B-v2.1-chat",
+    max_seq_length=2048,
+    dtype=None,
+    load_in_4bit=True,
+)
+
+# 2. 載入 LoRA adapter
+model = PeftModel.from_pretrained(
+    base_model,
+    "YOUR_USERNAME/taiwan-llm-taigi-lora"  # 記得改成你的用戶名
+)
+
+# 3. 切換到推論模式
+FastLanguageModel.for_inference(model)
+```
+
+### 推論範例
+
+```python
+# 準備 prompt
+prompt = """USER: 請仔細閱讀以下台文文章，並回答問題。
+
+文章：
+明年元旦起，交通部新規定，汽車輪胎胎紋深度將納入定期檢驗項之一，一旦深度未達一.六公厘，佮一個月內無換胎，將會去予吊銷牌仔照。
+
+問題：汽車胎紋未到達多毫米將會予人吊銷牌?
+
+選項：
+1. 一.五公厘
+2. 一.四厘
+3. 一.三公厘
+4. 一.六公厘
+
+請回答： ASSISTANT:"""
+
+# 生成答案
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=10,
+    temperature=0.1,
+    do_sample=False,
+)
+
+answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(answer)  # 輸出: 4
+```
+
+## 🔧 訓練細節
+
+### 訓練超參數
+
+- Epochs: 3
+- Learning Rate: 2e-4 → 5e-5 (cosine decay)
+- Batch Size: 16 (4 per device × 4 gradient accumulation)
+- Optimizer: AdamW 8-bit
+- Weight Decay: 0.01
+- Warmup Ratio: 0.1
+
+### LoRA 配置
+
+```python
+{
+    "r": 32,
+    "lora_alpha": 32,
+    "lora_dropout": 0.05,
+    "bias": "none",
+    "target_modules": [
+        "q_proj", "k_proj", "v_proj", "o_proj",
+        "gate_proj", "up_proj", "down_proj"
+    ]
+}
+```
+
+### 訓練過程
+
+- Total Steps: ~45,500
+- Best Checkpoint: Step 30,000-31,000
+- Training Loss: 1.697 → 0.428
+- Validation Loss: 1.741 → 0.506
+
+## 📈 訓練曲線
+
+最佳驗證點在 Step 30,000 左右，之後出現輕微過擬合。
+
+## ⚠️ 限制
+
+- 僅適用於台語閱讀理解任務
+- 需要特定的 prompt 格式（USER: ... ASSISTANT:）
+- 繼承 Taiwan-LLM-7B 的所有限制
+- 對於非台語文本可能表現不佳
+
+## 📄 授權
+
+Apache 2.0
+
+## 🙏 致謝
+
+- 基礎模型: [Taiwan-LLM](https://huggingface.co/yentinglin/Taiwan-LLM-7B-v2.1-chat)
+- 訓練框架: [Unsloth](https://github.com/unslothai/unsloth)

adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "LlamaForCausalLM",
+    "parent_library": "transformers.models.llama.modeling_llama",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "yentinglin/Taiwan-LLM-7B-v2.1-chat",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "gate_proj",
+    "v_proj",
+    "up_proj",
+    "q_proj",
+    "k_proj",
+    "down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4edf1e8fba0510ad016b088922193585060e59e5a2bb1149728ad4fa55008117
+size 319876032

chat_template.jinja

@@ -0,0 +1 @@
+{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'].strip() %}{% else %}{% set loop_messages = messages %}{% set system_message = '你是人工智慧助理，以下是用戶和人工智能助理之間的對話。你要對用戶的問題提供有用、安全、詳細和禮貌的回答。' %}{% endif %}{{system_message + eos_token}}{% for message in loop_messages %}{% if message['role'] == 'user' %}USER: {{ message['content'].strip() + eos_token }}{% elif message['role'] == 'system' %}{{message['content'].strip() + eos_token}}{% elif message['role'] == 'assistant' %}ASSISTANT: {{ message['content'].strip() + eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{'ASSISTANT:'}}{% endif %}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 4096,
+  "pad_token": "<unk>",
+  "padding_side": "left",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}