File size: 1,751 Bytes
330e205 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | ---
base_model: Qwen/Qwen3.5-0.8B
library_name: peft
tags:
- lora
- peft
- knowledge-distillation
- gkd
---
# qwen3.5-0.8b-finance-lora (LoRA adapter)
LoRA adapter distilled from **Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter** into **Qwen/Qwen3.5-0.8B** using
[Generalized Knowledge Distillation](https://arxiv.org/abs/2306.13649) (GKD).
This repo holds the **adapter only**. For a single ready-to-run checkpoint see
[`siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance`](https://huggingface.co/siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance).
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")
tok = AutoTokenizer.from_pretrained("siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")
messages = [{"role": "user", "content": "How does compound interest work?"}]
inputs = tok.apply_chat_template(messages, return_tensors="pt",
add_generation_prompt=True)
print(tok.decode(model.generate(inputs, max_new_tokens=128)[0]))
```
## Training
| | |
|---|---|
| Student (base) | `Qwen/Qwen3.5-0.8B` |
| Teacher | `Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter` |
| Dataset | `gbharti/finance-alpaca` |
| Method | GKD (on-policy, JSD loss) |
| LoRA rank / alpha | 32 / 64 |
| Target modules | `down_proj`, `gate_proj`, `in_proj_qkv`, `in_proj_z`, `k_proj`, `o_proj`, `out_proj`, `q_proj`, `up_proj`, `v_proj` |
| Steps | 300 |
| Effective batch | 4 |
| Learning rate | 0.0002 |
| GKD lmbda / beta | 0.5 / 0.5 |
|