README.md

---
base_model: Qwen/Qwen3.5-0.8B
library_name: peft
tags:
- lora
- peft
- knowledge-distillation
- gkd
---

# qwen3.5-0.8b-finance-lora (LoRA adapter)

LoRA adapter distilled from **Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter** into **Qwen/Qwen3.5-0.8B** using
[Generalized Knowledge Distillation](https://arxiv.org/abs/2306.13649) (GKD).

This repo holds the **adapter only**. For a single ready-to-run checkpoint see
[`siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance`](https://huggingface.co/siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance).

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")
tok = AutoTokenizer.from_pretrained("siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")

messages = [{"role": "user", "content": "How does compound interest work?"}]
inputs = tok.apply_chat_template(messages, return_tensors="pt",
                                 add_generation_prompt=True)
print(tok.decode(model.generate(inputs, max_new_tokens=128)[0]))
```

## Training

| | |
|---|---|
| Student (base) | `Qwen/Qwen3.5-0.8B` |
| Teacher | `Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter` |
| Dataset | `gbharti/finance-alpaca` |
| Method | GKD (on-policy, JSD loss) |
| LoRA rank / alpha | 32 / 64 |
| Target modules | `down_proj`, `gate_proj`, `in_proj_qkv`, `in_proj_z`, `k_proj`, `o_proj`, `out_proj`, `q_proj`, `up_proj`, `v_proj` |
| Steps | 300 |
| Effective batch | 4 |
| Learning rate | 0.0002 |
| GKD lmbda / beta | 0.5 / 0.5 |