metadata
base_model: Qwen/Qwen3.5-0.8B
library_name: peft
tags:
- lora
- peft
- knowledge-distillation
- gkd
qwen3.5-0.8b-finance-lora (LoRA adapter)
LoRA adapter distilled from Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter into Qwen/Qwen3.5-0.8B using Generalized Knowledge Distillation (GKD).
This repo holds the adapter only. For a single ready-to-run checkpoint see
siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")
tok = AutoTokenizer.from_pretrained("siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")
messages = [{"role": "user", "content": "How does compound interest work?"}]
inputs = tok.apply_chat_template(messages, return_tensors="pt",
add_generation_prompt=True)
print(tok.decode(model.generate(inputs, max_new_tokens=128)[0]))
Training
| Student (base) | Qwen/Qwen3.5-0.8B |
| Teacher | Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter |
| Dataset | gbharti/finance-alpaca |
| Method | GKD (on-policy, JSD loss) |
| LoRA rank / alpha | 32 / 64 |
| Target modules | down_proj, gate_proj, in_proj_qkv, in_proj_z, k_proj, o_proj, out_proj, q_proj, up_proj, v_proj |
| Steps | 300 |
| Effective batch | 4 |
| Learning rate | 0.0002 |
| GKD lmbda / beta | 0.5 / 0.5 |