qwen3.5-0.8b-finance-lora (LoRA adapter)

LoRA adapter distilled from Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter into Qwen/Qwen3.5-0.8B using Generalized Knowledge Distillation (GKD).

This repo holds the adapter only. For a single ready-to-run checkpoint see siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")
tok = AutoTokenizer.from_pretrained("siddhartha-addy-globalids-labs/qwen3.5-0.8b-finance-lora")

messages = [{"role": "user", "content": "How does compound interest work?"}]
inputs = tok.apply_chat_template(messages, return_tensors="pt",
                                 add_generation_prompt=True)
print(tok.decode(model.generate(inputs, max_new_tokens=128)[0]))

Training

Student (base)	Qwen/Qwen3.5-0.8B
Teacher	Qwen/Qwen3.5-2B + /Users/globalids/.cache/kd-runner/peft-adapter
Dataset	gbharti/finance-alpaca
Method	GKD (on-policy, JSD loss)
LoRA rank / alpha	32 / 64
Target modules	down_proj, gate_proj, in_proj_qkv, in_proj_z, k_proj, o_proj, out_proj, q_proj, up_proj, v_proj
Steps	300
Effective batch	4
Learning rate	0.0002
GKD lmbda / beta	0.5 / 0.5