Barryzbr12/qwen2.5-7b-instruct-dpo-lima-lora

Qwen2.5-7B-Instruct DPO LoRA adapter (LIMA × PairRM)

This is a LoRA adapter produced by Direct Preference Optimization (DPO) applied to Qwen/Qwen2.5-7B-Instruct. It was trained on the preference dataset Barryzbr12/lima-qwen2.5-7b-pairrm-preferences, which was constructed for Assignment 4 of the alignment course by (1) sampling 50 LIMA instructions, (2) generating 5 responses each from Qwen2.5-7B-Instruct, and (3) ranking them with llm-blender/PairRM.

Training configuration

Base model	Qwen/Qwen2.5-7B-Instruct
Method	DPO (sigmoid loss, β = 0.1)
LoRA rank / alpha	16 / 32
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters	40.4 M (~0.53 % of full model)
Epochs / Effective batch	3 / 8
Learning rate	5e-6, cosine schedule, 10 % warmup
Sequence length	2048
Precision	bf16
Seed	42

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base, "Barryzbr12/qwen2.5-7b-instruct-dpo-lima-lora")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "How to make your dog more playful?"}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tok(prompt, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=512)[0], skip_special_tokens=True))

Author

Barryzbr12 — Assignment 4 (PairRM + DPO).