README.md

---
library_name: peft
base_model: Qwen/Qwen2.5-7B-Instruct
tags:
  - dpo
  - alignment
  - peft
  - lora
  - lima
license: apache-2.0
---

# Qwen2.5-7B-Instruct DPO LoRA adapter (LIMA × PairRM)

This is a LoRA adapter produced by **Direct Preference Optimization (DPO)**
applied to [`Qwen/Qwen2.5-7B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct).
It was trained on the preference dataset
[`Barryzbr12/lima-qwen2.5-7b-pairrm-preferences`](https://huggingface.co/datasets/Barryzbr12/lima-qwen2.5-7b-pairrm-preferences),
which was constructed for Assignment 4 of the alignment course by
(1) sampling 50 LIMA instructions, (2) generating 5 responses each from
Qwen2.5-7B-Instruct, and (3) ranking them with `llm-blender/PairRM`.

## Training configuration

| | |
|---|---|
| Base model | `Qwen/Qwen2.5-7B-Instruct` |
| Method | DPO (sigmoid loss, β = 0.1) |
| LoRA rank / alpha | 16 / 32 |
| LoRA target modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` |
| Trainable parameters | 40.4 M (~0.53 % of full model) |
| Epochs / Effective batch | 3 / 8 |
| Learning rate | 5e-6, cosine schedule, 10 % warmup |
| Sequence length | 2048 |
| Precision | bf16 |
| Seed | 42 |

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base, "Barryzbr12/qwen2.5-7b-instruct-dpo-lima-lora")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "How to make your dog more playful?"}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tok(prompt, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=512)[0], skip_special_tokens=True))
```

## Author

[`Barryzbr12`](https://huggingface.co/Barryzbr12) — Assignment 4 (PairRM + DPO).