rl-clarify-orig-prompt-d1-1

LoRA fine-tune of Qwen2.5-Coder-7B-Instruct trained with PPO-Lagrangian constrained RL on HumanEvalComm.

PPO-Lagrangian RL fine-tune of Qwen2.5-Coder-7B-Instruct (LoRA rank 16) on HumanEvalComm with question budget d1=1. All iter_* checkpoints included. Eval checkpoint: iter_0079 (80 iterations completed).

Training setup

• Algorithm: PPO with Lagrangian constraint on avg questions per episode
• LoRA rank: 16, alpha 32
• Question budget (d1): 1.0
• Eval checkpoint: iter_0079
• Eval set: 469 problems (full HumanEvalComm eval split, greedy decoding)

Checkpoints

Each iter_XXXX/ folder contains LoRA adapter weights and a log.json with per-iteration training metrics (avg_reward, avg_questions, lambda1).

Training was run in two segments (job preemption on HPC); checkpoints are merged from both segments in chronological order.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base, "acv1229/rl-clarify-orig-prompt-d1-1", subfolder="iter_0079")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")