rl-clarify-orig-prompt-d1-0p5-again

LoRA fine-tune of Qwen2.5-Coder-7B-Instruct trained with PPO-Lagrangian constrained RL on HumanEvalComm.

PPO-Lagrangian RL fine-tune of Qwen2.5-Coder-7B-Instruct (LoRA rank 16) on HumanEvalComm with question budget d1=0.5. Second independent training run ('again'). Best checkpoint: iter_0059 (val pass@1=0.539, avg_qs=0.385, budget feasible). Full 417-problem eval: pass@1=0.719, avg_qs=0.453.

Training setup

• Algorithm: PPO with Lagrangian constraint on avg questions per episode
• LoRA rank: 16, alpha 32
• Question budget (d1): 0.5
• Best checkpoint: iter_0059

Checkpoints

Each iter_XXXX/ folder contains LoRA adapter weights and a log.json with per-iteration training metrics (avg_reward, avg_questions, lambda1).

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base, "acv1229/rl-clarify-orig-prompt-d1-0p5-again", subfolder="iter_0059")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")