rl-clarify-orig-prompt-d1-0p75

LoRA fine-tune of Qwen2.5-Coder-7B-Instruct trained with PPO-Lagrangian constrained RL on HumanEvalComm.

Training setup

• Algorithm: PPO with Lagrangian constraint on avg questions per episode
• LoRA rank: 16, alpha 32
• Question budget (d1): 0.75
• Iterations: 80
• Checkpoint dir: checkpoints/orig_prompt_v2/d1_0.75

Eval results (selected checkpoint: iter_0039)

• Final eval pass@1: 0.748 (417 problems, greedy decoding)
• Final eval avg questions: 0.7

Checkpoints

Each iter_XXXX/ folder contains LoRA adapter weights and a log.json with per-iteration training metrics (avg_reward, avg_questions, lambda1, lambda2, kl_per_seq).

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base, "acv1229/rl-clarify-orig-prompt-d1-0p75", subfolder="iter_0039")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")