qwen3-4b-cai-rlaif (RL-CAI stage)

The RLAIF stage of a Constitutional AI reproduction (Bai et al. 2022): a LoRA DPO adapter on top of the SL-CAI model (qwen3-4b-cai-sft), trained on AI-labeled preference pairs. The model judged which of two answers better followed a constitution principle; only pairs whose verdict was consistent across both answer orders were kept (the rest were position-bias artifacts). Code + writeup: https://github.com/shipbehaves/constitutional-cai

research artifact, not a better assistant

This stage pushes over-refusal on benign prompts to 0.484 (safety ~1.0). It reproduces the harmless-vs-overrefusal regression; see the repo and the benign-mix variant (qwen3-4b-cai-rlaif-v2) that partially recovers it.

training

LoRA DPO (TRL), r=16/alpha=32, beta 0.1, 1 epoch, on ~1,777 both-orders-agreed preference pairs. Reference = the adapter-disabled SL-CAI model (RL-CAI starts from the SL-CAI model).

use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("yavuz-ai/qwen3-4b-cai-sft")
model = PeftModel.from_pretrained(base, "yavuz-ai/qwen3-4b-cai-rlaif")
tok = AutoTokenizer.from_pretrained("yavuz-ai/qwen3-4b-cai-sft")