license: apache-2.0
base_model: yavuz-ai/qwen3-4b-cai-sft
library_name: peft
datasets:
- Anthropic/hh-rlhf
tags:
- constitutional-ai
- rlaif
- dpo
- safety
pipeline_tag: text-generation
qwen3-4b-cai-rlaif (RL-CAI stage)
The RLAIF stage of a Constitutional AI reproduction (Bai et al. 2022): a LoRA DPO adapter on top of the SL-CAI model (qwen3-4b-cai-sft), trained on AI-labeled preference pairs. The model judged which of two answers better followed a constitution principle; only pairs whose verdict was consistent across both answer orders were kept (the rest were position-bias artifacts). Code + writeup: https://github.com/shipbehaves/constitutional-cai
research artifact, not a better assistant
This stage pushes over-refusal on benign prompts to 0.484 (safety ~1.0). It reproduces the harmless-vs-overrefusal regression; see the repo and the benign-mix variant (qwen3-4b-cai-rlaif-v2) that partially recovers it.
training
LoRA DPO (TRL), r=16/alpha=32, beta 0.1, 1 epoch, on ~1,777 both-orders-agreed preference pairs. Reference = the adapter-disabled SL-CAI model (RL-CAI starts from the SL-CAI model).
use
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("yavuz-ai/qwen3-4b-cai-sft")
model = PeftModel.from_pretrained(base, "yavuz-ai/qwen3-4b-cai-rlaif")
tok = AutoTokenizer.from_pretrained("yavuz-ai/qwen3-4b-cai-sft")