Qwen3.5-4B DCI RL Reward-V2 Step 10 LoRA
This private research checkpoint is a PEFT LoRA adapter for
Qwen/Qwen3.5-4B. It was initialized from
eigentom/qwen35-4b-lora-reasoningcore3384-epoch4-ckpt850 at revision
60421d289c38442f5aa99b5ac77d6d44c5b74484, then updated with the DCI
deep-search RL Reward-V2 experiment.
Checkpoint identity
- RL run:
qwen35_4b_rewardv2_bsz512_full_20260727 - verl checkpoint:
global_step_10 - Base model revision:
851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a - LoRA rank / alpha / dropout:
64 / 128 / 0.05 - Task type:
CAUSAL_LM - Adapter SHA-256:
d85161af687e56e15e68904943ef1c1d115a9399e8c4241357334abc8ca48a1c
global_step_10 means ten global RL updates, not ten epochs. Each update used
64 question groups with eight rollouts per question, or 512 trajectories per
step. This is roughly one quarter of the 2,567-question curriculum.
Observed behavior
On the fixed 114-question validation set, answer correctness increased from 7.89% at step 0 to 20.18% at step 10. This checkpoint is nevertheless an intermediate research artifact: the Reward-V2 run also showed late-step degradation in stopping behavior, parseability, and length-boundary rate. It should not be treated as a production-quality agent.
Loading
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.5-4B",
torch_dtype="auto",
device_map="auto",
)
model = PeftModel.from_pretrained(
base,
"eigentom/qwen35-4b-dci-rl-rewardv2-step10",
)
The original 8-way FSDP checkpoint also retains optimizer, scheduler, RNG, and dataloader state for exact verl continuation. Those training-state shards are not included in this Hugging Face adapter repository.
- Downloads last month
- 6