amayuelas/Qwen3.5-4B-MatRL-MT-RL

Qwen3.5-4B — MatRL multi-turn RL

Multi-turn RL on top of amayuelas/Qwen3.5-4B-MatRL-MT-SFT, for crystal-structure inverse design. The agent proposes candidate structures, evaluates them against a machine-learning interatomic potential, refines, and submits — rewarded on validity, novelty, and thermodynamic stability.

Results

SUN rate (Stable, Unique, Novel) — n=100 prompts × r=8 = 800 rollouts/track. equflashv2 is the reward-path judge and therefore the reference; orb-mpa and eqv3 are held-out judges that never touched training. Judging protocol is independent of the reward path (200 relaxation steps, fmax 0.02).

track	equflashv2 (reference)	orb-mpa	eqv3	held-out consensus
mt_id (in-distribution)	15.5%	16.8%	17.8%	14.4%
mt_ood (out-of-distribution)	15.1%	17.4%	17.9%	15.2%
mt_uncond (unconditional)	17.0%	20.1%	24.6%	19.8%

The base model scores 0% on these tracks under strict scoring — it almost never calls submit (3 times in 2,880 rollouts), so nothing is reconstructed to score. Learning to commit within the turn budget is most of the gap.

On reward hacking

The policy is optimized against equflashv2, so the number that matters is whether judges it never saw agree:

track	reward judge	held-out consensus	gap
mt_id	15.5%	14.4%	+1.1 pp
mt_ood	15.1%	15.2%	−0.1 pp
mt_uncond	17.0%	19.8%	−2.8 pp

Held-out judges track the reward judge closely and exceed it on the unconditional track. The gains are not artifacts of the training judge.

Training

init	amayuelas/Qwen3.5-4B-MatRL-MT-SFT
algorithm	GRPO-style multi-turn RL, 10 turns / 8 evaluations per episode
reward	validity + novelty + relaxed stability (equflashv2 MLIP, 20 steps / fmax 0.1)
steps	200
seq len	16,384
precision	bf16
trainer	prime-rl

Trained on 8×A100-40GB.

Known training caveat

Steps ~150–200 ran with 9–15% of each batch clipped at the trainer's 16,384-token window, so those episodes' endings — including the submit turn — contributed no gradient. This checkpoint is released as-is because the held-out evaluation above shows it is not inflated: independent judges match or exceed the reward judge on every track. A truncation-clean redo of the final 50 steps was trained separately and scored lower (13.5 / 12.5 / 10.8%), so the clipping does not appear to have produced a falsely strong model.

Important: thinking channel

Reasoning lives in the native thinking channel, preserved across tool calls. Do not sample with enable_thinking=false — it disables the behavior this model was trained to use.

Serving

Includes preprocessor_config.json / video_preprocessor_config.json — vLLM requires them for this VL-capable model class even for text-only serving.