chenliforai/mars2-saucy-aet-grpo

AET-GRPO — Team Saucy, MARS2 2026 (ECCV 2026), VTG track

Weights for Team Saucy's entry to the Video Temporal Grounding track of the MARS2 2026 Multimodal Reasoning Challenge. Score 0.5891353 (submission 577915) on the public leaderboard.

Code, reproduction instructions and the full pipeline: https://github.com/chenlicodebank/mars2-saucy-eccv2026 (branch main)

What is in this repository

Folder	What	Size
base_stage1_merged/	Qwen3-VL-4B-Instruct with our stage-1 Charades-STA GRPO adapter merged in. The adapter below attaches to this, not to the stock release.	8.3 GB
adapter_stage2/	The deployed model. Stage-2 LoRA adapter, step 600. Scored 0.5891353 (submission 577915).	2.2 GB

We also hold a later checkpoint that scored higher but is not shown on the public leaderboard, because it was evaluated only after the challenge went inactive. It is published separately at chenliforai/mars2-saucy-aet-grpo-requeued, with the matching code on the alternate-if-requeued-counted branch.

No MARS2 challenge data or predictions are included — the MARS2 2026 Data Use and Sharing License Agreement forbids redistributing the dataset or any derivative of it.

Method

Qwen3-VL-4B-Instruct post-trained with Group Relative Policy Optimization on a pure temporal-IoU reward, reading an Audio-Evidence Timeline: word-level ASR (faster-whisper large-v3, rendered word-fine) and non-speech sound events (BEATs / AudioSet-Strong) as timestamped text beside frames sampled at 2 fps under a whole-video budget of 14,680,064 pixels.

LoRA (r=64, alpha=16, dropout 0.05) is the only trainable component; the vision tower, language model and merger are frozen. Two stages, both beta=0 (no KL), constant LR 2e-6, batch 1 × gradient accumulation 8, 8 generations per prompt, bf16:

1. Transfer — 1,800 steps on a 2,500-clip Charades-STA band, seed 42 (6.5 h + 3.2 h on one GPU). Merged into the base to give base_stage1_merged/.
2. Domain — 600 steps on a 1,164-row mixture of Charades and evidence-checked advertisement boundaries, total_tokens 17408 (9.1 h on one GPU). This is adapter_stage2/.

Total training cost 18.9 GPU-hours. One full pass over the 3,108-item test set takes ≈2.4 h at 2.8 s/video on one GPU.

Reward-label quality, not reward shaping, was the dominant factor: replacing model-voted boundaries with evidence-checked ones was the change that moved the score once the recipe had settled.

Usage — the adapter must NOT be merged

from transformers import Qwen3VLForConditionalGeneration
from peft import PeftModel

REPO = "chenliforai/mars2-saucy-aet-grpo"
base = Qwen3VLForConditionalGeneration.from_pretrained(
    REPO, subfolder="base_stage1_merged", dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(base, REPO, subfolder="adapter_stage2")
model.eval()          # do NOT call merge_and_unload()

merge_and_unload() computes W + BA*scale and casts the result back to bf16. For this rank-64 adapter, 97.3% of the trained delta does not survive that cast — the elementwise update is smaller than the bf16 ulp of the corresponding base weight. Keeping the adapter as a live low-rank branch is worth +0.001 on the leaderboard over the merged copy of the same checkpoint. Every number we reported was produced with --no-merge-adapter.

Note that target_modules includes embed_tokens and lm_head, which is why the adapter is larger than a typical rank-64 LoRA.

Prompt format

The model expects the timeline in the prompt, not raw audio. Build the ASR and SED stores with scripts/asr_mars2.py and scripts/sed_mars2.py from the code repository, then run scripts/run_zeroshot_vtg.py --asr-format word-fine. The exact prompt construction lives in that script; a mismatched prompt format costs several points, because the model was trained inside one.

Citation

Please cite the MARS2 2026 challenge report. Underlying components: Qwen3-VL, GRPO (DeepSeekMath), LoRA, Whisper, PretrainedSED, Charades-STA, AdsQA, and the TimeLens GRPO trainer, which this work vendors.

Acknowledgements

We acknowledge the use of the MARS2 Dataset provided by the MARS2 2026 Challenge (https://eval.ai/web/challenges/challenge-page/4688/overview).