chenliforai/mars2-saucy-aet-grpo-requeued

AET-GRPO (clock-repair continuation) — Team Saucy, MARS2 2026 (ECCV 2026), VTG track

Weights for Team Saucy's best evaluated entry to the Video Temporal Grounding track of the MARS2 2026 Multimodal Reasoning Challenge. Score 0.5942413 (submission 578199).

That submission was accepted before the deadline but only evaluated after the challenge went inactive, so it is not shown on the public leaderboard; the displayed entry is submission 577915 at 0.5891353, whose adapter is published at chenliforai/mars2-saucy-aet-grpo.

Code, reproduction instructions and the full pipeline: https://github.com/chenlicodebank/mars2-saucy-eccv2026 (branch alternate-if-requeued-counted)

What is in this repository

Folder	What	Size
base_stage1_merged/	Qwen3-VL-4B-Instruct with our stage-1 Charades-STA GRPO adapter merged in. The adapter below attaches to this, not to the stock release.	8.3 GB
adapter_stage3_clockrepair/	The deployed model. Stage-3 clock-repair LoRA adapter, step 65. Scored 0.5942413 (submission 578199).	2.2 GB

No MARS2 challenge data or predictions are included — the MARS2 2026 Data Use and Sharing License Agreement forbids redistributing the dataset or any derivative of it.

Method

Qwen3-VL-4B-Instruct post-trained with Group Relative Policy Optimization on a pure temporal-IoU reward, reading an Audio-Evidence Timeline: word-level ASR (faster-whisper large-v3, rendered word-fine) and non-speech sound events (BEATs / AudioSet-Strong) as timestamped text beside frames sampled at 2 fps under a whole-video budget of 14,680,064 pixels.

LoRA (r=64, alpha=16, dropout 0.05) is the only trainable component; the vision tower, language model and merger are frozen. Three stages, all beta=0 (no KL), batch 1 × gradient accumulation 8, 8 generations per prompt, bf16:

1. Transfer — 1,800 steps on a 2,500-clip Charades-STA band, LR 2e-6, seed 42 (6.5 h + 3.2 h on one GPU). Merged into the base to give base_stage1_merged/.
2. Domain — 600 steps on a 1,164-row mixture of Charades and evidence-checked advertisement boundaries, LR 2e-6, total_tokens 17408 (9.1 h on one GPU).
3. Clock repair — continues from stage 2 on a rebuild of those same 1,164 rows with the word-level transcript clock corrected (identical ids, identical gold spans), LR 5e-6, DDP across 3 GPUs with a fourth reserved as an evaluation meter, checkpointing every five steps. Step 65 is the argmax on a held-out human-labelled meter (2.1 h wall-clock ≈ 6.2 GPU-h). This is adapter_stage3_clockrepair/.

Total training cost 18.9 + 6.2 GPU-hours. One full pass over the 3,108-item test set took 55 min 4-way across 4 GPUs at 4.3 s/video (≈3.7 GPU-h).

The clock-repair stage is the interesting one: the advertisement transcripts had been reaching the model on a coarser clock than the gold spans are defined against, so the model was being rewarded for quoting boundaries it could not actually see. Fixing the clock and re-training for 65 steps was worth +0.0051, more than any architectural or decoding change we tried after the recipe settled.

Usage — the adapter must NOT be merged

from transformers import Qwen3VLForConditionalGeneration
from peft import PeftModel

REPO = "chenliforai/mars2-saucy-aet-grpo-requeued"
base = Qwen3VLForConditionalGeneration.from_pretrained(
    REPO, subfolder="base_stage1_merged", dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(base, REPO, subfolder="adapter_stage3_clockrepair")
model.eval()          # do NOT call merge_and_unload()

merge_and_unload() computes W + BA*scale and casts the result back to bf16. For this rank-64 adapter, 97.3% of the trained delta does not survive that cast — the elementwise update is smaller than the bf16 ulp of the corresponding base weight. Keeping the adapter as a live low-rank branch is worth +0.001 on the leaderboard over the merged copy of the same checkpoint. Every number we reported was produced with --no-merge-adapter.

Note that target_modules includes embed_tokens and lm_head, which is why the adapter is larger than a typical rank-64 LoRA.

Prompt format

The model expects the timeline in the prompt, not raw audio. Build the ASR and SED stores with scripts/asr_mars2.py and scripts/sed_mars2.py from the code repository, then run scripts/run_zeroshot_vtg.py --asr-format word-fine. The exact prompt construction lives in that script; a mismatched prompt format costs several points, because the model was trained inside one.

Citation

Please cite the MARS2 2026 challenge report. Underlying components: Qwen3-VL, GRPO (DeepSeekMath), LoRA, Whisper, PretrainedSED, Charades-STA, AdsQA, and the TimeLens GRPO trainer, which this work vendors.

Acknowledgements

We acknowledge the use of the MARS2 Dataset provided by the MARS2 2026 Challenge (https://eval.ai/web/challenges/challenge-page/4688/overview).