---
license: apache-2.0
base_model: Qwen/Qwen2.5-VL-3B-Instruct
tags:
  - robotics
  - vision-language-action
  - vla
  - qwen2_5_vl
  - lerobot
  - maniskill
---

# BaseVLA — ManiSkill Franka (Qwen2.5-VL-3B), vision-only, with augmentation

Plain behaviour cloning baseline: **no V-JEPA, no latent alignment, no
decorrelation**. Just `L1(action)` on ManiSkill's merged Franka set, trained
**without proprioception** and with domain augmentation on.

Checkpoint at **step 40,000**. The run was stopped early at ~44k steps (of a
planned 100k) because the loss had flattened — 0.0487 at 30k to 0.0474 at 42k.

## Data

`maniskill-franka-merged` from [`Ngseo/stage1-data`](https://huggingface.co/datasets/Ngseo/stage1-data),
single camera (`observation.images.anchor`, 256×256 h264), 30 fps, action
`action.ee_delta_pose` (7-D).

**The raw set needs filtering.** 17.1% of its `action.ee_delta_pose` rows are
NaN, and they are not spread evenly — whole tasks were never populated:

| task | frames | NaN (ee_delta) | NaN (joint_delta) | NaN (joint_pos) |
|---|---|---|---|---|
| PushT-v1 | 168,934 | **100%** | **100%** | **100%** |
| PokeCube-v1 | 68,545 | 69.7% | 32.5% | **100%** |
| PlaceSphere-v1 | 118,223 | 12.1% | 2.6% | 0.8% |
| PegInsertionSide-v1 | 150,055 | 6.2% | 0.7% | 0.7% |
| other 7 tasks | 946,166 | 0.4–2.1% | | |

The ~1% floor is the last frame of each episode (no next action) and is normal.
PushT has no usable action under *any* of the three control modes.

Training on it unfiltered silently produces a broken model: NaNs get zeroed by
`nan_to_num`, so the policy learns "output nothing" on 17% of frames, and the
normalisation statistics come out as NaN.

This run therefore drops episodes whose action column is NaN in more than 5% of
rows (`--max_nan_frac 0.05`):

```
9,216 / 12,257 episodes kept  →  1,218,188 frames (83.3% of the original)
```

## Architecture

```
Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
                                            └─ Head B → z_b ─┴─ concat → ResNetActionHead → 30×7

L = L1(action)        ← that is the whole objective
```

| | |
|---|---|
| LoRA | r=32 |
| Heads | dual `AttentiveLatentHead`, proj 4096, 8 queries, depth 2 |
| Inputs | 1 RGB frame @224 + task string — **no proprioceptive state** |
| Output | 30-step action chunk (1.0 s @ 30 fps) |
| Optimiser | AdamW, lr 5e-5, batch 32, bf16 |
| Augmentation | ColorJitter/SharpnessJitter ×2 + DomainRandomization p=0.7 |

`--no_state` is used, so the state token degenerates to a constant learned bias
(`state_proj(0)`) and no proprioception reaches the model. ManiSkill's merged
set has no `observation.state` column to begin with.

## Training curve

| step | loss |
|---|---|
| 1,000 | 0.0962 |
| 10,000 | 0.0601 |
| 20,000 | 0.0524 |
| 30,000 | 0.0487 |
| **40,000** | **0.0476** |

> **Not evaluated in simulation or on a robot.** The number above is a
> training-set L1 on normalised actions. No ManiSkill success rate is reported.

## Contents

`checkpoint.pt` holds `policy` (full VLM incl. LoRA), `latent_head`,
`free_latent_head`, `action_head`, `optimizer`, `agg_stats` (normalisation
statistics) and `args` (the exact CLI configuration, including the NaN filter
threshold).

```python
import torch
ck = torch.load("checkpoint.pt", map_location="cpu", weights_only=False)
ck["step"]        # 40000
ck["args"]        # full training config
ck["agg_stats"]   # action normalisation used at train time
```