BaseVLA ManiSkill Franka, vision-only + aug — step 40k

README.md

@@ -0,0 +1,101 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-VL-3B-Instruct
+tags:
+  - robotics
+  - vision-language-action
+  - vla
+  - qwen2_5_vl
+  - lerobot
+  - maniskill
+---
+
+# BaseVLA — ManiSkill Franka (Qwen2.5-VL-3B), vision-only, with augmentation
+
+Plain behaviour cloning baseline: **no V-JEPA, no latent alignment, no
+decorrelation**. Just `L1(action)` on ManiSkill's merged Franka set, trained
+**without proprioception** and with domain augmentation on.
+
+Checkpoint at **step 40,000**. The run was stopped early at ~44k steps (of a
+planned 100k) because the loss had flattened — 0.0487 at 30k to 0.0474 at 42k.
+
+## Data
+
+`maniskill-franka-merged` from [`Ngseo/stage1-data`](https://huggingface.co/datasets/Ngseo/stage1-data),
+single camera (`observation.images.anchor`, 256×256 h264), 30 fps, action
+`action.ee_delta_pose` (7-D).
+
+**The raw set needs filtering.** 17.1% of its `action.ee_delta_pose` rows are
+NaN, and they are not spread evenly — whole tasks were never populated:
+
+| task | frames | NaN (ee_delta) | NaN (joint_delta) | NaN (joint_pos) |
+|---|---|---|---|---|
+| PushT-v1 | 168,934 | **100%** | **100%** | **100%** |
+| PokeCube-v1 | 68,545 | 69.7% | 32.5% | **100%** |
+| PlaceSphere-v1 | 118,223 | 12.1% | 2.6% | 0.8% |
+| PegInsertionSide-v1 | 150,055 | 6.2% | 0.7% | 0.7% |
+| other 7 tasks | 946,166 | 0.4–2.1% | | |
+
+The ~1% floor is the last frame of each episode (no next action) and is normal.
+PushT has no usable action under *any* of the three control modes.
+
+Training on it unfiltered silently produces a broken model: NaNs get zeroed by
+`nan_to_num`, so the policy learns "output nothing" on 17% of frames, and the
+normalisation statistics come out as NaN.
+
+This run therefore drops episodes whose action column is NaN in more than 5% of
+rows (`--max_nan_frac 0.05`):
+
+```
+9,216 / 12,257 episodes kept  →  1,218,188 frames (83.3% of the original)
+```
+
+## Architecture
+
+```
+Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
+                                            └─ Head B → z_b ─┴─ concat → ResNetActionHead → 30×7
+
+L = L1(action)        ← that is the whole objective
+```
+
+| | |
+|---|---|
+| LoRA | r=32 |
+| Heads | dual `AttentiveLatentHead`, proj 4096, 8 queries, depth 2 |
+| Inputs | 1 RGB frame @224 + task string — **no proprioceptive state** |
+| Output | 30-step action chunk (1.0 s @ 30 fps) |
+| Optimiser | AdamW, lr 5e-5, batch 32, bf16 |
+| Augmentation | ColorJitter/SharpnessJitter ×2 + DomainRandomization p=0.7 |
+
+`--no_state` is used, so the state token degenerates to a constant learned bias
+(`state_proj(0)`) and no proprioception reaches the model. ManiSkill's merged
+set has no `observation.state` column to begin with.
+
+## Training curve
+
+| step | loss |
+|---|---|
+| 1,000 | 0.0962 |
+| 10,000 | 0.0601 |
+| 20,000 | 0.0524 |
+| 30,000 | 0.0487 |
+| **40,000** | **0.0476** |
+
+> **Not evaluated in simulation or on a robot.** The number above is a
+> training-set L1 on normalised actions. No ManiSkill success rate is reported.
+
+## Contents
+
+`checkpoint.pt` holds `policy` (full VLM incl. LoRA), `latent_head`,
+`free_latent_head`, `action_head`, `optimizer`, `agg_stats` (normalisation
+statistics) and `args` (the exact CLI configuration, including the NaN filter
+threshold).
+
+```python
+import torch
+ck = torch.load("checkpoint.pt", map_location="cpu", weights_only=False)
+ck["step"]        # 40000
+ck["args"]        # full training config
+ck["agg_stats"]   # action normalisation used at train time
+```

checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7dc199af450be5fd52a156313430c2a67078e4ee1efc952590a014e6c1f8d75
+size 9915326025