---
license: apache-2.0
base_model: lerobot/smolvla_base
datasets:
- ahmedsohail2003/so101-sim-pickplace
tags:
- lerobot
- smolvla
- vla
- robotics
- manipulation
- so-arm100
- mujoco
pipeline_tag: robotics
library_name: lerobot
---

# smolvla-so101-pickplace — language-conditioned pick-and-place

> **Successor:** [smolvla-so101-pickplace-v2](https://huggingface.co/ahmedsohail2003/smolvla-so101-pickplace-v2) reaches **90%** on the same protocol (trained on the 160-episode `-v2` recovery set, which fixed this model's right-side coverage failures).

**SmolVLA (450M)** fine-tuned on
[`ahmedsohail2003/so101-sim-pickplace`](https://huggingface.co/datasets/ahmedsohail2003/so101-sim-pickplace) —
100 language-labeled MuJoCo demonstrations of an SO-ARM100 arm performing:

> *"Pick up the red block and place it in the blue tray."*

![demo episode](smolvla_episode.gif)

## Results — measured, same protocol for every row

20 evaluation episodes in the MuJoCo work-cell (fixed eval seed disjoint from
training data, nominal scene, 240-step cap at 15 Hz):

| Policy | Success | Notes |
|---|---|---|
| SmolVLA base, zero-shot | 0/20 (0%) | never engages the block |
| **SmolVLA fine-tuned (this model)** | **11/20 (55%)** | consistent when it engages: every success ≈105 steps, tray placement within ±3 mm |
| ACT (~52M, specialist BC, same 100 demos) | 13/20 (65%) | from the companion [sim2cell](https://huggingface.co/datasets/ahmedsohail2003/so101-sim-pickplace) project |
| ACT + temporal ensembling | 15/20 (75%) | |

**Honest read:** 12k steps of expert-only fine-tuning takes a generalist VLA
from 0% to within 10 points of a specialist behavior-cloning baseline on the
same data — while being commandable in natural language. Failures are
non-engagements clustered on right-side block spawns (under-represented in the
100 demos), not failed grasps: the policy either commits and succeeds
consistently, or never engages.

## Fine-tuning recipe (free-tier: single Kaggle T4)

- LeRobot `0.6.0`, `lerobot-train`, AMP fp16, batch 8, **12k steps** (~8 h)
- Frozen VLM backbone, action expert only (`freeze_vision_encoder=True`,
  `train_expert_only=True`) → ~100M trainable of 450M total
- lr 1e-4 peak, cosine decay, warmup 1k; loss 0.97 → ~0.06
- Camera mapping from the pretrained base's naming:
  `--rename_map='{"observation.images.front": "observation.images.camera1", "observation.images.wrist": "observation.images.camera2"}'`
  (2 dataset cameras are a subset of the base's 3; the absent third is skipped)

## Use

```python
import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

repo = "ahmedsohail2003/smolvla-so101-pickplace"
policy = SmolVLAPolicy.from_pretrained(repo).to("cuda").eval()
pre, post = make_pre_post_processors(
    policy_cfg=policy.config, pretrained_path=repo,
    preprocessor_overrides={"device_processor": {"device": "cuda"}},
)

batch = {
    "observation.state": state_6d,                      # (1, 6) float32
    "observation.images.camera1": front_rgb,            # (1, 3, 224, 224) in [0, 1]
    "observation.images.camera2": wrist_rgb,            # (1, 3, 224, 224) in [0, 1]
    "task": "Pick up the red block and place it in the blue tray.",
}
action = post(policy.select_action(pre(batch)))         # (1, 6) joint setpoints
```

Evaluation script: [`eval_smolvla.py`](https://huggingface.co/ahmedsohail2003/smolvla-so101-pickplace)
in the TalkToTheCell project (env + protocol from the sim2cell work-cell).

## Provenance

- Base: [lerobot/smolvla_base](https://huggingface.co/lerobot/smolvla_base) (Apache-2.0)
- Data: author-recorded scripted-expert demonstrations in MuJoCo
  ([dataset card](https://huggingface.co/datasets/ahmedsohail2003/so101-sim-pickplace))
- Robot model: [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie)
  `trs_so_arm100` (Apache-2.0)