---
title: Brain x Robot x World Model Lab
emoji: 🧠
colorFrom: gray
colorTo: blue
sdk: gradio
python_version: '3.12'
app_file: app.py
pinned: false
---

# Brain x Robot x World Model Lab (ZeroGPU)

This Space is a publish-ready **Gradio ZeroGPU** demo with four user-facing labs sharing one runtime:

- **World Scout (SSL)**: CPU keyframe/metadata pass plus fast world-state inference with Depth Anything V2 Small and DINOv2.
- **Brainrot Lab (TRIBE v2)**: `facebook/tribev2` inference for text, image, and video, with cortical activity visualizations and a demo-layer Brainrot / Not Brainrot score.
- **Robot Action Lab (OpenVLA)**: `openvla/openvla-7b` zero-shot action proposals from a video frame and instruction.
- **Fusion Report**: heuristic Brain / Robot / World actionability score from TRIBE activity, depth structure, motion, and OpenVLA robustness.

The design goal is **public demo reliability on ZeroGPU**. Heavy models are lazy-loaded, inactive stacks are explicitly unloaded before modality switches, and deep world-model paths are opt-in instead of default.

## ZeroGPU Runtime

This app uses `@spaces.GPU` only around model forward-pass functions. CPU preprocessing runs outside GPU allocation:

- video probing, hashing, keyframe extraction, and image-to-video adaptation
- frame selection for OpenVLA and world-state inference
- result formatting, fusion scoring, and UI rendering

The app enables Gradio queueing with `default_concurrency_limit=1` and launches with `ssr_mode=False`.

## Configuration

- `HF_TOKEN`: optional globally. Required if a gated dependency is needed, especially TRIBE text mode if gated LLaMA access is triggered.
- `TRIBEV2_MODEL_ID`: default `facebook/tribev2`
- `OPENVLA_MODEL_ID`: default `openvla/openvla-7b`
- `ROBOT_OUTPUT_MODE`: default `normalized`
- `DEPTH_MODEL_ID`: default `depth-anything/Depth-Anything-V2-Small-hf`
- `DINO_MODEL_ID`: default `facebook/dinov2-small`
- `SAM_MODEL_ID`: default `facebook/sam3`
- `SAM_FALLBACK_MODEL_ID`: default `facebook/sam2-hiera-tiny`
- `MAP_ANYTHING_MODEL_ID`: default `facebook/map-anything-apache`

## Brainrot Lab

TRIBE predicts cortical activity \(y_{t,v}\), where \(t\) indexes timesteps and \(v\) indexes cortical vertices. For each region \(r\), with vertex set \(V_r\):

\[
\bar y_v = \frac{1}{T}\sum_{t=1}^{T} y_{t,v},\quad
p_v = \max_t |y_{t,v}|
\]

\[
s_r = 0.4 \cdot \mu_{v \in V_r} |\bar y_v| + 0.6 \cdot \mu_{v \in V_r} p_v
\]

Region scores are normalized against whole-brain activity and aggregated into an overall score:

\[
S = 0.30S_{attention} + 0.30S_{emotion} + 0.20S_{memory} + 0.20S_{reward}
\]

The binary label defaults to **Brainrot** when \(S \ge 60\). This label is a demo interpretation layer, not an official Meta/TRIBE metric and not medical advice.

## World Scout And Fast World State

World Scout computes immediate CPU-side signals before heavyweight inference:

- SHA-256 input hash and duration/FPS/frame-count metadata
- uniformly sampled keyframes and coarse scene cuts
- frame-difference motion energy

Fast World State adds two small self-supervised perception layers:

- **Depth Anything V2 Small** for depth structure and spatial discontinuities
- **DINOv2 small** for SSL frame embeddings and cached nearest-frame similarity

Deep World State is deliberately opt-in:

- SAM3 / SAM2 for segmentation probes
- CoTracker3 for object-memory timelines
- Map-Anything Apache for future 3D scene-card support

HunyuanWorld / HY-World links are included as world-generation research references, but they are not part of the default ZeroGPU request path.

## Robot Action Lab

The prompt format is:

```text
In: What action should the robot take to {instruction}?
Out:
```

OpenVLA returns a 7-DoF action:

\[
(\Delta x,\Delta y,\Delta z,\Delta roll,\Delta pitch,\Delta yaw,g)
\]

The default output mode is **normalized**. `bridge_orig` is exposed only as an explicit opt-in mode because robot-specific unnormalization is not safe for arbitrary uploaded robots.

The VLA Robustness Debugger samples three timestamps and two prompt variants, then reports mean action, action variance, uncertainty norm, and a 0-100 stability score.

## Licenses And Disclaimers

- TRIBE v2 is associated with Meta research release terms and non-commercial usage constraints; verify the upstream model/repo terms before use.
- OpenVLA is a research visual policy model. Its output is illustrative and not guaranteed executable on arbitrary robot embodiments.
- Depth Anything, DINOv2, SAM, CoTracker, Map-Anything, HY-World, and HunyuanWorld have independent licenses/model cards. Check each upstream page before redistribution or commercial use.
- This Space is a research/demo app. It is not medical advice and not a robot controller.

## References

- [TRIBE v2](https://huggingface.co/facebook/tribev2) and [facebookresearch/tribev2](https://github.com/facebookresearch/tribev2)
- [OpenVLA 7B](https://huggingface.co/openvla/openvla-7b)
- [Depth Anything V2 Small](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf)
- [DINOv2 small](https://huggingface.co/facebook/dinov2-small)
- [SAM3](https://huggingface.co/facebook/sam3), [SAM2 tiny](https://huggingface.co/facebook/sam2-hiera-tiny), [CoTracker3](https://huggingface.co/facebook/cotracker3)
- [Map-Anything Apache](https://huggingface.co/facebook/map-anything-apache), [VGGT-1B](https://huggingface.co/facebook/VGGT-1B)
- [HunyuanWorld-1](https://huggingface.co/tencent/HunyuanWorld-1), [HY-World 2.0](https://huggingface.co/tencent/HY-World-2.0), [HY-WorldPlay](https://huggingface.co/tencent/HY-WorldPlay)
- Hugging Face [ZeroGPU docs](https://huggingface.co/docs/hub/en/spaces-zerogpu)