README.md

metadata

title: Brain x Robot x World Model Lab
emoji: 🧠
colorFrom: gray
colorTo: blue
sdk: gradio
python_version: '3.12'
app_file: app.py
pinned: false

Brain x Robot x World Model Lab (ZeroGPU)

This Space is a publish-ready Gradio ZeroGPU demo with four user-facing labs sharing one runtime:

• World Scout (SSL): CPU keyframe/metadata pass plus fast world-state inference with Depth Anything V2 Small and DINOv2.
• Brainrot Lab (TRIBE v2): facebook/tribev2 inference for text, image, and video, with cortical activity visualizations and a demo-layer Brainrot / Not Brainrot score.
• Robot Action Lab (OpenVLA): openvla/openvla-7b zero-shot action proposals from a video frame and instruction.
• Fusion Report: heuristic Brain / Robot / World actionability score from TRIBE activity, depth structure, motion, and OpenVLA robustness.

The design goal is public demo reliability on ZeroGPU. Heavy models are lazy-loaded, inactive stacks are explicitly unloaded before modality switches, and deep world-model paths are opt-in instead of default.

ZeroGPU Runtime

This app uses @spaces.GPU only around model forward-pass functions. CPU preprocessing runs outside GPU allocation:

• video probing, hashing, keyframe extraction, and image-to-video adaptation
• frame selection for OpenVLA and world-state inference
• result formatting, fusion scoring, and UI rendering

The app enables Gradio queueing with default_concurrency_limit=1 and launches with ssr_mode=False.

Configuration

• HF_TOKEN: optional globally. Required if a gated dependency is needed, especially TRIBE text mode if gated LLaMA access is triggered.
• TRIBEV2_MODEL_ID: default facebook/tribev2
• OPENVLA_MODEL_ID: default openvla/openvla-7b
• ROBOT_OUTPUT_MODE: default normalized
• DEPTH_MODEL_ID: default depth-anything/Depth-Anything-V2-Small-hf
• DINO_MODEL_ID: default facebook/dinov2-small
• SAM_MODEL_ID: default facebook/sam3
• SAM_FALLBACK_MODEL_ID: default facebook/sam2-hiera-tiny
• MAP_ANYTHING_MODEL_ID: default facebook/map-anything-apache

Brainrot Lab

TRIBE predicts cortical activity (y_{t,v}), where (t) indexes timesteps and (v) indexes cortical vertices. For each region (r), with vertex set (V_r):

[ \bar y_v = \frac{1}{T}\sum_{t=1}^{T} y_{t,v},\quad p_v = \max_t |y_{t,v}| ]

[ s_r = 0.4 \cdot \mu_{v \in V_r} |\bar y_v| + 0.6 \cdot \mu_{v \in V_r} p_v ]

Region scores are normalized against whole-brain activity and aggregated into an overall score:

[ S = 0.30S_{attention} + 0.30S_{emotion} + 0.20S_{memory} + 0.20S_{reward} ]

The binary label defaults to Brainrot when (S \ge 60). This label is a demo interpretation layer, not an official Meta/TRIBE metric and not medical advice.

World Scout And Fast World State

World Scout computes immediate CPU-side signals before heavyweight inference:

• SHA-256 input hash and duration/FPS/frame-count metadata
• uniformly sampled keyframes and coarse scene cuts
• frame-difference motion energy

Fast World State adds two small self-supervised perception layers:

• Depth Anything V2 Small for depth structure and spatial discontinuities
• DINOv2 small for SSL frame embeddings and cached nearest-frame similarity

Deep World State is deliberately opt-in:

• SAM3 / SAM2 for segmentation probes
• CoTracker3 for object-memory timelines
• Map-Anything Apache for future 3D scene-card support

HunyuanWorld / HY-World links are included as world-generation research references, but they are not part of the default ZeroGPU request path.

Robot Action Lab

The prompt format is:

In: What action should the robot take to {instruction}?
Out:

OpenVLA returns a 7-DoF action:

[ (\Delta x,\Delta y,\Delta z,\Delta roll,\Delta pitch,\Delta yaw,g) ]

The default output mode is normalized. bridge_orig is exposed only as an explicit opt-in mode because robot-specific unnormalization is not safe for arbitrary uploaded robots.

The VLA Robustness Debugger samples three timestamps and two prompt variants, then reports mean action, action variance, uncertainty norm, and a 0-100 stability score.

Licenses And Disclaimers

• TRIBE v2 is associated with Meta research release terms and non-commercial usage constraints; verify the upstream model/repo terms before use.
• OpenVLA is a research visual policy model. Its output is illustrative and not guaranteed executable on arbitrary robot embodiments.
• Depth Anything, DINOv2, SAM, CoTracker, Map-Anything, HY-World, and HunyuanWorld have independent licenses/model cards. Check each upstream page before redistribution or commercial use.
• This Space is a research/demo app. It is not medical advice and not a robot controller.

References

• TRIBE v2 and facebookresearch/tribev2
• OpenVLA 7B
• Depth Anything V2 Small
• DINOv2 small
• SAM3, SAM2 tiny, CoTracker3
• Map-Anything Apache, VGGT-1B
• HunyuanWorld-1, HY-World 2.0, HY-WorldPlay
• Hugging Face ZeroGPU docs