QUICKSTART.md

FoundationPose Quick Start Guide

Overview

This Hugging Face Space provides two modes:

1. Placeholder Mode (default) - Returns empty results, useful for testing the API without GPU requirements
2. Real Mode - Uses actual FoundationPose model for 6D pose estimation (requires GPU and model weights)

Testing Locally (Placeholder Mode)

The easiest way to test the API structure:

cd foundationpose
pip install -r requirements.txt
python app.py

Visit http://localhost:7860 to see the UI.

Deploying to Hugging Face Spaces

Option 1: Placeholder Mode (No Setup Required)

Just push to your Space:

cd foundationpose
git add .
git commit -m "Deploy FoundationPose Space"
git push

The Space will run in placeholder mode by default. This is useful for:

• Testing the API structure
• Developing client integrations
• Demos without GPU costs

Option 2: Real FoundationPose (Requires Setup)

Step 1: Clone FoundationPose Repository

# Inside your local foundationpose directory
git clone https://github.com/NVlabs/FoundationPose.git

Step 2: Download Model Weights

Download weights from the official Google Drive: https://drive.google.com/drive/folders/1GCyGE-LbFGgRC-FuGsF3a1zeBuzsQ1Da

Extract to:

foundationpose/weights/
  ├── 2023-10-28-18-33-37/  (refiner weights)
  └── 2024-01-11-20-02-45/  (scorer weights)

Step 3: Add Weights to Git LFS

git lfs install
git lfs track "weights/**/*.pth"
git lfs track "weights/**/*.ckpt"
git add .gitattributes
git add weights/
git commit -m "Add model weights"

Step 4: Enable Real Mode

Add to your Space settings (or use .env file locally):

USE_REAL_MODEL=true

Step 5: Push to HF

git push

Using the API

Python Client

from foundationpose.client import FoundationPoseClient
import cv2
import numpy as np

# Initialize client
client = FoundationPoseClient("https://gpue-foundationpose.hf.space")

# Load reference images
ref_images = []
for i in range(1, 16):
    img = cv2.imread(f"reference/image_{i:03d}.jpg")
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    ref_images.append(img)

# Register object
client.initialize("target_cube", ref_images)

# Estimate pose
query_img = cv2.imread("query.jpg")
query_img = cv2.cvtColor(query_img, cv2.COLOR_BGR2RGB)

poses = client.estimate_pose("target_cube", query_img)
print(f"Detected {len(poses)} poses")
for pose in poses:
    print(f"Position: {pose['position']}")
    print(f"Orientation: {pose['orientation']}")
    print(f"Confidence: {pose['confidence']}")

Direct HTTP API

# Initialize
curl -X POST https://gpue-foundationpose.hf.space/api/initialize \
  -H "Content-Type: application/json" \
  -d '{
    "object_id": "target_cube",
    "reference_images_b64": ["'$(base64 -w 0 ref1.jpg)'", "'$(base64 -w 0 ref2.jpg)'"],
    "camera_intrinsics": "{\"fx\": 500, \"fy\": 500, \"cx\": 320, \"cy\": 240}"
  }'

# Estimate
curl -X POST https://gpue-foundationpose.hf.space/api/estimate \
  -H "Content-Type: application/json" \
  -d '{
    "object_id": "target_cube",
    "query_image_b64": "'$(base64 -w 0 query.jpg)'"
  }'

Integration with robot-ml Training

Update /training/nova_sim_trainer/perception/foundation_pose_wrapper.py:

from foundationpose.client import FoundationPoseClient

class FoundationPoseWrapper(PoseEstimator):
    def __init__(self, api_url: str, tracked_objects: List[Dict], **kwargs):
        super().__init__()
        self.client = FoundationPoseClient(api_url)

        # Initialize each tracked object
        for obj_config in tracked_objects:
            object_id = obj_config["object_id"]
            ref_dir = Path(obj_config["reference_images_dir"])

            # Load reference images
            ref_images = []
            for img_path in sorted(ref_dir.glob("*.jpg")):
                img = cv2.imread(str(img_path))
                img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
                ref_images.append(img)

            # Register object
            self.client.initialize(object_id, ref_images)
            logger.info(f"Registered {object_id} with {len(ref_images)} images")

    def estimate_poses(self, frame, camera_intrinsics, scene_objects):
        # Call API for pose estimation
        poses = self.client.estimate_pose(
            self.tracked_objects[0]["object_id"],  # For now, single object
            frame,
            camera_intrinsics
        )

        # Convert to DetectedPose format
        return [DetectedPose(**pose) for pose in poses]

Update observations.yaml:

perception:
  enabled: true
  model: foundation_pose
  api_url: https://gpue-foundationpose.hf.space
  tracked_objects:
    - object_id: target_cube
      reference_images_dir: ./perception/reference/target_cube

Performance Tips

Cold Start Latency

• First request takes 15-30s (GPU allocation + model loading)
• Subsequent requests: 0.5-2s

Keeping GPU Warm

Send periodic keep-alive requests:

import time
import threading

def keep_warm():
    while True:
        try:
            client.estimate_pose("target_cube", dummy_image)
        except:
            pass
        time.sleep(60)  # Every minute

threading.Thread(target=keep_warm, daemon=True).start()

Batch Processing

For recorded episodes, process all frames in one session:

# Initialize once
client.initialize("target_cube", ref_images)

# Process all frames
poses_list = []
for frame in frames:
    poses = client.estimate_pose("target_cube", frame)
    poses_list.append(poses)

Troubleshooting

Space shows "Placeholder mode"

• Set USE_REAL_MODEL=true in Space secrets
• Verify weights are uploaded correctly
• Check Space logs for errors

"Model weights not found"

• Ensure weights are in weights/ directory
• Check git-lfs tracked files: git lfs ls-files
• Re-upload if needed

GPU timeout

• Increase @spaces.GPU(duration=X) in app.py
• Reduce image resolution
• Process fewer reference images

Out of memory

• Use lower resolution images
• Process fewer objects simultaneously
• Request more GPU resources in Space settings

Cost Optimization

ZeroGPU is free but has usage limits:

• Development: Use placeholder mode
• Testing: Enable real mode for specific tests only
• Production: Consider dedicated GPU deployment (RunPod, Modal, etc.)

Next Steps

1. Test locally in placeholder mode
2. Upload weights for real mode
3. Integrate with robot-ml training pipeline
4. Monitor GPU usage and costs
5. Optimize batch processing for your use case

Support

• Issues: https://github.com/gpuschel/robot-ml/issues
• FoundationPose: https://github.com/NVlabs/FoundationPose
• HF Spaces: https://huggingface.co/docs/hub/spaces