DEPLOYMENT.md

FoundationPose Hugging Face Space Deployment Guide

This directory contains the code for deploying FoundationPose on Hugging Face Spaces with ZeroGPU support.

Current Status

• ✅ Gradio app structure created
• ✅ API endpoints defined (/initialize, /estimate)
• ✅ ZeroGPU decorators added (@spaces.GPU)
• ✅ Client library for API calls created
• ⚠️ FoundationPose model integration incomplete (placeholder code)

Next Steps

1. Complete FoundationPose Integration

The current app.py has placeholder code marked with # TODO comments. You need to:

1. Install FoundationPose in the Space:

   • Add FoundationPose installation to requirements.txt or use a custom Dockerfile
   • Download pre-trained weights (need to be included in the Space or downloaded at startup)

2. Implement model initialization (line ~40 in app.py):

   # Replace the TODO with actual FoundationPose initialization
   from FoundationPose import FoundationPoseEstimator
   self.model = FoundationPoseEstimator(device=self.device)
   

3. Implement object registration (line ~70):

   # Replace the TODO with actual registration
   self.model.register_object(object_id, reference_images, camera_intrinsics)
   

4. Implement pose estimation (line ~120):

   # Replace the TODO with actual inference
   result = self.model.estimate_pose(object_id, query_image, camera_intrinsics)
   

2. Handle Model Weights

FoundationPose requires pre-trained weights. Options:

Option A: Git LFS (Recommended)

cd foundationpose
git lfs install
mkdir weights
# Download weights from FoundationPose repo
wget https://... -O weights/model.pth
git lfs track "weights/*.pth"
git add weights/model.pth .gitattributes
git commit -m "Add model weights"

Option B: Download at Runtime Add to app.py:

def download_weights():
    from huggingface_hub import hf_hub_download
    weights_path = hf_hub_download(
        repo_id="NVlabs/FoundationPose",
        filename="model.pth"
    )
    return weights_path

3. Test Locally

Before deploying, test the Space locally:

cd foundationpose
pip install -r requirements.txt
python app.py

This will start a local Gradio server at http://localhost:7860

4. Deploy to Hugging Face

cd foundationpose
git add .
git commit -m "Add FoundationPose inference implementation"
git push

The Space will automatically rebuild and deploy.

5. Monitor GPU Usage

After deployment:

1. Check the Space logs for GPU allocation messages
2. Monitor inference times (cold start vs warm)
3. Adjust @spaces.GPU(duration=X) parameters if needed

6. Integrate with Training Pipeline

Once the Space is working, update the training code:

In training/nova_sim_trainer/perception/foundation_pose_wrapper.py:

from foundationpose.client import FoundationPoseClient

class FoundationPoseWrapper(PoseEstimator):
    def __init__(self, api_url: str, ...):
        self.client = FoundationPoseClient(api_url)
        # Initialize with reference images
        ref_images = load_reference_images(reference_dir)
        self.client.initialize(object_id, ref_images)

    def estimate_poses(self, frame, camera_intrinsics, scene_objects):
        poses = self.client.estimate_pose(self.object_id, frame, camera_intrinsics)
        return [DetectedPose(**pose) for pose in poses]

In training/observations.yaml:

perception:
  enabled: true
  model: foundation_pose
  api_url: https://gpue-foundationpose.hf.space
  tracked_objects:
    - object_id: target_cube
      reference_images_dir: ./perception/reference/target_cube

Performance Considerations

ZeroGPU Latency

• Cold start: 15-30 seconds (GPU allocation + model loading)
• Warm inference: 0.5-2 seconds per query
• GPU duration: Tune the duration parameter in @spaces.GPU decorators

Recommended Usage

• ✅ Batch processing: Process multiple frames in one GPU allocation
• ✅ Validation: Check perception quality on recorded episodes
• ✅ Demos: Show 6D pose estimation capabilities
• ⚠️ Real-time training: Too slow for 30 Hz control loop - use dummy estimator instead

Optimization Tips

1. Batch multiple queries to amortize cold start time
2. Keep GPU warm by sending periodic keep-alive requests
3. Use lower resolution if inference is too slow
4. Cache results for static scenes

Troubleshooting

Space won't start

• Check Space logs for errors
• Verify all dependencies in requirements.txt
• Check Python version compatibility (3.12)

GPU timeout

• Increase duration in @spaces.GPU(duration=X)
• Optimize model inference code
• Reduce image resolution

Out of memory

• Reduce batch size
• Use smaller model variant
• Request more GPU memory in Space settings

Alternative: Docker Deployment

If ZeroGPU is too restrictive, consider running locally with Docker:

cd foundationpose
docker build -t foundationpose .
docker run -p 7860:7860 --gpus all foundationpose

Then set api_url: http://localhost:7860 in observations.yaml.

References

• FoundationPose GitHub
• Hugging Face Spaces
• ZeroGPU Documentation
• Gradio Documentation