DEPLOYMENT.md

# FoundationPose Hugging Face Space Deployment Guide

This directory contains the code for deploying FoundationPose on Hugging Face Spaces with ZeroGPU support.

## Current Status

- ✅ Gradio app structure created
- ✅ API endpoints defined (/initialize, /estimate)
- ✅ ZeroGPU decorators added (@spaces.GPU)
- ✅ Client library for API calls created
- ⚠️ FoundationPose model integration incomplete (placeholder code)

## Next Steps

### 1. Complete FoundationPose Integration

The current `app.py` has placeholder code marked with `# TODO` comments. You need to:

1. **Install FoundationPose in the Space**:
   - Add FoundationPose installation to requirements.txt or use a custom Dockerfile
   - Download pre-trained weights (need to be included in the Space or downloaded at startup)

2. **Implement model initialization** (line ~40 in app.py):
   ```python
   # Replace the TODO with actual FoundationPose initialization
   from FoundationPose import FoundationPoseEstimator
   self.model = FoundationPoseEstimator(device=self.device)
   ```

3. **Implement object registration** (line ~70):
   ```python
   # Replace the TODO with actual registration
   self.model.register_object(object_id, reference_images, camera_intrinsics)
   ```

4. **Implement pose estimation** (line ~120):
   ```python
   # Replace the TODO with actual inference
   result = self.model.estimate_pose(object_id, query_image, camera_intrinsics)
   ```

### 2. Handle Model Weights

FoundationPose requires pre-trained weights. Options:

**Option A: Git LFS (Recommended)**
```bash
cd foundationpose
git lfs install
mkdir weights
# Download weights from FoundationPose repo
wget https://... -O weights/model.pth
git lfs track "weights/*.pth"
git add weights/model.pth .gitattributes
git commit -m "Add model weights"
```

**Option B: Download at Runtime**
Add to `app.py`:
```python
def download_weights():
    from huggingface_hub import hf_hub_download
    weights_path = hf_hub_download(
        repo_id="NVlabs/FoundationPose",
        filename="model.pth"
    )
    return weights_path
```

### 3. Test Locally

Before deploying, test the Space locally:

```bash
cd foundationpose
pip install -r requirements.txt
python app.py
```

This will start a local Gradio server at http://localhost:7860

### 4. Deploy to Hugging Face

```bash
cd foundationpose
git add .
git commit -m "Add FoundationPose inference implementation"
git push
```

The Space will automatically rebuild and deploy.

### 5. Monitor GPU Usage

After deployment:
1. Check the Space logs for GPU allocation messages
2. Monitor inference times (cold start vs warm)
3. Adjust `@spaces.GPU(duration=X)` parameters if needed

### 6. Integrate with Training Pipeline

Once the Space is working, update the training code:

**In training/nova_sim_trainer/perception/foundation_pose_wrapper.py**:
```python
from foundationpose.client import FoundationPoseClient

class FoundationPoseWrapper(PoseEstimator):
    def __init__(self, api_url: str, ...):
        self.client = FoundationPoseClient(api_url)
        # Initialize with reference images
        ref_images = load_reference_images(reference_dir)
        self.client.initialize(object_id, ref_images)

    def estimate_poses(self, frame, camera_intrinsics, scene_objects):
        poses = self.client.estimate_pose(self.object_id, frame, camera_intrinsics)
        return [DetectedPose(**pose) for pose in poses]
```

**In training/observations.yaml**:
```yaml
perception:
  enabled: true
  model: foundation_pose
  api_url: https://gpue-foundationpose.hf.space
  tracked_objects:
    - object_id: target_cube
      reference_images_dir: ./perception/reference/target_cube
```

## Performance Considerations

### ZeroGPU Latency
- **Cold start**: 15-30 seconds (GPU allocation + model loading)
- **Warm inference**: 0.5-2 seconds per query
- **GPU duration**: Tune the `duration` parameter in `@spaces.GPU` decorators

### Recommended Usage
- ✅ **Batch processing**: Process multiple frames in one GPU allocation
- ✅ **Validation**: Check perception quality on recorded episodes
- ✅ **Demos**: Show 6D pose estimation capabilities
- ⚠️ **Real-time training**: Too slow for 30 Hz control loop - use dummy estimator instead

### Optimization Tips
1. **Batch multiple queries** to amortize cold start time
2. **Keep GPU warm** by sending periodic keep-alive requests
3. **Use lower resolution** if inference is too slow
4. **Cache results** for static scenes

## Troubleshooting

### Space won't start
- Check Space logs for errors
- Verify all dependencies in requirements.txt
- Check Python version compatibility (3.12)

### GPU timeout
- Increase `duration` in `@spaces.GPU(duration=X)`
- Optimize model inference code
- Reduce image resolution

### Out of memory
- Reduce batch size
- Use smaller model variant
- Request more GPU memory in Space settings

## Alternative: Docker Deployment

If ZeroGPU is too restrictive, consider running locally with Docker:

```bash
cd foundationpose
docker build -t foundationpose .
docker run -p 7860:7860 --gpus all foundationpose
```

Then set `api_url: http://localhost:7860` in observations.yaml.

## References

- [FoundationPose GitHub](https://github.com/NVlabs/FoundationPose)
- [Hugging Face Spaces](https://huggingface.co/docs/hub/spaces)
- [ZeroGPU Documentation](https://huggingface.co/docs/hub/spaces-gpus-zerogpu)
- [Gradio Documentation](https://www.gradio.app/docs)