STATUS.md

# FoundationPose Deployment - Current Status

**Last Updated:** 2026-01-28
**Repository:** `/Users/georgpuschel/repos/robot-ml/foundationpose/`
**Hugging Face Space:** https://huggingface.co/spaces/gpue/foundationpose

---

## 📦 What's Been Completed

### Core Files Created

1. **app.py** (570 lines)
   - Complete Gradio application with ZeroGPU integration
   - Dual mode: Placeholder (default) and Real FoundationPose
   - REST API endpoints: `/api/initialize` and `/api/estimate`
   - Web UI with tabs for initialization, estimation, and API docs
   - Environment variable `USE_REAL_MODEL` controls mode

2. **estimator.py** (350+ lines)
   - FoundationPoseEstimator class wrapping the real FoundationPose API
   - Methods: `register_object()`, `estimate_pose()`, `reset_tracking()`
   - Handles camera intrinsics, depth images, and segmentation masks
   - Quaternion and rotation matrix conversions
   - Mesh loading and reconstruction (placeholder for BundleSDF)

3. **client.py** (200+ lines)
   - Python client for calling the API from robot-ml
   - FoundationPoseClient class with initialize() and estimate_pose()
   - Image encoding/decoding utilities
   - Example usage code

4. **requirements.txt**
   - Core dependencies: gradio, spaces, torch, opencv-python
   - 3D vision: trimesh, pyrender, scikit-image
   - Placeholder for FoundationPose installation

5. **Dockerfile**
   - CUDA 12.1 base image
   - System dependencies (eigen3, OpenGL, etc.)
   - FoundationPose repository clone and build
   - NVDiffRast, Kaolin, PyTorch3D installation

6. **download_weights.py**
   - Script to check for and download model weights
   - Instructions for manual weight setup
   - Git-LFS integration guide

7. **deploy.sh**
   - Interactive deployment script
   - Checks for weights and git status
   - Offers placeholder vs real mode deployment
   - Guides through git-lfs setup

8. **Documentation**
   - README.md (updated with full details)
   - DEPLOYMENT.md (step-by-step deployment guide)
   - QUICKSTART.md (quick start for both modes)
   - STATUS.md (this file)

9. **.gitignore**
   - Python cache files
   - Virtual environments
   - Model weights (for git-lfs)
   - Test images

---

## 🎯 How It Works

### Placeholder Mode (Default)

- **Purpose**: API testing without GPU requirements
- **Behavior**: Returns empty pose results with success=true
- **Use Cases**:
  - Developing client integrations
  - Testing API structure
  - Demos without GPU costs

### Real Mode (Requires Setup)

- **Purpose**: Actual 6D pose estimation
- **Requirements**:
  - Model weights in `weights/` directory
  - FoundationPose repository cloned
  - Environment variable `USE_REAL_MODEL=true`
- **Behavior**: Uses actual FoundationPose inference
- **Use Cases**:
  - Production pose estimation
  - Validation and testing with real data
  - Integration with robot-ml training

---

## 🚀 Deployment Options

### Option 1: Test Locally (Placeholder)

```bash
cd foundationpose
pip install -r requirements.txt
python app.py
# Visit http://localhost:7860
```

### Option 2: Deploy to HF (Placeholder)

```bash
./deploy.sh
# Or manually:
git add .
git commit -m "Deploy FoundationPose Space"
git push origin main
```

### Option 3: Deploy to HF (Real Mode)

**Requirements:**
1. Download weights from Google Drive
2. Set up git-lfs
3. Enable USE_REAL_MODEL=true

**Steps:**
```bash
# 1. Download weights (manual step)
# Visit: https://drive.google.com/drive/folders/1GCyGE-LbFGgRC-FuGsF3a1zeBuzsQ1Da
# Extract to: weights/2023-10-28-18-33-37/ and weights/2024-01-11-20-02-45/

# 2. Set up git-lfs
git lfs install
git lfs track "weights/**"
git add .gitattributes

# 3. Add weights
git add weights/
git commit -m "Add model weights"

# 4. Deploy
git push origin main

# 5. Set Space secret: USE_REAL_MODEL=true
```

---

## 🔗 Integration with robot-ml

### Update FoundationPose Wrapper

Edit `/training/nova_sim_trainer/perception/foundation_pose_wrapper.py`:

```python
from foundationpose.client import FoundationPoseClient
from pathlib import Path
import cv2

class FoundationPoseWrapper(PoseEstimator):
    def __init__(self, api_url: str, tracked_objects: List[Dict], **kwargs):
        super().__init__()
        self.client = FoundationPoseClient(api_url)
        self.object_ids = []

        # Initialize each tracked object
        for obj_config in tracked_objects:
            if not obj_config.get("enabled", True):
                continue

            object_id = obj_config["object_id"]
            ref_dir = Path(obj_config["reference_images_dir"])

            # Load reference images
            ref_images = []
            for img_path in sorted(ref_dir.glob("*.jpg")):
                img = cv2.imread(str(img_path))
                img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
                ref_images.append(img)

            # Register object with API
            logger.info(f"Registering {object_id} with {len(ref_images)} images...")
            self.client.initialize(object_id, ref_images)
            self.object_ids.append(object_id)

    def estimate_poses(self, frame, camera_intrinsics, scene_objects):
        detected_poses = []

        for object_id in self.object_ids:
            poses = self.client.estimate_pose(object_id, frame, camera_intrinsics)

            for pose in poses:
                detected_poses.append(DetectedPose(
                    object_id=pose["object_id"],
                    position=pose["position"],
                    orientation=pose["orientation"],
                    confidence=pose["confidence"],
                    timestamp=0.0,
                    dimensions=tuple(pose.get("dimensions", [0.1, 0.1, 0.1]))
                ))

        return detected_poses
```

### Update Configuration

Edit `/training/observations.yaml`:

```yaml
perception:
  enabled: true
  model: foundation_pose
  api_url: https://gpue-foundationpose.hf.space  # Your deployed Space URL
  camera: aux_top
  inference_fps: 5

  tracked_objects:
    - object_id: target_cube
      scene_object_name: t_object
      enabled: true
      reference_images_dir: ./perception/reference/target_cube
      dimensions: [0.1, 0.1, 0.1]
```

---

## ⚠️ What's NOT Done Yet

### Missing Pieces

1. **Model Weights**
   - Not included in repo (too large)
   - Must be downloaded manually
   - Requires git-lfs setup

2. **FoundationPose Repository**
   - Not included (git submodule or clone needed)
   - C++ extensions need to be built
   - Tested only with placeholder code

3. **BundleSDF Integration**
   - Mesh reconstruction not implemented
   - Currently uses placeholder cube mesh
   - Needed for model-free mode

4. **Segmentation**
   - Object segmentation uses placeholder mask
   - Should integrate SAM (Segment Anything Model)
   - Optional but improves accuracy

5. **Pose Visualization**
   - Estimation results don't show overlays yet
   - TODO: Render detected pose on query image
   - Would improve debugging

### Testing Status

- ✅ Placeholder mode tested locally
- ⚠️ Real mode NOT tested (no weights)
- ⚠️ API integration NOT tested end-to-end
- ⚠️ ZeroGPU behavior unknown (not deployed yet)

---

## 📊 Performance Expectations

### ZeroGPU Characteristics

- **Cold Start**: 15-30 seconds (GPU allocation + model loading)
- **Warm Inference**: 0.5-2 seconds per query
- **Free Tier**: Limited monthly usage
- **Timeout**: GPU allocation lasts for duration specified in decorator

### robot-ml Training Integration

**Not suitable for real-time training loop (30 Hz):**
- 5 Hz perception requires 200ms per frame
- ZeroGPU latency: 500ms-2s warm, 15-30s cold
- ❌ Too slow for synchronous training

**Suitable for:**
- ✅ Batch processing recorded episodes
- ✅ Validation and testing
- ✅ Demos and visualization
- ✅ Reference data collection

**Recommendation:**
- Use dummy estimator during training (reads ground truth from sim)
- Use FoundationPose API for validation/testing only
- Consider local GPU deployment for production

---

## 📝 Next Steps

### Immediate (Before Deployment)

1. **Test Locally**
   ```bash
   cd foundationpose
   python app.py
   # Test UI at http://localhost:7860
   ```

2. **Deploy Placeholder Mode**
   ```bash
   ./deploy.sh
   # Choose "N" for real mode
   ```

3. **Verify Space Works**
   - Visit https://huggingface.co/spaces/gpue/foundationpose
   - Test initialization with test images
   - Check logs for errors

### Short Term (With Weights)

1. **Download Model Weights**
   - Get from Google Drive (see DEPLOYMENT.md)
   - Extract to `weights/` directory

2. **Test Real Mode Locally**
   ```bash
   export USE_REAL_MODEL=true
   python app.py
   # Upload real reference images
   # Test pose estimation
   ```

3. **Deploy Real Mode**
   ```bash
   # Set up git-lfs
   git lfs track "weights/**"
   git add .gitattributes weights/
   git commit -m "Add model weights"
   git push

   # Set Space secret: USE_REAL_MODEL=true
   ```

### Long Term (Production)

1. **Optimize Performance**
   - Implement batch inference
   - Add caching for frequently used objects
   - Tune GPU duration parameters

2. **Improve Accuracy**
   - Integrate SAM for segmentation
   - Add depth image support
   - Implement BundleSDF reconstruction

3. **Production Deployment**
   - Consider dedicated GPU (RunPod, Modal, etc.)
   - Set up monitoring and logging
   - Implement retry logic and error handling

---

## 📚 Reference Links

- **FoundationPose GitHub**: https://github.com/NVlabs/FoundationPose
- **Research Paper**: https://arxiv.org/abs/2312.08344
- **Model Weights**: https://drive.google.com/drive/folders/1GCyGE-LbFGgRC-FuGsF3a1zeBuzsQ1Da
- **HF Spaces Docs**: https://huggingface.co/docs/hub/spaces
- **ZeroGPU Docs**: https://huggingface.co/docs/hub/spaces-gpus-zerogpu
- **Gradio Docs**: https://www.gradio.app/docs

---

## 🤝 Citation

If you use this in your work:

```bibtex
@inproceedings{wen2023foundationpose,
  title={FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects},
  author={Wen, Bowen and Yang, Wei and Kautz, Jan and Birchfield, Stan},
  booktitle={CVPR},
  year={2024}
}
```

---

**Summary:** The FoundationPose Space is fully set up and ready to deploy. It defaults to placeholder mode (no GPU needed) for testing the API structure. To enable real pose estimation, you need to manually download the model weights and set USE_REAL_MODEL=true. The integration code for robot-ml is ready but untested without the actual weights.