Hugging Face's logo

Spaces:
gpue
/
foundationpose
Paused

App Files Community
foundationpose / STATUS.md
Georg
initial commit
24857f8
preview code
|
Raw
History Blame
10.5 kB

FoundationPose Deployment - Current Status

Last Updated: 2026-01-28 Repository: /Users/georgpuschel/repos/robot-ml/foundationpose/ Hugging Face Space: https://huggingface.co/spaces/gpue/foundationpose

📦 What's Been Completed

Core Files Created

  1. app.py (570 lines)

    • Complete Gradio application with ZeroGPU integration
    • Dual mode: Placeholder (default) and Real FoundationPose
    • REST API endpoints: /api/initialize and /api/estimate
    • Web UI with tabs for initialization, estimation, and API docs
    • Environment variable USE_REAL_MODEL controls mode

  2. estimator.py (350+ lines)

    • FoundationPoseEstimator class wrapping the real FoundationPose API
    • Methods: register_object(), estimate_pose(), reset_tracking()
    • Handles camera intrinsics, depth images, and segmentation masks
    • Quaternion and rotation matrix conversions
    • Mesh loading and reconstruction (placeholder for BundleSDF)

  3. client.py (200+ lines)

    • Python client for calling the API from robot-ml
    • FoundationPoseClient class with initialize() and estimate_pose()
    • Image encoding/decoding utilities
    • Example usage code

  4. requirements.txt

    • Core dependencies: gradio, spaces, torch, opencv-python
    • 3D vision: trimesh, pyrender, scikit-image
    • Placeholder for FoundationPose installation

  5. Dockerfile

    • CUDA 12.1 base image
    • System dependencies (eigen3, OpenGL, etc.)
    • FoundationPose repository clone and build
    • NVDiffRast, Kaolin, PyTorch3D installation

  6. download_weights.py

    • Script to check for and download model weights
    • Instructions for manual weight setup
    • Git-LFS integration guide

  7. deploy.sh

    • Interactive deployment script
    • Checks for weights and git status
    • Offers placeholder vs real mode deployment
    • Guides through git-lfs setup

  8. Documentation

    • README.md (updated with full details)
    • DEPLOYMENT.md (step-by-step deployment guide)
    • QUICKSTART.md (quick start for both modes)
    • STATUS.md (this file)

  9. .gitignore

    • Python cache files
    • Virtual environments
    • Model weights (for git-lfs)
    • Test images

🎯 How It Works

Placeholder Mode (Default)

  • Purpose: API testing without GPU requirements
  • Behavior: Returns empty pose results with success=true
  • Use Cases:
    • Developing client integrations
    • Testing API structure
    • Demos without GPU costs

Real Mode (Requires Setup)

  • Purpose: Actual 6D pose estimation
  • Requirements:
    • Model weights in weights/ directory
    • FoundationPose repository cloned
    • Environment variable USE_REAL_MODEL=true
  • Behavior: Uses actual FoundationPose inference
  • Use Cases:
    • Production pose estimation
    • Validation and testing with real data
    • Integration with robot-ml training

🚀 Deployment Options

Option 1: Test Locally (Placeholder) 

cd foundationpose
pip install -r requirements.txt
python app.py
# Visit http://localhost:7860

Option 2: Deploy to HF (Placeholder) 

./deploy.sh
# Or manually:
git add .
git commit -m "Deploy FoundationPose Space"
git push origin main

Option 3: Deploy to HF (Real Mode)

Requirements:

  1. Download weights from Google Drive
  2. Set up git-lfs
  3. Enable USE_REAL_MODEL=true

Steps:

# 1. Download weights (manual step)
# Visit: https://drive.google.com/drive/folders/1GCyGE-LbFGgRC-FuGsF3a1zeBuzsQ1Da
# Extract to: weights/2023-10-28-18-33-37/ and weights/2024-01-11-20-02-45/

# 2. Set up git-lfs
git lfs install
git lfs track "weights/**"
git add .gitattributes

# 3. Add weights
git add weights/
git commit -m "Add model weights"

# 4. Deploy
git push origin main

# 5. Set Space secret: USE_REAL_MODEL=true

🔗 Integration with robot-ml

Update FoundationPose Wrapper

Edit /training/nova_sim_trainer/perception/foundation_pose_wrapper.py:

from foundationpose.client import FoundationPoseClient
from pathlib import Path
import cv2

class FoundationPoseWrapper(PoseEstimator):
    def __init__(self, api_url: str, tracked_objects: List[Dict], **kwargs):
        super().__init__()
        self.client = FoundationPoseClient(api_url)
        self.object_ids = []

        # Initialize each tracked object
        for obj_config in tracked_objects:
            if not obj_config.get("enabled", True):
                continue

            object_id = obj_config["object_id"]
            ref_dir = Path(obj_config["reference_images_dir"])

            # Load reference images
            ref_images = []
            for img_path in sorted(ref_dir.glob("*.jpg")):
                img = cv2.imread(str(img_path))
                img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
                ref_images.append(img)

            # Register object with API
            logger.info(f"Registering {object_id} with {len(ref_images)} images...")
            self.client.initialize(object_id, ref_images)
            self.object_ids.append(object_id)

    def estimate_poses(self, frame, camera_intrinsics, scene_objects):
        detected_poses = []

        for object_id in self.object_ids:
            poses = self.client.estimate_pose(object_id, frame, camera_intrinsics)

            for pose in poses:
                detected_poses.append(DetectedPose(
                    object_id=pose["object_id"],
                    position=pose["position"],
                    orientation=pose["orientation"],
                    confidence=pose["confidence"],
                    timestamp=0.0,
                    dimensions=tuple(pose.get("dimensions", [0.1, 0.1, 0.1]))
                ))

        return detected_poses

Update Configuration

Edit /training/observations.yaml:

perception:
  enabled: true
  model: foundation_pose
  api_url: https://gpue-foundationpose.hf.space  # Your deployed Space URL
  camera: aux_top
  inference_fps: 5

  tracked_objects:
    - object_id: target_cube
      scene_object_name: t_object
      enabled: true
      reference_images_dir: ./perception/reference/target_cube
      dimensions: [0.1, 0.1, 0.1]

⚠️ What's NOT Done Yet

Missing Pieces

  1. Model Weights

    • Not included in repo (too large)
    • Must be downloaded manually
    • Requires git-lfs setup

  2. FoundationPose Repository

    • Not included (git submodule or clone needed)
    • C++ extensions need to be built
    • Tested only with placeholder code

  3. BundleSDF Integration

    • Mesh reconstruction not implemented
    • Currently uses placeholder cube mesh
    • Needed for model-free mode

  4. Segmentation

    • Object segmentation uses placeholder mask
    • Should integrate SAM (Segment Anything Model)
    • Optional but improves accuracy

  5. Pose Visualization

    • Estimation results don't show overlays yet
    • TODO: Render detected pose on query image
    • Would improve debugging

Testing Status

  • ✅ Placeholder mode tested locally
  • ⚠️ Real mode NOT tested (no weights)
  • ⚠️ API integration NOT tested end-to-end
  • ⚠️ ZeroGPU behavior unknown (not deployed yet)

📊 Performance Expectations

ZeroGPU Characteristics

  • Cold Start: 15-30 seconds (GPU allocation + model loading)
  • Warm Inference: 0.5-2 seconds per query
  • Free Tier: Limited monthly usage
  • Timeout: GPU allocation lasts for duration specified in decorator

robot-ml Training Integration

Not suitable for real-time training loop (30 Hz):

  • 5 Hz perception requires 200ms per frame
  • ZeroGPU latency: 500ms-2s warm, 15-30s cold
  • ❌ Too slow for synchronous training

Suitable for:

  • ✅ Batch processing recorded episodes
  • ✅ Validation and testing
  • ✅ Demos and visualization
  • ✅ Reference data collection

Recommendation:

  • Use dummy estimator during training (reads ground truth from sim)
  • Use FoundationPose API for validation/testing only
  • Consider local GPU deployment for production

📝 Next Steps

Immediate (Before Deployment)

  1. Test Locally

    cd foundationpose
python app.py
# Test UI at http://localhost:7860

  2. Deploy Placeholder Mode

    ./deploy.sh
# Choose "N" for real mode

  3. Verify Space Works

Short Term (With Weights)

  1. Download Model Weights

    • Get from Google Drive (see DEPLOYMENT.md)
    • Extract to weights/ directory

  2. Test Real Mode Locally

    export USE_REAL_MODEL=true
python app.py
# Upload real reference images
# Test pose estimation

  3. Deploy Real Mode

    # Set up git-lfs
git lfs track "weights/**"
git add .gitattributes weights/
git commit -m "Add model weights"
git push

# Set Space secret: USE_REAL_MODEL=true

Long Term (Production)

  1. Optimize Performance

    • Implement batch inference
    • Add caching for frequently used objects
    • Tune GPU duration parameters

  2. Improve Accuracy

    • Integrate SAM for segmentation
    • Add depth image support
    • Implement BundleSDF reconstruction

  3. Production Deployment

    • Consider dedicated GPU (RunPod, Modal, etc.)
    • Set up monitoring and logging
    • Implement retry logic and error handling

📚 Reference Links

🤝 Citation

If you use this in your work:

@inproceedings{wen2023foundationpose,
  title={FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects},
  author={Wen, Bowen and Yang, Wei and Kautz, Jan and Birchfield, Stan},
  booktitle={CVPR},
  year={2024}
}

Summary: The FoundationPose Space is fully set up and ready to deploy. It defaults to placeholder mode (no GPU needed) for testing the API structure. To enable real pose estimation, you need to manually download the model weights and set USE_REAL_MODEL=true. The integration code for robot-ml is ready but untested without the actual weights.