# Hardware Test Plan — Comprehensive Robot Integration Tests

## Approach

Use the Wireless unit connected over the network. Tests run from the laptop
using the same code paths as production (Marionette `run()` loop + HTTP API).
The reachy_mini SDK handles wireless streaming transparently.

## Existing Tests (8 passing, 3 audio skipped)

- Startup reaches idle
- Silent recording captures motion at ~100Hz
- Playback of silent move completes
- Record and delete lifecycle
- Recording produces correct JSON (frame count, timestamps, pose structure)
- Full record → replay → delete lifecycle
- Stop cancels queued recording
- Recording transitions through countdown → recording → idle

## New Tests to Add

### 1. Record-while-playing (motion accuracy)

**Idea:** Play a known move and simultaneously record it. Compare the
recorded motion to the original. This exercises the full pipeline and
measures end-to-end accuracy.

**How it works:**
1. Load a reference move from disk (an existing recording in the test dataset)
2. Start a recording via POST /api/record (silent, same duration as reference)
3. Immediately start playback of the reference move via `_stream_playback()`
   in a background thread (directly on the Marionette instance, bypassing
   the API since the API only allows one operation at a time)
4. Wait for recording to complete
5. Load the newly recorded move from disk
6. Compare frame-by-frame using `distance_between_poses()`

**What to measure:**
- **Per-frame magic distance** between reference and recorded head poses
- **RMSE** of magic distances over the full trajectory
- **Max error** (worst single frame)
- **Mean error**
- **Antenna RMSE** (L2 of joint angle differences)

**Thresholds (tunable):**
- Mean magic distance < 50 (50mm or 50 degrees equivalent — generous)
- Max magic distance < 100
- RMSE < 60

These are deliberately loose — we want to catch "robot didn't move" or
"completely wrong pose" bugs, not sub-millimeter tracking.

**Challenge:** The API doesn't support simultaneous record + play.
**Solution:** We use the `hw_marionette` instance directly:
- Call `_stream_playback()` in a thread to move the robot
- Separately, the `_capture_motion()` records what actually happens
- OR: We do it in two phases: first play a reference move, then start
  recording and play it again. The recording captures the actual motion.

Actually, simplest approach:
1. First, ensure a reference move exists (record a 3s silent move)
2. POST /api/record to start recording (3s, silent)
3. During the countdown + recording, play the reference move via the SDK directly
4. After recording completes, compare the two JSONs

### 2. Timing / performance benchmarks

**Tests:**
- **Startup time**: How long from `run()` start to `mode=idle`?
  Already measured implicitly (STARTUP_TIMEOUT=30s), but add explicit timing.
- **Recording start latency**: Time from POST /api/record to mode=countdown.
  Should be < 200ms.
- **Playback start latency**: Time from POST /api/play to actual motor movement.
  Measured by comparing first frame timestamp to request time.
- **Recording frame rate**: Verify actual ~100Hz (already tested via frame count).
- **Playback smoothness**: During playback, poll pose at high rate, verify
  it changes continuously (no freezes > 200ms).

### 3. Audio tests (expanded, with duration comparison)

**Tests:**
- **Duration match**: Record with mic, verify WAV duration matches requested
  duration within 0.5s tolerance.
- **Waveform not silent**: Play a known sound file on the robot speaker while
  recording with mic. Verify the recorded WAV has energy (RMS > threshold),
  not just zeros.
- **Playback-with-audio completes**: Play a move that has audio, verify timing.

### 4. Pose comparison utilities

Create a test helper module `tests/pose_utils.py` with:

```python
from reachy_mini.utils.interpolation import distance_between_poses
import numpy as np

def compare_trajectories(ref_times, ref_frames, rec_times, rec_frames):
    """Compare two recorded trajectories frame-by-frame.

    Interpolates the recorded trajectory to match reference timestamps.
    Returns dict with RMSE, max_error, mean_error, per-frame distances.
    """
    ...
```

### 5. Multi-move benchmark

Record and play back 3 different moves:
- A short move (1s)
- A medium move (3s)
- A longer move (5s)

For each, measure recording frame rate, playback completion time, and
verify the move data is well-formed.

## File Changes

| Action | File |
|--------|------|
| Create | `tests/pose_utils.py` — trajectory comparison helpers |
| Modify | `tests/test_hardware.py` — add new test classes |
| Modify | `tests/run_tests.py` — update class descriptions |
| Modify | `TESTING.md` — document new test classes |

## Test Classes (proposed)

| Class | Tests | Description |
|-------|-------|-------------|
| TestMotionAccuracy | 2-3 | Play reference move while recording, compare trajectories |
| TestPerformance | 3-4 | Startup timing, recording latency, frame rate, playback smoothness |
| TestHardwareAudio | 3 | (existing) Audio recording and playback |

## Open Questions

1. Can we play a move via the SDK while the Marionette API is recording?
   The run() loop handles one job at a time, but `set_target_head_pose()`
   is a direct SDK call that should work independently.

2. What's a reasonable accuracy threshold? Need to calibrate on one run,
   then set thresholds with margin.

3. Should benchmarks be hard-fail or just print results? Suggest: print
   results always, fail only on extreme regressions.