tests/HARDWARE_TEST_PLAN.md

Hardware Test Plan — Comprehensive Robot Integration Tests

Approach

Use the Wireless unit connected over the network. Tests run from the laptop using the same code paths as production (Marionette run() loop + HTTP API). The reachy_mini SDK handles wireless streaming transparently.

Existing Tests (8 passing, 3 audio skipped)

• Startup reaches idle
• Silent recording captures motion at ~100Hz
• Playback of silent move completes
• Record and delete lifecycle
• Recording produces correct JSON (frame count, timestamps, pose structure)
• Full record → replay → delete lifecycle
• Stop cancels queued recording
• Recording transitions through countdown → recording → idle

New Tests to Add

1. Record-while-playing (motion accuracy)

Idea: Play a known move and simultaneously record it. Compare the recorded motion to the original. This exercises the full pipeline and measures end-to-end accuracy.

How it works:

1. Load a reference move from disk (an existing recording in the test dataset)
2. Start a recording via POST /api/record (silent, same duration as reference)
3. Immediately start playback of the reference move via _stream_playback() in a background thread (directly on the Marionette instance, bypassing the API since the API only allows one operation at a time)
4. Wait for recording to complete
5. Load the newly recorded move from disk
6. Compare frame-by-frame using distance_between_poses()

What to measure:

• Per-frame magic distance between reference and recorded head poses
• RMSE of magic distances over the full trajectory
• Max error (worst single frame)
• Mean error
• Antenna RMSE (L2 of joint angle differences)

Thresholds (tunable):

• Mean magic distance < 50 (50mm or 50 degrees equivalent — generous)
• Max magic distance < 100
• RMSE < 60

These are deliberately loose — we want to catch "robot didn't move" or "completely wrong pose" bugs, not sub-millimeter tracking.

Challenge: The API doesn't support simultaneous record + play. Solution: We use the hw_marionette instance directly:

• Call _stream_playback() in a thread to move the robot
• Separately, the _capture_motion() records what actually happens
• OR: We do it in two phases: first play a reference move, then start recording and play it again. The recording captures the actual motion.

Actually, simplest approach:

1. First, ensure a reference move exists (record a 3s silent move)
2. POST /api/record to start recording (3s, silent)
3. During the countdown + recording, play the reference move via the SDK directly
4. After recording completes, compare the two JSONs

2. Timing / performance benchmarks

Tests:

• Startup time: How long from run() start to mode=idle? Already measured implicitly (STARTUP_TIMEOUT=30s), but add explicit timing.
• Recording start latency: Time from POST /api/record to mode=countdown. Should be < 200ms.
• Playback start latency: Time from POST /api/play to actual motor movement. Measured by comparing first frame timestamp to request time.
• Recording frame rate: Verify actual ~100Hz (already tested via frame count).
• Playback smoothness: During playback, poll pose at high rate, verify it changes continuously (no freezes > 200ms).

3. Audio tests (expanded, with duration comparison)

Tests:

• Duration match: Record with mic, verify WAV duration matches requested duration within 0.5s tolerance.
• Waveform not silent: Play a known sound file on the robot speaker while recording with mic. Verify the recorded WAV has energy (RMS > threshold), not just zeros.
• Playback-with-audio completes: Play a move that has audio, verify timing.

4. Pose comparison utilities

Create a test helper module tests/pose_utils.py with:

from reachy_mini.utils.interpolation import distance_between_poses
import numpy as np

def compare_trajectories(ref_times, ref_frames, rec_times, rec_frames):
    """Compare two recorded trajectories frame-by-frame.

    Interpolates the recorded trajectory to match reference timestamps.
    Returns dict with RMSE, max_error, mean_error, per-frame distances.
    """
    ...

5. Multi-move benchmark

Record and play back 3 different moves:

• A short move (1s)
• A medium move (3s)
• A longer move (5s)

For each, measure recording frame rate, playback completion time, and verify the move data is well-formed.

File Changes

Action	File
Create	tests/pose_utils.py — trajectory comparison helpers
Modify	tests/test_hardware.py — add new test classes
Modify	tests/run_tests.py — update class descriptions
Modify	TESTING.md — document new test classes

Test Classes (proposed)

Class	Tests	Description
TestMotionAccuracy	2-3	Play reference move while recording, compare trajectories
TestPerformance	3-4	Startup timing, recording latency, frame rate, playback smoothness
TestHardwareAudio	3	(existing) Audio recording and playback

Open Questions

1. Can we play a move via the SDK while the Marionette API is recording? The run() loop handles one job at a time, but set_target_head_pose() is a direct SDK call that should work independently.

2. What's a reasonable accuracy threshold? Need to calibrate on one run, then set thresholds with margin.

3. Should benchmarks be hard-fail or just print results? Suggest: print results always, fail only on extreme regressions.