TESTING.md

Testing — Marionette

Run the tests

The quickest way to run everything and log results:

cd marionette
python tests/run_tests.py

This will:

1. Auto-detect your OS (Linux / macOS / Windows)
2. Ask which robot model you have connected (None / Lite / Wireless)
3. Run unit tests with verbose output
4. Run E2E tests for each available Playwright browser
5. Print a per-class test catalog and coverage matrix
6. Save results to tests/test_results.json
7. Export marionette/static/test_coverage.json for the web UI

After running, commit marionette/static/test_coverage.json and push so other contributors can see the latest coverage on the web page.

Install dependencies

cd marionette
pip install -e ".[dev]"

For E2E browser tests you also need Playwright browsers:

pip install pytest-playwright
playwright install --with-deps chromium firefox

WebKit (Safari's browser engine) is available on some platforms (playwright install webkit) but is not required.

Running individual suites

Unit tests only (~4 s)

cd marionette
pytest tests/test_api.py

No hardware or running server needed — tests use FastAPI's TestClient.

E2E single browser (~20 s)

cd marionette
pytest tests/e2e --browser chromium

E2E multi-browser

cd marionette
pytest tests/e2e --browser chromium --browser firefox --browser webkit

Hardware integration tests (local)

Requires a physical Reachy Mini robot connected and powered on. This runs tests on the local machine (useful when developing directly on the robot or when the robot is accessible locally):

cd marionette
pytest tests/test_hardware.py -m hardware

Or via the test runner:

cd marionette
python tests/run_tests.py --hardware

Hardware tests on the robot (recommended)

Hardware tests should run on the robot itself because:

• Audio playback (play_sound()) requires local speakers — WAV files are silently dropped over WebRTC
• The robot has direct hardware access (local mic, speakers, motors)
• Running locally eliminates network latency from test measurements

One-time SSH setup:

# Copy your SSH key to the robot (password: pollen)
ssh-copy-id pollen@reachy-mini.local

# Verify passwordless access
ssh pollen@reachy-mini.local echo OK

Run tests remotely:

cd marionette
python tests/run_on_robot.py

This will:

1. rsync the marionette package + tests to /tmp/marionette_test/ on the robot
2. Install dev deps if missing (pytest, httpx, pytest-json-report, scipy)
3. Run pytest -m hardware on the robot, streaming output in real-time
4. Fetch the JSON report back and print a summary

Options:

python tests/run_on_robot.py --dry-run             # show what would be synced
python tests/run_on_robot.py --host 192.168.1.42   # custom host/IP
python tests/run_on_robot.py --user pollen          # custom SSH user
python tests/run_on_robot.py -k test_playback       # extra pytest args

Or via the main test runner:

python tests/run_tests.py --on-robot
python tests/run_tests.py --on-robot --host 192.168.1.42

All tests at once (no hardware)

cd marionette
pytest tests/ --browser chromium

What's tested

Unit tests (tests/test_api.py)

Class	Tests	Description
TestSlugify	7	Move label slug generation
TestStateEndpoint	9	GET /api/state shape and initial values
TestStartingUpMode	3	Starting-up mode rejects commands
TestRecordEndpoint	9	POST /api/record validation and state transitions
TestPlayEndpoint	3	POST /api/play validation
TestStopEndpoints	3	POST /api/play/stop and /api/record/stop
TestMoveDelete	4	DELETE /api/moves/:id and file cleanup
TestDatasets	11	Dataset create, select, root change, origin
TestRegistryPersistence	3	Registry file survival across restarts
TestMovesRefresh	4	Move listing and metadata
TestExperiments	4	Feature toggles and experimental settings
TestHfAutoLogin	5	HF auto-login detection and caching
TestSensorData	1	Sensor data dummy endpoint
TestUploadAudio	7	POST /api/upload-audio validation and integration
TestMotionModelEndpoint	4	POST /api/motion-model enable, set, persist
TestCommunityDatasets	3	Community dataset listing and download validation
TestCorruptData	4	Malformed JSON and corrupt registry recovery
TestConcurrentStateChanges	4	Concurrent operations rejected when busy
TestDurationEdgeCases	3	Duration boundary validation (gt=0.5, le=300)
TestSyncDatasetExtended	4	Sync endpoint edge cases and validation

E2E browser tests (tests/e2e/test_ui.py)

Class	Tests	Description
TestPageLoad	3	Page loads and main sections visible
TestIdleState	4	Idle state display and controls
TestSimplifiedForm	3	Simplified form layout
TestFormValidation	5	Form fields and HTML5 validation
TestRecordingSubmission	1	Recording submission changes mode
TestDatasetUI	2	Dataset UI elements visible
TestRecordingLifecycle	4	Recording submit, stop, and injected move visibility
TestPlaybackLifecycle	3	Play button, queued playback, and move metadata
TestDeleteMove	3	Delete button, confirm/cancel dialog handling
TestCreateDataset	4	New dataset form show/hide/create/validate
TestSwitchDataset	2	Dataset switching and dropdown population
TestAudioSourceRadios	3	Audio source radio buttons and upload area toggle
TestSettingsPanel	3	Settings expand, experimental toggle, root display
TestFormEdgeCases	4	Empty/long/special labels and localStorage persistence

Hardware tests (tests/test_hardware.py)

Class	Tests	Description
TestHardwareStartup	1	Robot reaches idle after startup
TestHardwareRecording	1	Record silent and verify motion capture
TestHardwarePlayback	2	Playback and delete (silent)
TestFullPipeline	4	Full record → verify files → replay → delete lifecycle
TestMotionAccuracy	3	Synthetic playback accuracy — reference vs observed poses
TestMultiDuration	7	Recording and playback across 1s/3s/5s/10s durations
TestPerformance	3	Startup, recording, and playback latency benchmarks
TestExistingRecordingPlayback	2	Play back existing audio and silent recordings
TestRecordingRoundTrip	2	Record → playback fidelity and timing verification
TestAntennaAndBodyYaw	2	Verify antenna and body_yaw data in recordings
TestPlaybackAntennas	1	Synthetic antenna oscillation playback
TestHardwareAudio	3	Audio recording and playback (may skip on mic issues)

View the matrix without running tests

cd marionette
python tests/show_matrix.py

Matrix format

Coverage Matrix (latest result per combination)

             │ Unit Tests │ Chromium  │ Firefox   │ WebKit
─────────────┼────────────┼───────────┼───────────┼──────────
Linux/None   │ ✅ Feb 20  │ ✅ Feb 20 │ —         │ —
Linux/Lite   │ —          │ —         │ —         │ —
macOS/None   │ —          │ —         │ —         │ —
Windows/None │ —          │ —         │ —         │ —
...

Each cell shows the latest result for that OS + robot + test-suite combination. — means never tested. A "Hardware" column appears when hardware test results are available.

Web coverage

The Space's index.html displays a Test Coverage section that loads marionette/static/test_coverage.json (exported by run_tests.py). This is visible on the HuggingFace Space page, not the app's web GUI at localhost:8042.

Preview locally

To see the coverage table as it appears on the Space, open index.html in a browser. Because it fetches a JSON file, you need a local server:

cd marionette
python -m http.server 8080

Then open http://localhost:8080 — you should see the Space landing page with the Test Coverage matrix near the bottom.

Commit and push results

After running the tests, commit the updated coverage file so the Space page reflects the latest results:

cd marionette
git add marionette/static/test_coverage.json
git commit -m "Update test coverage matrix"
git push

Markers

• @pytest.mark.hardware — tests that require a physical robot (skipped by default unless you run with -m hardware)

Project layout

tests/
├── conftest.py          # Shared fixtures (TestClient, temp paths, make_wav_bytes)
├── test_api.py          # Unit tests — backend API
├── test_hardware.py     # Hardware integration tests (real robot)
├── pose_utils.py        # Trajectory comparison utilities
├── e2e/
│   ├── conftest.py      # Playwright server fixture (port 18042)
│   └── test_ui.py       # E2E browser tests
├── run_tests.py         # Test runner + matrix logger + web export
├── run_on_robot.py      # Remote test runner (SSH + rsync to robot)
├── run_smoke_on_robot.py # Smoke test runner (SSH + rsync)
├── smoke_test_on_robot.py # Smoke test script (runs on robot)
├── show_matrix.py       # Matrix viewer
├── REMOTE_TESTING.md    # SSH/rsync workflow documentation
├── HARDWARE_TEST_PLAN.md # Hardware test plan
└── test_results.json    # Auto-generated results log (git-ignored)

marionette/static/
└── test_coverage.json   # Web matrix export (committed)