RemiFabre Claude Opus 4.6 commited on
Commit
dbc544f
·
1 Parent(s): f8ae01c

Refactor marionette into modules, fix audio sync, improve tests

Browse files

Split 2349-line main.py into 7 focused modules (app, state, recording,
datasets, routes, audio, models). Fix audio-motion sync by compensating
320ms push_audio_sample pipeline latency (AUDIO_LEAD_MS). Fix countdown
beep sample rate (was 48kHz, should be 16kHz). Fix dataset re-download.
Add comprehensive sync test suite and 142+ unit tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.gitignore CHANGED
@@ -9,3 +9,9 @@ emotion_dataset_example/
9
  .report.json
10
  pytest-of-*/
11
  temp_uploads/
 
 
 
 
 
 
 
9
  .report.json
10
  pytest-of-*/
11
  temp_uploads/
12
+
13
+ # Test artifacts (plots, recordings, results)
14
+ tests/*.png
15
+ tests/*.wav
16
+ tests/*.json
17
+ !tests/conftest.py
ARCHITECTURE.md ADDED
@@ -0,0 +1,176 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Marionette Architecture
2
+
3
+ This document explains the design of the Marionette app for developers who want to understand, modify, or debug it.
4
+
5
+ ## What Marionette Does
6
+
7
+ Marionette records and plays back head movements (+ optional audio) for the Reachy Mini robot. A user moves the robot's head by hand while the app records the pose at 100 Hz, then replays it as an animated movement. Audio can be recorded from the mic, uploaded as a file, downloaded from YouTube, or picked from the robot's filesystem.
8
+
9
+ ## System Overview
10
+
11
+ ```
12
+ ┌─────────────────┐ HTTP (polling) ┌──────────────────┐
13
+ │ Browser (UI) │ ◄──────────────────────────── │ FastAPI Server │
14
+ │ main.js │ ──────────────────────────── ►│ (Uvicorn) │
15
+ │ index.html │ POST /api/record │ │
16
+ │ style.css │ POST /api/play │ marionette/ │
17
+ └─────────────────┘ GET /api/state │ ├── app.py │
18
+ │ ├── routes.py │
19
+ │ ├── recording.py│
20
+ │ ├── datasets.py │
21
+ │ ├── audio.py │
22
+ │ ├── state.py │
23
+ │ └── models.py │
24
+ └────────┬─────────┘
25
+
26
+ ┌────────▼─────────┐
27
+ │ Reachy Mini │
28
+ │ (Robot SDK) │
29
+ └──────────────────┘
30
+ ```
31
+
32
+ ### Two-Machine vs Single-Machine
33
+
34
+ - **Reachy Mini**: Backend runs on the robot (Linux ARM), browser runs on a laptop. Connected via WiFi.
35
+ - **Reachy Mini Light**: Backend and browser run on the same laptop. The robot connects via USB.
36
+
37
+ ## Module Map
38
+
39
+ | Module | Purpose | Key classes/functions |
40
+ |--------|---------|---------------------|
41
+ | `models.py` | Data types, constants, Pydantic models | `RecordingMetadata`, `RecordingRequest`, `DatasetEntry`, all `*Payload` classes |
42
+ | `audio.py` | Stateless audio functions | `play_wav_chunked()`, `preload_wav()`, `play_preloaded_wav()` |
43
+ | `state.py` | Thread-safe state read/write | `StateMixin._serialize_state()`, `_set_state()`, `_set_idle_state()` |
44
+ | `recording.py` | Motion capture + playback | `RecordingMixin._capture_motion()`, `_perform_recording()`, `_perform_playback()` |
45
+ | `datasets.py` | Dataset filesystem + HF sync | `DatasetMixin._load_dataset_registry()`, `_sync_dataset()`, `_check_hf_login()` |
46
+ | `routes.py` | HTTP endpoint definitions | `register_routes()` — all FastAPI route closures |
47
+ | `app.py` | Main class, run loop, robot helpers | `Marionette`, `create_app()` |
48
+ | `main.py` | Re-export hub | Imports and re-exports everything for backward compatibility |
49
+ | `motion_models.py` | Lead compensation model | `MotionModelRegistry`, shifts commands forward to counter mechanical lag |
50
+
51
+ ## Threading Model
52
+
53
+ The app has three types of threads:
54
+
55
+ 1. **Uvicorn thread** — Runs the FastAPI HTTP server. Handles all API requests. This is the thread that calls route handlers in `routes.py`.
56
+ 2. **Main robot thread** — Runs `Marionette.run()`. Polls for pending jobs (recording or playback) in a 50ms loop. Executes recording/playback synchronously.
57
+ 3. **Audio threads** — Spawned as daemon threads during playback. Push audio chunks to the robot's GStreamer pipeline.
58
+
59
+ ### Thread Safety
60
+
61
+ All shared state is protected by `_state_lock` (a `threading.Lock`):
62
+
63
+ - The HTTP thread writes: `_pending_recording`, `_pending_playback`, `_mode`
64
+ - The robot thread reads and clears: `_pending_recording`, `_pending_playback`
65
+ - Both threads read: `_mode`, `_recordings`, `_datasets`
66
+
67
+ The lock is held briefly — never during I/O or network calls.
68
+
69
+ ### Cancel Events
70
+
71
+ - `_recording_cancel_event` — Set by the HTTP thread (POST /api/record/stop), checked by the robot thread's capture loop.
72
+ - `_playback_cancel_event` — Set by the HTTP thread (POST /api/play/stop), checked by the robot thread's playback loop.
73
+
74
+ ## State Machine
75
+
76
+ ```
77
+ POST /api/record
78
+ idle ──────────────────── queued
79
+ ▲ │
80
+ │ (robot thread picks up)
81
+ │ ▼
82
+ │ countdown (3s)
83
+ │ │
84
+ │ ▼
85
+ └──────────────────── recording
86
+
87
+ (duration elapsed or stop)
88
+
89
+ idle
90
+
91
+ POST /api/play
92
+ idle ──────────────────── queued
93
+ ▲ │
94
+ │ (robot thread picks up)
95
+ │ ▼
96
+ └──────────────────── playing
97
+
98
+ (move ends or stop)
99
+
100
+ idle
101
+ ```
102
+
103
+ Additional states: `starting_up` (during boot animation), `error` (transient, returns to idle).
104
+
105
+ ## Frontend Architecture
106
+
107
+ The frontend is vanilla JavaScript (no framework). Key patterns:
108
+
109
+ ### Polling Loop
110
+
111
+ The browser polls `GET /api/state` at regular intervals:
112
+ - **1500ms** when idle (nothing happening)
113
+ - **200ms** when active (recording, playing, countdown)
114
+
115
+ Each poll returns the full app state. The frontend updates the UI accordingly.
116
+
117
+ ### Dirty Flag Pattern (Flickering Fix)
118
+
119
+ The moves list and dataset dropdown are expensive to rebuild (full DOM replacement). Without optimization, they flicker on every poll. The fix:
120
+
121
+ - `movesListDirty` and `datasetsDirty` flags start as `true`
122
+ - `renderMoves()` / `updateDatasetUI()` only run when the flag is `true`
123
+ - Flags are set `true` after user actions (record, delete, switch dataset, etc.)
124
+ - During idle polling, only lightweight updates happen (mode badge, play/stop buttons)
125
+
126
+ ### NTP-Style Clock Sync
127
+
128
+ The backend sends `server_time` with each state response. The frontend computes `clockOffset = serverTime - localTime` and uses it to accurately display countdown timers and recording progress bars, even when the browser and robot are on different machines with unsynchronized clocks.
129
+
130
+ ### Phase Overlay
131
+
132
+ During countdown and recording, a full-screen overlay shows progress. This uses `requestAnimationFrame` for smooth 60fps animation, independent of the polling interval.
133
+
134
+ ## Dataset Layout
135
+
136
+ ```
137
+ ~/reachy_mini_datasets/ # dataset root (configurable)
138
+ ├── local_dataset/ # default dataset
139
+ │ └── data/ # all recordings live here
140
+ │ ├── happy-dance.json # motion trajectory
141
+ │ └── happy-dance.wav # optional audio
142
+ ├── my-custom-set/
143
+ │ └── data/
144
+ │ └── ...
145
+ └── user-community-set/ # downloaded from HF
146
+ └── data/
147
+ └── ...
148
+ ```
149
+
150
+ The `dataset_registry.json` file (next to the app) tracks which datasets exist, which is active, and per-dataset metadata (uploaded move IDs, origin, etc.).
151
+
152
+ ## Audio Playback Pipeline
153
+
154
+ Audio is played through the Reachy Mini's GStreamer pipeline using chunk-based pushing:
155
+
156
+ 1. **Preload**: Read WAV file, resample to output rate if needed
157
+ 2. **Prime pipeline**: Call `start_playing()` to initialize GStreamer
158
+ 3. **Wait for sync**: Audio thread waits for `start_signal` (set when first motion command is sent)
159
+ 4. **Push chunks**: Feed 20ms audio chunks at ~1.25x real-time
160
+ 5. **Drain buffer**: Wait for remaining audio to play out
161
+ 6. **Cleanup**: Call `stop_playing()` with timeout (GStreamer can hang)
162
+
163
+ This push-based approach allows stopping audio at any time, unlike `play_sound()` which creates an uninterruptible pipeline.
164
+
165
+ ## Lead Compensation
166
+
167
+ Mechanical lag means the robot's actual motion trails the commanded trajectory. The lead compensation model (in `motion_models.py`) shifts commands forward in time so the actual motion matches the original recording. Parameters (head lead, antenna lead) are tunable in the settings UI.
168
+
169
+ ## Testing
170
+
171
+ Tests live in `tests/test_api.py` and use FastAPI's `TestClient` for synchronous HTTP testing without a real robot. The `conftest.py` creates a Marionette instance with a temporary dataset directory.
172
+
173
+ Key test patterns:
174
+ - HF functions are monkeypatched on `marionette.datasets` (the canonical location)
175
+ - Audio functions are monkeypatched on `marionette.recording` (where they're imported)
176
+ - Recording/playback tests use fake `ReachyMini` objects with stub methods
marionette/app.py ADDED
@@ -0,0 +1,330 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Core Marionette application class.
2
+
3
+ This module defines the Marionette class that inherits from ReachyMiniApp
4
+ and composes all functionality via mixins (StateMixin, RecordingMixin,
5
+ DatasetMixin). It handles initialisation, the main run loop, robot
6
+ motion helpers, and the startup animation.
7
+ """
8
+ from __future__ import annotations
9
+
10
+ import time as _time_mod
11
+ _BOOT_T0 = _time_mod.perf_counter()
12
+
13
+ import logging
14
+ import platform
15
+ import threading
16
+ import time
17
+ from pathlib import Path
18
+ from typing import Any
19
+
20
+ import numpy as np
21
+
22
+ from reachy_mini import ReachyMini, ReachyMiniApp
23
+ from reachy_mini.reachy_mini import INIT_HEAD_POSE, SLEEP_HEAD_POSE
24
+ from reachy_mini.utils.interpolation import distance_between_poses
25
+
26
+ from marionette.audio import play_wav_chunked
27
+ from marionette.datasets import DatasetMixin
28
+ from marionette.models import (
29
+ DATASET_REGISTRY_FILENAME,
30
+ DEFAULT_DURATION,
31
+ RecordingMetadata,
32
+ RecordingRequest,
33
+ SEMI_AWAKEN_POSE,
34
+ )
35
+ from marionette.motion_models import MotionModelRegistry
36
+ from marionette.recording import RecordingMixin
37
+ from marionette.routes import register_routes
38
+ from marionette.state import StateMixin
39
+
40
+ try:
41
+ import soundfile as sf
42
+ except Exception: # pragma: no cover
43
+ sf = None
44
+
45
+ try:
46
+ from importlib.metadata import version as _pkg_version
47
+ __version__ = _pkg_version("marionette")
48
+ except Exception:
49
+ __version__ = "unknown"
50
+
51
+ logger = logging.getLogger(__name__)
52
+
53
+
54
+ class Marionette(DatasetMixin, RecordingMixin, StateMixin, ReachyMiniApp):
55
+ """Manual Marionette recorder for Reachy Mini."""
56
+
57
+ custom_app_url: str | None = "http://0.0.0.0:8042"
58
+
59
+ def __init__(
60
+ self,
61
+ registry_path: Path | None = None,
62
+ dataset_root: Path | None = None,
63
+ ) -> None:
64
+ super().__init__()
65
+ self._registry_path = registry_path or (
66
+ Path(__file__).resolve().parent.parent / DATASET_REGISTRY_FILENAME
67
+ )
68
+ self._dataset_root_override = dataset_root
69
+ self._datasets: dict[str, Any] = {}
70
+ self._active_dataset_id: str | None = None
71
+ self._dataset_root: Path
72
+ self._dataset_dir: Path
73
+ self._motion_model_registry = MotionModelRegistry()
74
+ self._preferred_duration: float = DEFAULT_DURATION
75
+ self._welcome_messages: int = 2 # 0=none, 1=intro only, 2=intro+second
76
+ self._load_dataset_registry() # may overwrite _preferred_duration and _welcome_messages
77
+
78
+ self._recordings: dict[str, RecordingMetadata] = {}
79
+ self._pending_recording: RecordingRequest | None = None
80
+ self._pending_playback: str | None = None
81
+ self._uploaded_audio: dict[str, Path] = {} # upload_id -> temp file path
82
+ self._playback_cancel_event = threading.Event()
83
+ self._recording_cancel_event = threading.Event()
84
+ self._mode: str = "idle"
85
+ self._message: str = "Ready to capture moves"
86
+ self._countdown_ends_at: float | None = None
87
+ self._active_record_started_at: float | None = None
88
+ self._active_record_duration: float | None = None
89
+ self._active_move: str | None = None
90
+ self._state_lock = threading.Lock()
91
+ self._audio_available = sf is not None
92
+ self._recording_stats: dict[str, Any] | None = None
93
+ self._hf_username: str | None = None
94
+ self._hf_checked = False
95
+
96
+ self._refresh_recordings()
97
+ print(f"[BOOT] __init__ done at +{time.perf_counter() - _BOOT_T0:.2f}s after module import", flush=True)
98
+ logger.info("Marionette v%s starting on %s (%s)", __version__, platform.node(), platform.system())
99
+ if self.settings_app is not None:
100
+ register_routes(self, version=__version__)
101
+
102
+ # ──────── Startup animation ───────────────────────────────────────
103
+
104
+ def _run_startup_animation(
105
+ self, reachy_mini: ReachyMini, stop_event: threading.Event
106
+ ) -> None:
107
+ """Startup greeting with configurable welcome messages.
108
+
109
+ Behaviour depends on self._welcome_messages (0, 1, or 2):
110
+ 0 messages — head up, head down, release torque (silent)
111
+ 1 message — head up, play intro, head down, release torque
112
+ 2 messages — head up, play intro, head down, pause, head semi-up,
113
+ play second message, release torque (full greeting)
114
+
115
+ Uses stop_event.wait() instead of time.sleep() so the startup
116
+ can be interrupted cleanly if the app is shutting down.
117
+ """
118
+ self._set_state(mode="starting_up", message="Starting up…", active_move=None)
119
+ assets = Path(__file__).parent / "assets"
120
+ n_messages = self._welcome_messages
121
+
122
+ # Step 1: Head up (all modes)
123
+ self._safe_enable_motors(reachy_mini)
124
+ print(f"[BOOT] anim: enable_motors done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
125
+ self._goto_pose_scaled(
126
+ reachy_mini,
127
+ INIT_HEAD_POSE,
128
+ antennas=[0.0, 0.0],
129
+ min_duration=0.2,
130
+ )
131
+ print(f"[BOOT] anim: head-up done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
132
+
133
+ if stop_event.is_set():
134
+ return
135
+
136
+ # Step 2: Play intro sound (1 or 2 messages)
137
+ if n_messages >= 1:
138
+ intro_path = assets / "intro_marionette.wav"
139
+ if intro_path.exists() and not stop_event.is_set():
140
+ try:
141
+ play_wav_chunked(reachy_mini, intro_path, stop_event)
142
+ except Exception as exc:
143
+ logger.warning("Failed to play intro sound: %s", exc)
144
+ print(f"[BOOT] anim: intro sound done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
145
+
146
+ if stop_event.is_set():
147
+ return
148
+
149
+ # Step 3: Head down + release torque (all modes)
150
+ self._goto_sleep_and_release(reachy_mini)
151
+
152
+ if stop_event.is_set():
153
+ return
154
+
155
+ # Step 4: Second message with semi-awaken pose (2 messages only)
156
+ # Motion first (fast, ~0.3s), then audio sequentially — same
157
+ # pattern as Step 2 for simplicity.
158
+ if n_messages >= 2:
159
+ second_path = assets / "please.wav"
160
+ if second_path.exists() and not stop_event.is_set():
161
+ self._safe_enable_motors(reachy_mini)
162
+ antenna_angle = np.deg2rad(15)
163
+ reachy_mini.goto_target(
164
+ SEMI_AWAKEN_POSE,
165
+ antennas=[-antenna_angle, antenna_angle],
166
+ duration=0.3,
167
+ )
168
+ try:
169
+ play_wav_chunked(reachy_mini, second_path, stop_event)
170
+ except Exception as exc:
171
+ logger.warning("Failed to play second sound: %s", exc)
172
+ self._goto_sleep_and_release(reachy_mini)
173
+
174
+ # ──────── Main loop ───────────────────────────────────────────────
175
+
176
+ def run(self, reachy_mini: ReachyMini, stop_event: threading.Event) -> None:
177
+ try:
178
+ _run_t0 = time.perf_counter()
179
+ print(f"[BOOT] run() entered at +{_run_t0 - _BOOT_T0:.2f}s after module import", flush=True)
180
+ self._run_startup_animation(reachy_mini, stop_event)
181
+ print(f"[BOOT] startup animation done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
182
+ self._set_idle_state()
183
+
184
+ # Main job-polling loop: check for pending recording or playback
185
+ # every 20ms. The HTTP thread queues jobs by setting _pending_recording
186
+ # or _pending_playback under the lock; we pick them up and execute
187
+ # synchronously on this thread (the robot thread).
188
+ while not stop_event.is_set():
189
+ job_type: str | None = None
190
+ payload: RecordingRequest | str | None = None
191
+
192
+ with self._state_lock:
193
+ if self._pending_recording is not None:
194
+ job_type = "record"
195
+ payload = self._pending_recording
196
+ self._pending_recording = None
197
+ elif self._pending_playback is not None:
198
+ job_type = "play"
199
+ payload = self._pending_playback
200
+ self._pending_playback = None
201
+
202
+ if job_type == "record" and isinstance(payload, RecordingRequest):
203
+ self._perform_recording(reachy_mini, stop_event, payload)
204
+ self._set_idle_state()
205
+ elif job_type == "play" and isinstance(payload, str):
206
+ self._perform_playback(reachy_mini, payload)
207
+ self._set_idle_state()
208
+ self._align_head_and_release(reachy_mini)
209
+ else:
210
+ stop_event.wait(0.02)
211
+ finally:
212
+ self._park_robot(reachy_mini)
213
+
214
+ # ──────── Robot motion helpers ────────────────────────────────────
215
+
216
+ def _scaled_duration(
217
+ self, reachy_mini: ReachyMini, target_head_pose: np.ndarray, *, min_duration: float = 0.05
218
+ ) -> float:
219
+ """Compute a motion duration proportional to the distance to travel.
220
+
221
+ Uses the SDK's distance_between_poses (combining rotation and
222
+ translation into a single scalar) and scales it to a comfortable
223
+ speed. This ensures slow moves for small adjustments and faster
224
+ moves for large transitions, instead of a fixed duration.
225
+ """
226
+ _, _, magic_distance = distance_between_poses(
227
+ reachy_mini.get_current_head_pose(),
228
+ target_head_pose,
229
+ )
230
+ duration = magic_distance * 20 / 1000
231
+ return max(min_duration, duration)
232
+
233
+ def _goto_pose_scaled(
234
+ self,
235
+ reachy_mini: ReachyMini,
236
+ head_pose: np.ndarray,
237
+ *,
238
+ antennas: list[float] | np.ndarray | None = None,
239
+ min_duration: float = 0.05,
240
+ ) -> None:
241
+ """Go to pose with duration scaled by distance. goto_target is blocking."""
242
+ duration = self._scaled_duration(reachy_mini, head_pose, min_duration=min_duration)
243
+ antennas_payload = list(antennas) if antennas is not None else None
244
+ reachy_mini.goto_target(
245
+ head=head_pose,
246
+ antennas=antennas_payload,
247
+ duration=duration,
248
+ )
249
+
250
+ def _goto_current_pose(self, reachy_mini: ReachyMini, duration: float = 0.05) -> None:
251
+ head_pose = reachy_mini.get_current_head_pose()
252
+ _, antennas = reachy_mini.get_current_joint_positions()
253
+ reachy_mini.goto_target(
254
+ head=head_pose,
255
+ antennas=list(antennas) if antennas is not None else None,
256
+ duration=max(0.02, duration),
257
+ )
258
+
259
+ def _safe_enable_motors(self, reachy_mini: ReachyMini) -> None:
260
+ """Enable motors without a physical jerk.
261
+
262
+ If we just call enable_motors(), the robot snaps to whatever target
263
+ pose was last commanded. By first setting the target to the current
264
+ actual pose, the transition is smooth even if the head was moved
265
+ by hand while motors were off.
266
+ """
267
+ self._goto_current_pose(reachy_mini, duration=0.05)
268
+ reachy_mini.enable_motors()
269
+ time.sleep(0.1)
270
+
271
+ def _park_robot(self, reachy_mini: ReachyMini) -> None:
272
+ self._safe_enable_motors(reachy_mini)
273
+ self._goto_pose_scaled(
274
+ reachy_mini,
275
+ INIT_HEAD_POSE,
276
+ antennas=[0.0, 0.0],
277
+ min_duration=0.2,
278
+ )
279
+
280
+ def _align_head_and_release(self, reachy_mini: ReachyMini) -> None:
281
+ """After playback, return to neutral pose then release motors.
282
+
283
+ Called after _perform_playback so the robot ends in a predictable
284
+ position ready for the user to grab and record the next move.
285
+ """
286
+ self._safe_enable_motors(reachy_mini)
287
+ self._goto_pose_scaled(
288
+ reachy_mini,
289
+ INIT_HEAD_POSE,
290
+ antennas=[0.0, 0.0],
291
+ min_duration=0.2,
292
+ )
293
+ self._goto_sleep_and_release(reachy_mini)
294
+
295
+ def _goto_sleep_and_release(self, reachy_mini: ReachyMini) -> None:
296
+ """Move head to sleep pose then disable motors (graceful torque-off)."""
297
+ # Use 15 degrees for antennas for style points
298
+ antenna_angle = np.deg2rad(15)
299
+ reachy_mini.goto_target(
300
+ SLEEP_HEAD_POSE,
301
+ antennas=[-antenna_angle, antenna_angle],
302
+ duration=1.0,
303
+ )
304
+ reachy_mini.disable_motors()
305
+
306
+
307
+ def create_app(
308
+ registry_path: Path | None = None,
309
+ dataset_root: Path | None = None,
310
+ ) -> tuple["Any", "Marionette"]:
311
+ """Create a Marionette instance and its FastAPI app.
312
+
313
+ Useful for testing with FastAPI's TestClient:
314
+ app, marionette = create_app(registry_path=tmp / "reg.json", dataset_root=tmp / "ds")
315
+ client = TestClient(app)
316
+ """
317
+ marionette = Marionette(
318
+ registry_path=registry_path,
319
+ dataset_root=dataset_root,
320
+ )
321
+ assert marionette.settings_app is not None
322
+ return marionette.settings_app, marionette
323
+
324
+
325
+ if __name__ == "__main__":
326
+ app = Marionette()
327
+ try:
328
+ app.wrapped_run()
329
+ except KeyboardInterrupt:
330
+ app.stop()
marionette/audio.py ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Audio playback and recording helper functions.
2
+
3
+ All functions in this module are stateless (no class dependency) and can be
4
+ called from any context. They handle WAV file I/O, resampling, and chunked
5
+ push-based playback through the Reachy Mini media pipeline.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import logging
10
+ import threading
11
+ import time
12
+ from pathlib import Path
13
+
14
+ import numpy as np
15
+
16
+ try:
17
+ import soundfile as sf
18
+ except Exception: # pragma: no cover - optional dependency
19
+ sf = None
20
+
21
+ logger = logging.getLogger(__name__)
22
+
23
+
24
+ def generate_beep(
25
+ freq: float = 440,
26
+ duration: float = 0.1,
27
+ sr: int = 48000,
28
+ fade_ms: float = 10,
29
+ ) -> np.ndarray:
30
+ """Generate a short sine-wave beep with fade-in/out to avoid clicks."""
31
+ n = int(sr * duration)
32
+ t = np.arange(n, dtype=np.float32) / sr
33
+ samples = 0.3 * np.sin(2 * np.pi * freq * t).astype(np.float32)
34
+ # Apply fade-in/out
35
+ fade_samples = int(sr * fade_ms / 1000)
36
+ if fade_samples > 0 and 2 * fade_samples < n:
37
+ fade_in = np.linspace(0, 1, fade_samples, dtype=np.float32)
38
+ fade_out = np.linspace(1, 0, fade_samples, dtype=np.float32)
39
+ samples[:fade_samples] *= fade_in
40
+ samples[-fade_samples:] *= fade_out
41
+ return samples
42
+
43
+
44
+ def get_audio_duration(path: Path, fallback: float | None = 3.0) -> float | None:
45
+ """Return the duration of a WAV file in seconds, or *fallback* on error."""
46
+ if sf is not None:
47
+ try:
48
+ info = sf.info(str(path))
49
+ return float(info.duration)
50
+ except Exception:
51
+ pass
52
+ return fallback
53
+
54
+
55
+ def play_wav_chunked(
56
+ reachy_mini: object,
57
+ wav_path: Path,
58
+ stop_event: threading.Event,
59
+ chunk_duration: float = 0.02,
60
+ ) -> None:
61
+ """Play a WAV file via push_audio_sample() so it can be stopped at any time.
62
+
63
+ Unlike play_sound() which creates an orphan GStreamer pipeline that
64
+ cannot be interrupted, this loads the WAV, resamples to the output
65
+ rate, and pushes small chunks through the stoppable playback pipeline.
66
+ """
67
+ if sf is None:
68
+ return
69
+ try:
70
+ data, sr = sf.read(str(wav_path), dtype="float32")
71
+ except Exception as exc:
72
+ logger.warning("Failed to read %s: %s", wav_path, exc)
73
+ return
74
+
75
+ if data.ndim == 2:
76
+ data = data.mean(axis=1)
77
+
78
+ # Resample if the file's rate doesn't match the output device.
79
+ try:
80
+ sr_out = int(reachy_mini.media.get_output_audio_samplerate() or 16000)
81
+ except Exception:
82
+ sr_out = 16000
83
+ if sr != sr_out:
84
+ try:
85
+ from scipy.signal import resample
86
+
87
+ num_samples = int(len(data) * sr_out / sr)
88
+ data = resample(data, num_samples).astype(np.float32)
89
+ sr = sr_out
90
+ except Exception as exc:
91
+ logger.warning("Resample failed: %s — playing at original rate", exc)
92
+
93
+ reachy_mini.media.start_playing()
94
+
95
+ samples_per_chunk = max(1, int(sr * chunk_duration))
96
+ play_start = time.perf_counter()
97
+ try:
98
+ for offset in range(0, len(data), samples_per_chunk):
99
+ if stop_event.is_set():
100
+ break
101
+ chunk = data[offset : offset + samples_per_chunk]
102
+ reachy_mini.media.push_audio_sample(chunk)
103
+ # Sleep slightly less than chunk duration to keep buffer fed
104
+ stop_event.wait(chunk_duration * 0.8)
105
+ else:
106
+ # All chunks pushed — wait for the audio buffer to drain.
107
+ # We pushed ~20% faster than real-time, so the playback
108
+ # pipeline still has buffered audio to play out.
109
+ audio_duration = len(data) / sr
110
+ elapsed = time.perf_counter() - play_start
111
+ remaining = audio_duration - elapsed
112
+ if remaining > 0:
113
+ stop_event.wait(remaining)
114
+ finally:
115
+ call_with_timeout(reachy_mini.media.stop_playing)
116
+
117
+
118
+ def preload_wav(wav_path: Path, target_sr: int | None = None) -> tuple[np.ndarray, int] | None:
119
+ """Read a WAV file, returning (data, sample_rate) or None.
120
+
121
+ If target_sr is given and differs from the file's rate, resample.
122
+ """
123
+ if sf is None:
124
+ return None
125
+ try:
126
+ data, sr = sf.read(str(wav_path), dtype="float32")
127
+ except Exception as exc:
128
+ logger.warning("Failed to read %s: %s", wav_path, exc)
129
+ return None
130
+ if data.ndim == 2:
131
+ data = data.mean(axis=1)
132
+ if target_sr and sr != target_sr:
133
+ try:
134
+ from scipy.signal import resample
135
+
136
+ num_samples = int(len(data) * target_sr / sr)
137
+ data = resample(data, num_samples).astype(np.float32)
138
+ sr = target_sr
139
+ except Exception as exc:
140
+ logger.warning("Resample failed: %s — using original rate", exc)
141
+ return data, sr
142
+
143
+
144
+ def play_preloaded_wav(
145
+ reachy_mini: object,
146
+ wav_data: tuple[np.ndarray, int],
147
+ stop_event: threading.Event,
148
+ chunk_duration: float = 0.02,
149
+ pipeline_ready: bool = False,
150
+ start_signal: threading.Event | None = None,
151
+ ) -> None:
152
+ """Push preloaded WAV data through the playback pipeline."""
153
+ data, sr = wav_data
154
+ if not pipeline_ready:
155
+ reachy_mini.media.start_playing()
156
+ # If a start_signal is provided, wait until the caller says go
157
+ # (e.g. after the first motion command is sent).
158
+ if start_signal is not None:
159
+ start_signal.wait(timeout=5.0)
160
+ samples_per_chunk = max(1, int(sr * chunk_duration))
161
+ play_start = time.perf_counter()
162
+ try:
163
+ for offset in range(0, len(data), samples_per_chunk):
164
+ if stop_event.is_set():
165
+ break
166
+ chunk = data[offset : offset + samples_per_chunk]
167
+ reachy_mini.media.push_audio_sample(chunk)
168
+ stop_event.wait(chunk_duration * 0.8)
169
+ else:
170
+ audio_duration = len(data) / sr
171
+ elapsed = time.perf_counter() - play_start
172
+ remaining = audio_duration - elapsed
173
+ if remaining > 0:
174
+ stop_event.wait(remaining)
175
+ finally:
176
+ call_with_timeout(reachy_mini.media.stop_playing)
177
+
178
+
179
+ def call_with_timeout(fn: object, timeout: float = 2.0, retries: int = 2) -> bool:
180
+ """Call fn() in a daemon thread with timeout and retries.
181
+
182
+ GStreamer's set_state() can hang when the audio device is slow or the
183
+ GLib MainLoop is starved. We run the call in a disposable daemon thread
184
+ and retry up to `retries` times — often the second attempt succeeds
185
+ because the MainLoop has had time to process pending bus messages.
186
+
187
+ Returns True if the call completed within the timeout, False if all
188
+ attempts timed out (the last thread is abandoned as a daemon).
189
+ """
190
+ for attempt in range(1, retries + 1):
191
+ t = threading.Thread(target=fn, daemon=True)
192
+ t.start()
193
+ t.join(timeout)
194
+ if not t.is_alive():
195
+ return True
196
+ logger.warning(
197
+ "%s attempt %d/%d did not complete within %.1fs",
198
+ fn, attempt, retries, timeout,
199
+ )
200
+ logger.error("%s hung after %d attempts — abandoning", fn, retries)
201
+ return False
marionette/datasets.py ADDED
@@ -0,0 +1,706 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Dataset management mixin for the Marionette app.
2
+
3
+ Handles dataset registry CRUD, Hugging Face sync/download, community
4
+ dataset listing, and filesystem layout management.
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import json
9
+ import logging
10
+ import platform
11
+ import shutil
12
+ import time
13
+ from datetime import datetime
14
+ from pathlib import Path
15
+ from tempfile import TemporaryDirectory
16
+ from textwrap import dedent
17
+ from typing import Any
18
+
19
+ from fastapi import HTTPException
20
+
21
+ from marionette.models import (
22
+ COMMUNITY_DATASET_TAG,
23
+ DATASET_DATA_SUBDIR,
24
+ DATASET_DIRNAME,
25
+ DEFAULT_DATASET_LABEL,
26
+ DEFAULT_DURATION,
27
+ HF_DATASETS_API_URL,
28
+ MAX_COMMUNITY_DATASETS,
29
+ DatasetEntry,
30
+ DownloadDatasetPayload,
31
+ RecordingMetadata,
32
+ _slugify,
33
+ )
34
+
35
+ # ──────── Optional Hugging Face imports ───────────────────────────────
36
+
37
+ try:
38
+ from huggingface_hub import HfApi, snapshot_download, whoami as hf_whoami
39
+ from huggingface_hub import login as hf_login, logout as hf_logout
40
+ from huggingface_hub import get_token as hf_get_token
41
+ except Exception: # pragma: no cover - optional dependency
42
+ HfApi = None
43
+ snapshot_download = None
44
+ hf_whoami = None
45
+ hf_login = None
46
+ hf_logout = None
47
+ hf_get_token = None
48
+
49
+ try:
50
+ from huggingface_hub import DatasetFilter
51
+ except Exception: # pragma: no cover - optional helper
52
+ DatasetFilter = None
53
+
54
+ logger = logging.getLogger(__name__)
55
+
56
+
57
+ class DatasetMixin:
58
+ """Mixin providing dataset management methods.
59
+
60
+ Expects the host class to have: _registry_path, _dataset_root_override,
61
+ _datasets, _active_dataset_id, _dataset_root, _dataset_dir,
62
+ _motion_model_registry, _preferred_duration, _welcome_messages,
63
+ _recordings, _state_lock, _save_dataset_registry(), _refresh_recordings(),
64
+ _ensure_dataset_data_dir().
65
+ """
66
+
67
+ def _load_dataset_registry(self) -> None:
68
+ """Load the dataset registry JSON and discover dataset folders on disk.
69
+
70
+ The registry stores user preferences (motion model params, preferred
71
+ duration, welcome messages) alongside per-dataset metadata. On startup:
72
+ 1. Read the registry file (if it exists)
73
+ 2. Determine the dataset root: CLI override > registry > fallback > platform default
74
+ 3. Scan the root for dataset folders, merging stored metadata
75
+ 4. Ensure at least one dataset exists (creates "local_dataset" if empty)
76
+ """
77
+ if self._registry_path.exists():
78
+ try:
79
+ raw = json.loads(self._registry_path.read_text(encoding="utf-8"))
80
+ except Exception:
81
+ raw = {}
82
+ else:
83
+ raw = {}
84
+
85
+ raw_params = raw.get("motion_model_params", {})
86
+ if isinstance(raw_params, dict):
87
+ for model_name, params in raw_params.items():
88
+ if isinstance(params, dict):
89
+ try:
90
+ self._motion_model_registry.set_model_params(model_name, params)
91
+ except KeyError:
92
+ continue
93
+ self._preferred_duration = float(raw.get("preferred_duration", DEFAULT_DURATION))
94
+ self._welcome_messages = int(raw.get("welcome_messages", 2))
95
+
96
+ dataset_entries = raw.get("datasets", [])
97
+ metadata_by_folder: dict[str, dict[str, Any]] = {}
98
+ fallback_root: Path | None = None
99
+ for entry in dataset_entries:
100
+ folder = entry.get("folder")
101
+ path_str = entry.get("path")
102
+ if folder:
103
+ metadata_by_folder[folder] = entry
104
+ if not fallback_root and path_str:
105
+ resolved = self._resolve_dataset_path(path_str)
106
+ fallback_root = resolved.parent
107
+ elif path_str and not fallback_root:
108
+ resolved = self._resolve_dataset_path(path_str)
109
+ fallback_root = resolved.parent
110
+
111
+ raw_root = raw.get("root_path")
112
+ if self._dataset_root_override is not None:
113
+ root_path = self._dataset_root_override
114
+ elif raw_root:
115
+ root_path = self._resolve_dataset_path(raw_root)
116
+ elif fallback_root:
117
+ root_path = fallback_root
118
+ else:
119
+ root_path = self._default_dataset_root()
120
+ self._dataset_root = root_path
121
+ self._dataset_root.mkdir(parents=True, exist_ok=True)
122
+ logger.info("Using dataset root at %s", self._dataset_root)
123
+
124
+ datasets: dict[str, DatasetEntry] = {}
125
+ used_ids: set[str] = set()
126
+ for folder_path in sorted(self._dataset_root.iterdir()):
127
+ if not folder_path.is_dir():
128
+ continue
129
+ folder = folder_path.name
130
+ meta = metadata_by_folder.get(folder, {})
131
+ dataset_id = meta.get("id")
132
+ if not dataset_id or dataset_id in used_ids:
133
+ dataset_id = self._next_dataset_id(folder, used_ids)
134
+ else:
135
+ used_ids.add(dataset_id)
136
+ label = meta.get("label") or folder
137
+ uploaded_ids = set(meta.get("uploaded_move_ids") or [])
138
+ origin = meta.get("origin", "local")
139
+ entry = self._entry_from_folder(folder, label, dataset_id, uploaded_ids, origin=origin)
140
+ datasets[dataset_id] = entry
141
+
142
+ if not datasets:
143
+ default_id = "default" if "default" not in used_ids else self._next_dataset_id(DATASET_DIRNAME, used_ids)
144
+ entry = self._entry_from_folder(DATASET_DIRNAME, DEFAULT_DATASET_LABEL, default_id)
145
+ datasets[entry.dataset_id] = entry
146
+
147
+ self._datasets = datasets
148
+ active_id = raw.get("active")
149
+ if active_id not in self._datasets:
150
+ active_id = next(iter(self._datasets))
151
+ self._active_dataset_id = active_id
152
+ self._dataset_dir = self._ensure_dataset_data_dir(self._datasets[active_id])
153
+ self._save_dataset_registry()
154
+
155
+ def _save_dataset_registry(self) -> None:
156
+ data = {
157
+ "active": self._active_dataset_id,
158
+ "root_path": str(self._dataset_root),
159
+ "motion_model_params": {
160
+ "lead_compensation": self._motion_model_registry.get_model_params("lead_compensation"),
161
+ },
162
+ "preferred_duration": self._preferred_duration,
163
+ "welcome_messages": self._welcome_messages,
164
+ "datasets": [
165
+ {
166
+ "id": entry.dataset_id,
167
+ "label": entry.label,
168
+ "folder": entry.folder,
169
+ "path": str(entry.path),
170
+ "uploaded_move_ids": list(entry.uploaded_move_ids or []),
171
+ "origin": entry.origin,
172
+ }
173
+ for entry in self._datasets.values()
174
+ ],
175
+ }
176
+ self._registry_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
177
+
178
+ def _resolve_dataset_path(self, raw_path: str | Path) -> Path:
179
+ path = Path(raw_path).expanduser()
180
+ try:
181
+ path = path.resolve(strict=False)
182
+ except Exception:
183
+ path = path.expanduser().absolute()
184
+ return path
185
+
186
+ def _next_dataset_id(self, label: str, used: set[str]) -> str:
187
+ base = _slugify(label or "dataset")
188
+ candidate = base or "dataset"
189
+ suffix = 1
190
+ while candidate in used:
191
+ candidate = f"{base}-{suffix}"
192
+ suffix += 1
193
+ used.add(candidate)
194
+ return candidate
195
+
196
+ def _entry_from_folder(
197
+ self,
198
+ folder: str,
199
+ label: str,
200
+ dataset_id: str,
201
+ uploaded_move_ids: set[str] | None = None,
202
+ origin: str = "local",
203
+ ) -> DatasetEntry:
204
+ path = self._dataset_root / folder
205
+ entry = DatasetEntry(
206
+ dataset_id=dataset_id,
207
+ label=label,
208
+ folder=folder,
209
+ path=path,
210
+ uploaded_move_ids=uploaded_move_ids or set(),
211
+ origin=origin,
212
+ )
213
+ self._ensure_dataset_data_dir(entry)
214
+ return entry
215
+
216
+ def _default_dataset_root(self) -> Path:
217
+ system = platform.system().lower()
218
+ home = Path.home()
219
+ if "windows" in system:
220
+ base = home / "Documents" / "ReachyMini" / "datasets"
221
+ elif "darwin" in system:
222
+ base = home / "Library" / "Application Support" / "ReachyMini" / "datasets"
223
+ else:
224
+ base = home / "reachy_mini_datasets"
225
+ return base
226
+
227
+ def _ensure_dataset_data_dir(self, entry: DatasetEntry) -> Path:
228
+ """Ensure the dataset layout (root/folder/data/) exists and migrate legacy files.
229
+
230
+ Early versions stored .json/.wav directly in the dataset folder.
231
+ This migrates them into the data/ subdirectory for consistency with
232
+ the HF dataset layout.
233
+ """
234
+ entry.path.mkdir(parents=True, exist_ok=True)
235
+ data_dir = entry.path / DATASET_DATA_SUBDIR
236
+ data_dir.mkdir(parents=True, exist_ok=True)
237
+ legacy_moves = list(entry.path.glob("*.json"))
238
+ for json_path in legacy_moves:
239
+ target = data_dir / json_path.name
240
+ if not target.exists():
241
+ shutil.move(str(json_path), str(target))
242
+ for suffix in (".wav",):
243
+ source = json_path.with_suffix(suffix)
244
+ if source.exists():
245
+ shutil.move(str(source), str(data_dir / source.name))
246
+ return data_dir
247
+
248
+ def _create_dataset_internal(
249
+ self,
250
+ folder: str,
251
+ label: str,
252
+ *,
253
+ dataset_id: str | None = None,
254
+ save: bool = True,
255
+ origin: str = "local",
256
+ ) -> DatasetEntry:
257
+ folder_slug = _slugify(folder or label or "dataset")
258
+ if any(entry.folder == folder_slug for entry in self._datasets.values()):
259
+ if dataset_id is None:
260
+ raise HTTPException(status_code=409, detail=f"Dataset folder '{folder_slug}' already exists.")
261
+ path = self._dataset_root / folder_slug
262
+ path.mkdir(parents=True, exist_ok=True)
263
+
264
+ if dataset_id is None:
265
+ base = _slugify(label or folder_slug or "dataset")
266
+ dataset_id = base or "dataset"
267
+ suffix = 1
268
+ while dataset_id in self._datasets:
269
+ dataset_id = f"{base}-{suffix}"
270
+ suffix += 1
271
+
272
+ entry = DatasetEntry(
273
+ dataset_id=dataset_id, label=label or dataset_id, folder=folder_slug, path=path, origin=origin
274
+ )
275
+ self._datasets[entry.dataset_id] = entry
276
+ self._ensure_dataset_data_dir(entry)
277
+ if save:
278
+ self._save_dataset_registry()
279
+ return entry
280
+
281
+ def _create_dataset(self, name: str, label: str | None) -> DatasetEntry:
282
+ if not name:
283
+ raise HTTPException(status_code=400, detail="Dataset name is required.")
284
+ folder = _slugify(name)
285
+ if not folder:
286
+ raise HTTPException(status_code=400, detail="Dataset name is invalid.")
287
+ if any(entry.folder == folder for entry in self._datasets.values()):
288
+ raise HTTPException(status_code=409, detail="Dataset name already used.")
289
+ label_to_use = label or name or DEFAULT_DATASET_LABEL
290
+ entry = self._create_dataset_internal(folder, label_to_use)
291
+ return entry
292
+
293
+ def _select_dataset(self, dataset_id: str) -> None:
294
+ entry = self._datasets.get(dataset_id)
295
+ if entry is None:
296
+ raise HTTPException(status_code=404, detail=f"Dataset {dataset_id} not found.")
297
+ self._active_dataset_id = dataset_id
298
+ self._dataset_dir = self._ensure_dataset_data_dir(entry)
299
+ self._save_dataset_registry()
300
+
301
+ def _datasets_payload(self) -> dict[str, Any]:
302
+ entries = sorted(
303
+ [entry.to_payload() for entry in self._datasets.values()],
304
+ key=lambda e: (e.get("label") or e["id"]).lower(),
305
+ )
306
+ return {
307
+ "active_id": self._active_dataset_id,
308
+ "root_path": str(self._dataset_root),
309
+ "entries": entries,
310
+ }
311
+
312
+ def _set_dataset_root(self, raw_path: str) -> None:
313
+ new_root = self._resolve_dataset_path(raw_path)
314
+ new_root.mkdir(parents=True, exist_ok=True)
315
+ self._dataset_root = new_root
316
+ logger.info("Dataset root updated to %s", self._dataset_root)
317
+ for entry in self._datasets.values():
318
+ entry.path = self._dataset_root / entry.folder
319
+ self._ensure_dataset_data_dir(entry)
320
+ if self._active_dataset_id not in self._datasets and self._datasets:
321
+ self._active_dataset_id = next(iter(self._datasets))
322
+ if self._active_dataset_id:
323
+ self._dataset_dir = self._ensure_dataset_data_dir(self._datasets[self._active_dataset_id])
324
+ else:
325
+ default = self._create_dataset_internal(DATASET_DIRNAME, DEFAULT_DATASET_LABEL)
326
+ self._active_dataset_id = default.dataset_id
327
+ self._dataset_dir = self._ensure_dataset_data_dir(default)
328
+ self._save_dataset_registry()
329
+
330
+ def _check_hf_login(self) -> str | None:
331
+ """Return the logged-in Hugging Face username, or None.
332
+
333
+ Uses a two-tier check:
334
+ 1. Local: hf_get_token() reads ~/.cache/huggingface/token (no network).
335
+ If no token exists, we're definitely not logged in.
336
+ 2. Network: hf_whoami() fetches the username from the HF API.
337
+ If this fails (timeout, rate limit, network down), we still report
338
+ as logged in (token exists) — just without a username.
339
+
340
+ Results are cached so we don't hit the API on every poll.
341
+ """
342
+ if self._hf_checked:
343
+ return self._hf_username
344
+ self._hf_checked = True
345
+ # Tier 1: local token check (no network)
346
+ if hf_get_token is None or not hf_get_token():
347
+ self._hf_username = None
348
+ return None
349
+ # Tier 2: try to get the username (network call)
350
+ if hf_whoami is not None:
351
+ try:
352
+ info = hf_whoami()
353
+ self._hf_username = info.get("name") if isinstance(info, dict) else None
354
+ except Exception:
355
+ # Network failed but token exists — report as logged in
356
+ self._hf_username = "(token saved)"
357
+ else:
358
+ self._hf_username = "(token saved)"
359
+ return self._hf_username
360
+
361
+ def _ensure_hf_backend(self, purpose: str) -> None:
362
+ global HfApi, DatasetFilter, snapshot_download
363
+ if HfApi is not None and snapshot_download is not None:
364
+ return
365
+ try:
366
+ from huggingface_hub import HfApi as _HfApi, snapshot_download as _snapshot_download
367
+ try:
368
+ from huggingface_hub import DatasetFilter as _DatasetFilter
369
+ except Exception:
370
+ _DatasetFilter = DatasetFilter # keep existing (possibly None)
371
+ except Exception as exc: # pragma: no cover - optional dependency
372
+ logger.warning("huggingface_hub unavailable for %s: %s", purpose, exc)
373
+ raise HTTPException(
374
+ status_code=500,
375
+ detail=f"huggingface_hub is not installed. Install it to {purpose}",
376
+ ) from exc
377
+ DatasetFilter = _DatasetFilter
378
+ HfApi = _HfApi
379
+ snapshot_download = _snapshot_download
380
+
381
+ def _sync_dataset(self, hf_username: str, move_ids: list[str], dataset_slug: str | None) -> dict[str, Any]:
382
+ self._ensure_hf_backend("publish datasets.")
383
+ if not move_ids:
384
+ raise HTTPException(status_code=400, detail="No moves selected for synchronization.")
385
+
386
+ self._refresh_recordings()
387
+ missing = [move_id for move_id in move_ids if move_id not in self._recordings]
388
+ if missing:
389
+ raise HTTPException(
390
+ status_code=404,
391
+ detail=f"Moves not found in active dataset: {', '.join(missing)}",
392
+ )
393
+
394
+ entry = self._datasets.get(self._active_dataset_id or "")
395
+ if entry is None:
396
+ raise HTTPException(status_code=404, detail="No active dataset.")
397
+
398
+ repo_basename = dataset_slug or _slugify(entry.label or entry.dataset_id)
399
+ repo_id = f"{hf_username}/{repo_basename}"
400
+ exported_bytes = 0
401
+ selected_meta = [self._recordings[move_id] for move_id in move_ids]
402
+
403
+ with TemporaryDirectory(prefix="marionette-sync-") as tmpdir:
404
+ export_root = Path(tmpdir)
405
+ data_dir = export_root / "data"
406
+ data_dir.mkdir(parents=True, exist_ok=True)
407
+ for meta in selected_meta:
408
+ src_json = meta.json_path
409
+ dest_json = data_dir / src_json.name
410
+ shutil.copy2(src_json, dest_json)
411
+ exported_bytes += dest_json.stat().st_size
412
+ wav_path = src_json.with_suffix(".wav")
413
+ if meta.has_audio and wav_path.exists():
414
+ dest_wav = data_dir / wav_path.name
415
+ shutil.copy2(wav_path, dest_wav)
416
+ exported_bytes += dest_wav.stat().st_size
417
+
418
+ readme_path = export_root / "README.md"
419
+ readme_path.write_text(
420
+ self._build_hf_readme(entry, selected_meta, repo_id, exported_bytes),
421
+ encoding="utf-8",
422
+ )
423
+
424
+ api = HfApi()
425
+ try:
426
+ api.create_repo(
427
+ repo_id=repo_id,
428
+ repo_type="dataset",
429
+ exist_ok=True,
430
+ private=False,
431
+ )
432
+ api.upload_folder(
433
+ folder_path=str(export_root),
434
+ repo_id=repo_id,
435
+ repo_type="dataset",
436
+ )
437
+ except Exception as exc: # pragma: no cover - network failure
438
+ raise HTTPException(status_code=502, detail=f"Failed to upload dataset: {exc}") from exc
439
+
440
+ # Mark moves as uploaded
441
+ if entry.uploaded_move_ids is None:
442
+ entry.uploaded_move_ids = set()
443
+ for meta in selected_meta:
444
+ entry.uploaded_move_ids.add(meta.move_id)
445
+ self._save_dataset_registry()
446
+ self._refresh_recordings()
447
+
448
+ url = f"https://huggingface.co/datasets/{repo_id}"
449
+ return {
450
+ "status": "synced",
451
+ "repo_id": repo_id,
452
+ "uploaded_moves": len(selected_meta),
453
+ "url": url,
454
+ }
455
+
456
+ def _download_community_dataset(self, payload: DownloadDatasetPayload) -> DatasetEntry:
457
+ self._ensure_hf_backend("download datasets.")
458
+ repo_id = payload.repo_id.strip()
459
+ logger.info("Downloading community dataset %s", repo_id)
460
+ if "/" not in repo_id:
461
+ raise HTTPException(status_code=400, detail="repo_id must include the username (e.g. user/name).")
462
+ owner, repo_name = repo_id.split("/", 1)
463
+ folder_base = payload.name or f"{owner}-{repo_name}"
464
+ folder = _slugify(folder_base)
465
+ if not folder:
466
+ folder = _slugify(repo_id.replace("/", "-"))
467
+ existing = next((e for e in self._datasets.values() if e.folder == folder), None)
468
+ if existing is not None:
469
+ # Remove the old registry entry so we can re-download with updated content.
470
+ logger.info("Re-downloading dataset '%s' — removing old entry", folder)
471
+ del self._datasets[existing.dataset_id]
472
+ self._save_dataset_registry()
473
+ target_path = self._dataset_root / folder
474
+ if target_path.exists():
475
+ shutil.rmtree(target_path)
476
+ target_path.mkdir(parents=True, exist_ok=True)
477
+
478
+ try:
479
+ snapshot_download(
480
+ repo_id=repo_id,
481
+ repo_type="dataset",
482
+ local_dir=str(target_path),
483
+ )
484
+ except Exception as exc: # pragma: no cover - network failure
485
+ logger.exception("Failed to download dataset %s", repo_id)
486
+ raise HTTPException(status_code=502, detail=f"Failed to download dataset: {exc}") from exc
487
+
488
+ label = payload.label or repo_id
489
+ entry = self._create_dataset_internal(folder=folder, label=label, origin="downloaded")
490
+ self._select_dataset(entry.dataset_id)
491
+ logger.info("Dataset %s downloaded into %s", repo_id, entry.path)
492
+ return entry
493
+
494
+ def _list_community_datasets(self) -> list[dict[str, Any]]:
495
+ datasets: list[Any] = self._fetch_community_datasets_http()
496
+ if not datasets and HfApi is not None:
497
+ dataset_filter = None
498
+ if DatasetFilter is not None:
499
+ try:
500
+ dataset_filter = DatasetFilter(tags=[COMMUNITY_DATASET_TAG])
501
+ except Exception: # pragma: no cover - incompatible hub version
502
+ logger.debug("DatasetFilter unavailable, falling back to text search.")
503
+ dataset_filter = None
504
+ search_query = None if dataset_filter else COMMUNITY_DATASET_TAG
505
+ logger.info("Listing community datasets via HfApi for tag %s", COMMUNITY_DATASET_TAG)
506
+ try:
507
+ api = HfApi()
508
+ datasets = api.list_datasets(
509
+ filter=dataset_filter,
510
+ search=search_query,
511
+ limit=MAX_COMMUNITY_DATASETS,
512
+ full=True,
513
+ )
514
+ except Exception as exc: # pragma: no cover - network failure
515
+ logger.warning("HfApi dataset listing failed: %s", exc)
516
+ datasets = []
517
+ if not datasets:
518
+ raise HTTPException(status_code=502, detail="Unable to list community datasets from Hugging Face.")
519
+
520
+ filtered_items: list[Any] = []
521
+ for item in datasets:
522
+ tags = getattr(item, "tags", None) or item.get("tags") if isinstance(item, dict) else []
523
+ if tags and COMMUNITY_DATASET_TAG in tags:
524
+ filtered_items.append(item)
525
+ if not filtered_items:
526
+ filtered_items = datasets
527
+
528
+ results = []
529
+ for item in filtered_items:
530
+ repo_id = (
531
+ getattr(item, "id", None)
532
+ or getattr(item, "repo_id", None)
533
+ or (item.get("id") if isinstance(item, dict) else None)
534
+ or (item.get("repo_id") if isinstance(item, dict) else None)
535
+ )
536
+ if not repo_id:
537
+ continue
538
+ card = (
539
+ getattr(item, "cardData", None)
540
+ or getattr(item, "card_data", None)
541
+ or (item.get("cardData") if isinstance(item, dict) else None)
542
+ or {}
543
+ )
544
+ pretty = card.get("pretty_name") or repo_id
545
+ description = card.get("short_description") or card.get("description") or item.get("description", "")
546
+ updated = getattr(item, "lastModified", None) or item.get("lastModified")
547
+ if isinstance(updated, datetime):
548
+ updated_str = updated.isoformat()
549
+ else:
550
+ updated_str = str(updated) if updated else None
551
+
552
+ results.append(
553
+ {
554
+ "repo_id": repo_id,
555
+ "pretty_name": pretty,
556
+ "description": description,
557
+ "author": getattr(item, "author", None) or item.get("author"),
558
+ "likes": getattr(item, "likes", None) or item.get("likes"),
559
+ "downloads": getattr(item, "downloads", None) or item.get("downloads"),
560
+ "last_modified": updated_str,
561
+ "tags": getattr(item, "tags", None) or item.get("tags"),
562
+ }
563
+ )
564
+ return results
565
+
566
+ def _fetch_community_datasets_http(self) -> list[dict[str, Any]]:
567
+ """Try multiple HF API search strategies to find community datasets.
568
+
569
+ The HF API has changed its search syntax several times. We try three
570
+ approaches in order (tag:, keyword, legacy filter) and return the
571
+ first one that yields results. This makes us resilient to API changes.
572
+ """
573
+ base_params = {
574
+ "limit": MAX_COMMUNITY_DATASETS,
575
+ "full": "true",
576
+ "sort": "downloads",
577
+ "direction": "-1",
578
+ }
579
+ attempts = [
580
+ ("tag_search", {"search": f"tag:{COMMUNITY_DATASET_TAG}"}),
581
+ ("keyword_search", {"search": COMMUNITY_DATASET_TAG}),
582
+ ("legacy_filter", {"filter": COMMUNITY_DATASET_TAG}),
583
+ ]
584
+ for label, extra_params in attempts:
585
+ params = base_params.copy()
586
+ params.update(extra_params)
587
+ try:
588
+ import requests # lazy import — only needed for community datasets
589
+ logger.debug("HTTP dataset listing (%s) with params %s", label, params)
590
+ resp = requests.get(
591
+ HF_DATASETS_API_URL,
592
+ params=params,
593
+ timeout=30,
594
+ )
595
+ resp.raise_for_status()
596
+ data = resp.json()
597
+ if isinstance(data, list) and data:
598
+ logger.info("Fetched %d community datasets via HTTP (%s)", len(data), label)
599
+ return data
600
+ if isinstance(data, list):
601
+ logger.debug("HTTP dataset listing (%s) returned zero results.", label)
602
+ except Exception as exc:
603
+ logger.warning("HTTP dataset listing (%s) failed: %s", label, exc)
604
+ return []
605
+
606
+ def _build_hf_readme(
607
+ self,
608
+ dataset_entry: DatasetEntry,
609
+ selected_meta: list[RecordingMetadata],
610
+ repo_id: str,
611
+ exported_bytes: int,
612
+ ) -> str:
613
+ pretty_name = f"{dataset_entry.label} • Reachy Mini Moves"
614
+ num_examples = len(selected_meta)
615
+ total_duration = sum(meta.duration for meta in selected_meta)
616
+ audio_count = sum(1 for meta in selected_meta if meta.has_audio)
617
+ latest_timestamp = max((meta.created_at for meta in selected_meta), default=time.time())
618
+ created_iso = datetime.utcfromtimestamp(latest_timestamp).strftime("%Y-%m-%dT%H:%M:%SZ")
619
+ move_rows = []
620
+ for meta in selected_meta:
621
+ recorded_at = datetime.utcfromtimestamp(meta.created_at).strftime("%Y-%m-%d %H:%M")
622
+ audio_label = "Yes" if meta.has_audio else "No"
623
+ move_rows.append(
624
+ f"| `{meta.move_id}` | {meta.duration:.1f}s | {audio_label} | {recorded_at} |"
625
+ )
626
+ moves_table = "\n".join(move_rows) if move_rows else "| – | – | – | – |"
627
+ front_matter = dedent(
628
+ f"""\
629
+ ---
630
+ dataset_info:
631
+ features:
632
+ - name: move_id
633
+ dtype: string
634
+ - name: description
635
+ dtype: string
636
+ - name: duration_seconds
637
+ dtype: float64
638
+ - name: has_audio
639
+ dtype: bool
640
+ splits:
641
+ - name: train
642
+ num_examples: {num_examples}
643
+ num_bytes: {exported_bytes}
644
+ download_size: {exported_bytes}
645
+ dataset_size: {exported_bytes}
646
+ configs:
647
+ - config_name: default
648
+ data_files:
649
+ - split: train
650
+ path: data/*.json
651
+ task_categories:
652
+ - robotics
653
+ language:
654
+ - en
655
+ tags:
656
+ - reachy_mini_community_moves
657
+ pretty_name: {pretty_name}
658
+ license: apache-2.0
659
+ ---
660
+ """
661
+ ).strip()
662
+
663
+ body = dedent(
664
+ f"""
665
+ # {pretty_name}
666
+
667
+ Community-contributed Marionette recordings captured on Reachy Mini.
668
+
669
+ - **Moves uploaded:** {num_examples}
670
+ - **Total motion time:** {total_duration:.1f} seconds
671
+ - **Audio tracks:** {audio_count}
672
+ - **Last updated:** {created_iso}
673
+
674
+ Files live under `data/` — each move ships as a JSON trajectory (Reachy Mini emotions schema) plus an optional WAV recorded directly from the robot.
675
+
676
+ ## How this dataset was produced
677
+
678
+ These takes were recorded with the Marionette Reachy Mini app. Pick the moves to share, set your Hugging Face username, run `huggingface-cli login` once locally, then hit **Synchronize to Hugging Face dataset** inside Marionette. The app packages the selected files, generates this README, and uploads them to `{repo_id}`.
679
+
680
+ ## Selected moves
681
+
682
+ | Move | Duration | Audio | Recorded at (UTC) |
683
+ | --- | --- | --- | --- |
684
+ {moves_table}
685
+
686
+ ## Reuse
687
+
688
+ - Cite this dataset as `{repo_id}`.
689
+ - Keep the `reachy_mini_community_moves` tag when sharing derivatives so the community can discover related sets.
690
+ """
691
+ ).strip()
692
+ return f"{front_matter}\n\n{body}\n"
693
+
694
+ def _delete_move_files(self, move_id: str) -> None:
695
+ json_path = self._dataset_dir / f"{move_id}.json"
696
+ if not json_path.exists():
697
+ raise HTTPException(status_code=404, detail=f"Move {move_id} not found.")
698
+ try:
699
+ json_path.unlink()
700
+ # Delete associated audio files (including legacy denoised/noise files)
701
+ for suffix in (".wav", ".noise.wav", ".denoised.wav"):
702
+ path = self._dataset_dir / f"{move_id}{suffix}"
703
+ if path.exists():
704
+ path.unlink()
705
+ except OSError as exc:
706
+ raise HTTPException(status_code=500, detail=f"Failed to delete {move_id}: {exc}") from exc
marionette/main.py CHANGED
@@ -1,2225 +1,67 @@
1
- from __future__ import annotations
2
-
3
- import time as _time_mod
4
- _BOOT_T0 = _time_mod.perf_counter()
5
-
6
- import json
7
- import logging
8
- import platform
9
- import re
10
- import shutil
11
- import threading
12
- import time
13
- import uuid
14
- from dataclasses import dataclass
15
- from datetime import datetime
16
- from pathlib import Path
17
- from tempfile import TemporaryDirectory
18
- from textwrap import dedent
19
- from typing import Any, Callable
20
-
21
- _t = _time_mod.perf_counter(); import numpy as np; _BOOT_NUMPY = _time_mod.perf_counter() - _t # noqa: E702
22
- _BOOT_REQUESTS = 0.0 # requests is now lazy-imported (only needed for HF API calls)
23
- _t = _time_mod.perf_counter(); from fastapi import HTTPException, UploadFile, File; _BOOT_FASTAPI = _time_mod.perf_counter() - _t # noqa: E702
24
- _t = _time_mod.perf_counter(); from pydantic import BaseModel, Field; _BOOT_PYDANTIC = _time_mod.perf_counter() - _t # noqa: E702
25
 
26
- _t = _time_mod.perf_counter()
27
- from reachy_mini import ReachyMini, ReachyMiniApp
28
- from reachy_mini.motion.recorded_move import RecordedMove
29
- from reachy_mini.reachy_mini import INIT_HEAD_POSE, SLEEP_HEAD_POSE, SLEEP_ANTENNAS_JOINT_POSITIONS
30
- from reachy_mini.utils.interpolation import distance_between_poses
31
- _BOOT_REACHY = _time_mod.perf_counter() - _t
32
-
33
- _t = _time_mod.perf_counter(); from marionette.motion_models import MotionModelRegistry; _BOOT_MOTION = _time_mod.perf_counter() - _t # noqa: E702
34
-
35
- try:
36
- from importlib.metadata import version as _pkg_version
37
- __version__ = _pkg_version("marionette")
38
- except Exception:
39
- __version__ = "unknown"
40
 
41
- # Semi-awaken pose: identity rotation, same XY as sleep, Z raised 1.5cm
42
- SEMI_AWAKEN_POSE = np.array(
43
- [
44
- [1.0, 0.0, 0.0, -0.021],
45
- [0.0, 1.0, 0.0, 0.001],
46
- [0.0, 0.0, 1.0, -0.029],
47
- [0.0, 0.0, 0.0, 1.0],
48
- ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  )
50
 
51
- _t = _time_mod.perf_counter()
52
- try:
53
- import soundfile as sf
54
- except Exception: # pragma: no cover - optional dependency
55
- sf = None
56
- _BOOT_SOUNDFILE = _time_mod.perf_counter() - _t
57
-
58
- _BOOT_SCIPY = 0.0 # scipy is now lazy-imported (only needed for user-uploaded audio)
59
-
60
- _t = _time_mod.perf_counter()
61
- try:
62
- from huggingface_hub import HfApi, snapshot_download, whoami as hf_whoami
63
- from huggingface_hub import login as hf_login, logout as hf_logout
64
- from huggingface_hub import get_token as hf_get_token
65
- except Exception: # pragma: no cover - optional dependency
66
- HfApi = None
67
- snapshot_download = None
68
- hf_whoami = None
69
- hf_login = None
70
- hf_logout = None
71
- hf_get_token = None
72
- _BOOT_HF = _time_mod.perf_counter() - _t
73
-
74
- try:
75
- from huggingface_hub import DatasetFilter
76
- except Exception: # pragma: no cover - optional helper
77
- DatasetFilter = None
78
-
79
- _BOOT_IMPORTS_TOTAL = _time_mod.perf_counter() - _BOOT_T0
80
-
81
- # Log import times at module level (before logger is configured, use print)
82
- print(
83
- f"[BOOT] imports: total={_BOOT_IMPORTS_TOTAL:.2f}s | "
84
- f"numpy={_BOOT_NUMPY:.2f}s requests={_BOOT_REQUESTS:.2f}s "
85
- f"fastapi={_BOOT_FASTAPI:.2f}s pydantic={_BOOT_PYDANTIC:.2f}s "
86
- f"reachy_mini={_BOOT_REACHY:.2f}s motion_models={_BOOT_MOTION:.2f}s "
87
- f"soundfile={_BOOT_SOUNDFILE:.2f}s scipy={_BOOT_SCIPY:.2f}s "
88
- f"huggingface_hub={_BOOT_HF:.2f}s",
89
- flush=True,
90
  )
91
 
92
- AUDIO_SAMPLE_RATE = 44_100
93
- MOTION_SAMPLE_RATE = 100.0
94
- COUNTDOWN_SECONDS = 3
95
- DEFAULT_DURATION = 5.0
96
- DATASET_DIRNAME = "local_dataset"
97
- DATASET_REGISTRY_FILENAME = "dataset_registry.json"
98
- DEFAULT_DATASET_LABEL = "Local dataset"
99
- COMMUNITY_DATASET_TAG = "reachy_mini_community_moves"
100
- MAX_COMMUNITY_DATASETS = 50
101
- HF_DATASETS_API_URL = "https://huggingface.co/api/datasets"
102
- DATASET_DATA_SUBDIR = "data"
103
- NOISE_SUFFIX = ".noise.wav"
104
- DENOISED_SUFFIX = ".denoised.wav"
105
-
106
- logger = logging.getLogger(__name__)
107
-
108
-
109
- def _slugify(value: str) -> str:
110
- slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
111
- return slug or "take"
112
-
113
-
114
- @dataclass
115
- class RecordingMetadata:
116
- move_id: str
117
- label: str
118
- description: str
119
- duration: float
120
- created_at: float
121
- json_path: Path
122
- has_audio: bool
123
- has_noise_profile: bool
124
- has_denoised_audio: bool
125
- is_uploaded: bool = False
126
- audio_only: bool = False
127
-
128
- def to_payload(self) -> dict[str, Any]:
129
- return {
130
- "id": self.move_id,
131
- "label": self.label,
132
- "duration": self.duration,
133
- "created_at": self.created_at,
134
- "has_audio": self.has_audio,
135
- "description": self.description,
136
- "has_noise_profile": self.has_noise_profile,
137
- "has_denoised_audio": self.has_denoised_audio,
138
- "is_uploaded": self.is_uploaded,
139
- "audio_only": self.audio_only,
140
- }
141
-
142
-
143
- @dataclass
144
- class RecordingRequest:
145
- move_id: str
146
- label: str
147
- description: str
148
- duration: float
149
- record_audio: bool
150
- record_motion: bool = True
151
- uploaded_audio_path: Path | None = None
152
-
153
-
154
- @dataclass
155
- class DatasetEntry:
156
- dataset_id: str
157
- label: str
158
- folder: str
159
- path: Path
160
- uploaded_move_ids: set[str] | None = None
161
- origin: str = "local" # "local" or "downloaded"
162
-
163
- def __post_init__(self) -> None:
164
- if self.uploaded_move_ids is None:
165
- self.uploaded_move_ids = set()
166
-
167
- def to_payload(self) -> dict[str, Any]:
168
- return {
169
- "id": self.dataset_id,
170
- "label": self.label,
171
- "path": str(self.path),
172
- "folder": self.folder,
173
- "origin": self.origin,
174
- }
175
-
176
-
177
- class StartRecordingPayload(BaseModel):
178
- duration: float = Field(DEFAULT_DURATION, gt=0.5, le=300.0)
179
- record_audio: bool = Field(default=True)
180
- record_motion: bool = Field(default=True, description="When False, record audio only (no motion capture)")
181
- label: str | None = Field(default=None, max_length=80)
182
- description: str | None = Field(default=None, max_length=500)
183
- uploaded_audio_id: str | None = Field(default=None, description="ID of previously uploaded audio file")
184
-
185
-
186
- class PlayMovePayload(BaseModel):
187
- move_id: str = Field(..., description="Move identifier (filename stem)")
188
-
189
-
190
- class CreateDatasetPayload(BaseModel):
191
- name: str = Field(..., description="Folder name for the dataset", min_length=1, max_length=80)
192
- label: str | None = Field(default=None, max_length=80)
193
-
194
-
195
- class SelectDatasetPayload(BaseModel):
196
- dataset_id: str
197
-
198
-
199
- class SyncDatasetPayload(BaseModel):
200
- move_ids: list[str] = Field(..., min_items=1, description="Subset of moves to publish")
201
- hf_username: str | None = Field(default=None, min_length=2, max_length=80)
202
- dataset_slug: str | None = Field(
203
- default=None, description="Optional override for the Hugging Face dataset slug"
204
- )
205
-
206
-
207
- class UpdateDatasetRootPayload(BaseModel):
208
- path: str = Field(..., description="Filesystem directory containing all datasets")
209
-
210
-
211
- class DownloadDatasetPayload(BaseModel):
212
- repo_id: str = Field(..., description="Hugging Face dataset repository, e.g. user/name")
213
- label: str | None = Field(default=None, max_length=80)
214
- name: str | None = Field(
215
- default=None,
216
- description="Optional folder name; defaults to the dataset slug",
217
- max_length=80,
218
- )
219
-
220
-
221
- class UpdateLeadCompensationPayload(BaseModel):
222
- lead_frames_head: int | None = Field(
223
- default=None, ge=0, le=2000, description="Look-ahead for head/body in 100Hz frames"
224
- )
225
- lead_frames_antennas: int | None = Field(
226
- default=None, ge=0, le=2000, description="Look-ahead for antennas in 100Hz frames"
227
- )
228
-
229
-
230
- class UpdateExperimentsPayload(BaseModel):
231
- duration_seconds: float | None = Field(default=None, gt=0.5, le=300.0)
232
- welcome_messages: int | None = Field(default=None, ge=0, le=2, description="Number of welcome messages at startup (0, 1, or 2)")
233
-
234
-
235
- class HfTokenPayload(BaseModel):
236
- token: str = Field(..., min_length=5, description="Hugging Face access token (starts with hf_)")
237
-
238
-
239
- class Marionette(ReachyMiniApp):
240
- """Manual Marionette recorder for Reachy Mini."""
241
-
242
- custom_app_url: str | None = "http://0.0.0.0:8042"
243
-
244
- def __init__(
245
- self,
246
- registry_path: Path | None = None,
247
- dataset_root: Path | None = None,
248
- ) -> None:
249
- super().__init__()
250
- self._registry_path = registry_path or (
251
- Path(__file__).resolve().parent.parent / DATASET_REGISTRY_FILENAME
252
- )
253
- self._dataset_root_override = dataset_root
254
- self._datasets: dict[str, DatasetEntry] = {}
255
- self._active_dataset_id: str | None = None
256
- self._dataset_root: Path
257
- self._dataset_dir: Path
258
- self._motion_model_registry = MotionModelRegistry()
259
- self._preferred_duration: float = DEFAULT_DURATION
260
- self._welcome_messages: int = 2 # 0=none, 1=intro only, 2=intro+second
261
- self._load_dataset_registry() # may overwrite _preferred_duration and _welcome_messages
262
-
263
- self._recordings: dict[str, RecordingMetadata] = {}
264
- self._pending_recording: RecordingRequest | None = None
265
- self._pending_playback: str | None = None
266
- self._uploaded_audio: dict[str, Path] = {} # upload_id -> temp file path
267
- self._playback_cancel_event = threading.Event()
268
- self._recording_cancel_event = threading.Event()
269
- self._mode: str = "idle"
270
- self._message: str = "Ready to capture moves"
271
- self._countdown_ends_at: float | None = None
272
- self._active_record_started_at: float | None = None
273
- self._active_record_duration: float | None = None
274
- self._active_move: str | None = None
275
- self._state_lock = threading.Lock()
276
- self._audio_available = sf is not None
277
- self._recording_stats: dict[str, Any] | None = None
278
- self._hf_username: str | None = None
279
- self._hf_checked = False
280
-
281
- self._refresh_recordings()
282
- print(f"[BOOT] __init__ done at +{time.perf_counter() - _BOOT_T0:.2f}s after module import", flush=True)
283
- logger.info("Marionette v%s starting on %s (%s)", __version__, platform.node(), platform.system())
284
- if self.settings_app is not None:
285
- self._register_routes()
286
-
287
- # ──────── FastAPI routes ────────────────────────────────────────────
288
- def _register_routes(self) -> None:
289
- assert self.settings_app is not None
290
-
291
- @self.settings_app.get("/api/state")
292
- def get_state() -> dict[str, Any]:
293
- return self._serialize_state()
294
-
295
- @self.settings_app.get("/api/version")
296
- def get_version() -> dict[str, str]:
297
- return {"version": __version__, "platform": platform.system(), "hostname": platform.node()}
298
-
299
- # Dummy endpoint to silence 404 spam from external tools
300
- @self.settings_app.get("/sensor_data")
301
- def sensor_data() -> dict[str, Any]:
302
- return {}
303
-
304
- @self.settings_app.post("/api/upload-audio")
305
- async def upload_audio(file: UploadFile = File(...)) -> dict[str, Any]:
306
- if not file.filename:
307
- raise HTTPException(status_code=400, detail="No file provided.")
308
- ext = Path(file.filename).suffix.lower()
309
- if ext not in {".wav", ".mp3"}:
310
- raise HTTPException(
311
- status_code=400,
312
- detail=f"Unsupported audio format '{ext}'. Use .wav or .mp3.",
313
- )
314
- upload_id = str(uuid.uuid4())
315
- temp_dir = Path(__file__).resolve().parent.parent / "temp_uploads"
316
- temp_dir.mkdir(parents=True, exist_ok=True)
317
- temp_path = temp_dir / f"{upload_id}{ext}"
318
- try:
319
- content = await file.read()
320
- temp_path.write_bytes(content)
321
- except Exception as exc:
322
- raise HTTPException(status_code=500, detail=f"Failed to save upload: {exc}") from exc
323
-
324
- # Convert MP3 to WAV for better playback quality
325
- duration: float | None = None
326
- if ext == ".mp3" and sf is not None:
327
- try:
328
- data, samplerate = sf.read(str(temp_path))
329
- wav_path = temp_dir / f"{upload_id}.wav"
330
- # Resample to 48kHz for better quality (common output rate)
331
- target_rate = 48000
332
- if samplerate != target_rate:
333
- try:
334
- from scipy import signal
335
- original_rate = samplerate
336
- num_samples = int(len(data) * target_rate / samplerate)
337
- if data.ndim == 1:
338
- data = signal.resample(data, num_samples)
339
- else:
340
- data = signal.resample(data, num_samples, axis=0)
341
- samplerate = target_rate
342
- logger.info("Resampled audio from %d to %d Hz", original_rate, target_rate)
343
- except ImportError:
344
- logger.warning("scipy not available, skipping resample")
345
- # Write as 16-bit PCM WAV for maximum compatibility
346
- sf.write(str(wav_path), data, samplerate, subtype='PCM_16')
347
- duration = len(data) / samplerate
348
- # Remove original MP3 and use WAV
349
- temp_path.unlink()
350
- temp_path = wav_path
351
- logger.info("Converted MP3 to WAV: %s", wav_path.name)
352
- except Exception as exc:
353
- logger.warning("MP3 to WAV conversion failed, using original: %s", exc)
354
-
355
- # Get audio duration if not already determined
356
- if duration is None and sf is not None:
357
- try:
358
- info = sf.info(str(temp_path))
359
- duration = float(info.duration)
360
- except Exception as exc:
361
- logger.warning("Could not determine audio duration: %s", exc)
362
-
363
- self._uploaded_audio[upload_id] = temp_path
364
- return {"upload_id": upload_id, "filename": file.filename, "duration": duration}
365
-
366
- @self.settings_app.post("/api/record")
367
- def start_recording(payload: StartRecordingPayload) -> dict[str, Any]:
368
- uploaded_audio_path: Path | None = None
369
- if payload.uploaded_audio_id:
370
- uploaded_audio_path = self._uploaded_audio.get(payload.uploaded_audio_id)
371
- if not uploaded_audio_path or not uploaded_audio_path.exists():
372
- raise HTTPException(
373
- status_code=400,
374
- detail="Uploaded audio file not found. Please re-upload.",
375
- )
376
- elif payload.record_audio and not self._audio_available:
377
- raise HTTPException(
378
- status_code=400,
379
- detail="Audio capture backend unavailable.",
380
- )
381
-
382
- active_entry = self._datasets.get(self._active_dataset_id or "")
383
- if active_entry and active_entry.origin == "downloaded":
384
- raise HTTPException(
385
- status_code=409,
386
- detail="Cannot record into a downloaded dataset. Switch to a local dataset or create a new one.",
387
- )
388
-
389
- request = self._build_recording_request(payload, uploaded_audio_path)
390
-
391
- with self._state_lock:
392
- if (
393
- self._mode not in {"idle", "queued"}
394
- or self._pending_recording
395
- or self._pending_playback
396
- ):
397
- raise HTTPException(status_code=409, detail="Robot is busy.")
398
- self._preferred_duration = float(request.duration)
399
- self._pending_recording = request
400
- self._mode = "queued"
401
- self._message = "Recording scheduled"
402
- self._countdown_ends_at = None
403
- self._active_record_started_at = None
404
- self._active_record_duration = None
405
- self._save_dataset_registry()
406
-
407
- return {
408
- "accepted": True,
409
- "move_id": request.move_id,
410
- "label": request.label,
411
- }
412
-
413
- @self.settings_app.post("/api/play")
414
- def play_move(payload: PlayMovePayload) -> dict[str, Any]:
415
- if payload.move_id not in self._recordings:
416
- raise HTTPException(status_code=404, detail="Move not found.")
417
-
418
- with self._state_lock:
419
- if (
420
- self._mode != "idle"
421
- or self._pending_recording
422
- or self._pending_playback
423
- ):
424
- raise HTTPException(status_code=409, detail="Robot is busy.")
425
- self._pending_playback = payload.move_id
426
- self._mode = "queued"
427
- self._message = f"Playback queued for {payload.move_id}"
428
-
429
- return {"accepted": True, "move_id": payload.move_id}
430
-
431
- @self.settings_app.post("/api/play/stop")
432
- def stop_playback() -> dict[str, Any]:
433
- with self._state_lock:
434
- if self._mode != "playing":
435
- return {"stopped": False, "reason": "not_playing"}
436
- self._playback_cancel_event.set()
437
- return {"stopped": True}
438
-
439
- @self.settings_app.post("/api/record/stop")
440
- def stop_recording() -> dict[str, Any]:
441
- with self._state_lock:
442
- if self._mode not in {"recording", "countdown", "queued"}:
443
- return {"stopped": False, "reason": "not_recording"}
444
- if self._mode == "queued":
445
- self._pending_recording = None
446
- self._mode = "idle"
447
- self._message = "Recording cancelled"
448
- return {"stopped": True}
449
- self._recording_cancel_event.set()
450
- return {"stopped": True}
451
-
452
- @self.settings_app.delete("/api/moves/{move_id}")
453
- def delete_move(move_id: str) -> dict[str, Any]:
454
- self._delete_move_files(move_id)
455
- self._refresh_recordings()
456
- return {"status": "deleted", "move_id": move_id}
457
-
458
- @self.settings_app.get("/api/datasets")
459
- def list_datasets() -> dict[str, Any]:
460
- return self._datasets_payload()
461
-
462
- @self.settings_app.post("/api/datasets")
463
- def create_dataset(payload: CreateDatasetPayload) -> dict[str, Any]:
464
- with self._state_lock:
465
- if (
466
- self._mode not in {"idle", "queued"}
467
- or self._pending_recording
468
- or self._pending_playback
469
- ):
470
- raise HTTPException(status_code=409, detail="Robot is busy.")
471
- entry = self._create_dataset(payload.name, payload.label)
472
- self._select_dataset(entry.dataset_id)
473
- self._refresh_recordings()
474
- return {"status": "created", "dataset": entry.to_payload()}
475
-
476
- @self.settings_app.post("/api/datasets/select")
477
- def select_dataset(payload: SelectDatasetPayload) -> dict[str, Any]:
478
- with self._state_lock:
479
- if (
480
- self._mode not in {"idle", "queued"}
481
- or self._pending_recording
482
- or self._pending_playback
483
- ):
484
- raise HTTPException(status_code=409, detail="Robot is busy.")
485
- self._select_dataset(payload.dataset_id)
486
- self._refresh_recordings()
487
- return {"status": "selected", "active_id": payload.dataset_id}
488
-
489
- @self.settings_app.post("/api/datasets/sync")
490
- def sync_dataset(payload: SyncDatasetPayload) -> dict[str, Any]:
491
- with self._state_lock:
492
- if (
493
- self._mode not in {"idle", "queued"}
494
- or self._pending_recording
495
- or self._pending_playback
496
- ):
497
- raise HTTPException(status_code=409, detail="Robot is busy.")
498
- username = payload.hf_username or self._check_hf_login()
499
- if not username:
500
- raise HTTPException(
501
- status_code=400,
502
- detail="No Hugging Face username provided and not logged in via CLI.",
503
- )
504
- result = self._sync_dataset(username, payload.move_ids, payload.dataset_slug)
505
- return result
506
-
507
- @self.settings_app.post("/api/datasets/root")
508
- def update_dataset_root(payload: UpdateDatasetRootPayload) -> dict[str, Any]:
509
- with self._state_lock:
510
- if (
511
- self._mode not in {"idle", "queued"}
512
- or self._pending_recording
513
- or self._pending_playback
514
- ):
515
- raise HTTPException(status_code=409, detail="Robot is busy.")
516
- self._set_dataset_root(payload.path)
517
- self._refresh_recordings()
518
- return {"status": "updated", "root_path": str(self._dataset_root)}
519
-
520
- @self.settings_app.get("/api/datasets/community")
521
- def community_datasets() -> dict[str, Any]:
522
- datasets = self._list_community_datasets()
523
- return {"datasets": datasets}
524
-
525
- @self.settings_app.post("/api/datasets/download")
526
- def download_dataset(payload: DownloadDatasetPayload) -> dict[str, Any]:
527
- with self._state_lock:
528
- if self._mode not in {"idle", "queued"} or self._pending_recording or self._pending_playback:
529
- raise HTTPException(status_code=409, detail="Robot is busy.")
530
- entry = self._download_community_dataset(payload)
531
- self._refresh_recordings()
532
- return {"status": "downloaded", "dataset": entry.to_payload()}
533
-
534
- @self.settings_app.post("/api/experiments")
535
- def update_experiments(payload: UpdateExperimentsPayload) -> dict[str, Any]:
536
- updates = payload.dict(exclude_none=True)
537
- if not updates:
538
- return {"status": "unchanged"}
539
- with self._state_lock:
540
- for key, value in updates.items():
541
- if key == "duration_seconds":
542
- self._preferred_duration = float(value)
543
- elif key == "welcome_messages":
544
- self._welcome_messages = max(0, min(2, int(value)))
545
- self._save_dataset_registry()
546
- return {
547
- "status": "updated",
548
- "preferred_duration": self._preferred_duration,
549
- "motion_models": self._motion_model_registry.to_payload(),
550
- }
551
-
552
- @self.settings_app.post("/api/motion-model/lead")
553
- def update_motion_model_lead(payload: UpdateLeadCompensationPayload) -> dict[str, Any]:
554
- params = payload.dict(exclude_none=True)
555
- if not params:
556
- return {
557
- "status": "unchanged",
558
- "active": self._motion_model_registry.active,
559
- "params": self._motion_model_registry.get_model_params("lead_compensation"),
560
- }
561
- try:
562
- self._motion_model_registry.set_model_params("lead_compensation", params)
563
- except KeyError as exc:
564
- raise HTTPException(status_code=404, detail="Lead compensation model unavailable.") from exc
565
- self._save_dataset_registry()
566
- return {
567
- "status": "updated",
568
- "active": self._motion_model_registry.active,
569
- "params": self._motion_model_registry.get_model_params("lead_compensation"),
570
- }
571
-
572
- @self.settings_app.post("/api/hf-auth/save-token")
573
- def save_hf_token(payload: HfTokenPayload) -> dict[str, Any]:
574
- if hf_login is None:
575
- raise HTTPException(status_code=500, detail="huggingface_hub is not installed.")
576
- token = payload.token.strip()
577
- if not token.startswith("hf_"):
578
- raise HTTPException(status_code=422, detail="Token must start with 'hf_'.")
579
- try:
580
- hf_login(token=token, add_to_git_credential=False)
581
- except Exception as exc:
582
- raise HTTPException(status_code=401, detail=f"Invalid token: {exc}") from exc
583
- # Verify the token works
584
- self._hf_checked = False
585
- username = self._check_hf_login()
586
- if not username:
587
- raise HTTPException(status_code=401, detail="Token saved but could not verify identity.")
588
- return {"status": "logged_in", "username": username}
589
-
590
- @self.settings_app.delete("/api/hf-auth/token")
591
- def delete_hf_token() -> dict[str, Any]:
592
- if hf_logout is None:
593
- raise HTTPException(status_code=500, detail="huggingface_hub is not installed.")
594
- try:
595
- hf_logout()
596
- except Exception as exc:
597
- logger.warning("HF logout error: %s", exc)
598
- self._hf_username = None
599
- self._hf_checked = False
600
- return {"status": "logged_out"}
601
-
602
- # ──────── Main loop ────────────────────────────────────────────────
603
- @staticmethod
604
- def _get_audio_duration(path: Path, fallback: float = 3.0) -> float:
605
- if sf is not None:
606
- try:
607
- info = sf.info(str(path))
608
- return float(info.duration)
609
- except Exception:
610
- pass
611
- return fallback
612
-
613
- @staticmethod
614
- def _play_wav_chunked(
615
- reachy_mini: ReachyMini,
616
- wav_path: Path,
617
- stop_event: threading.Event,
618
- chunk_duration: float = 0.02,
619
- ) -> None:
620
- """Play a WAV file via push_audio_sample() so it can be stopped at any time.
621
-
622
- Unlike play_sound() which creates an orphan GStreamer pipeline that
623
- cannot be interrupted, this loads the WAV, resamples to the output
624
- rate, and pushes small chunks through the stoppable playback pipeline.
625
- """
626
- if sf is None:
627
- return
628
- try:
629
- data, sr = sf.read(str(wav_path), dtype="float32")
630
- except Exception as exc:
631
- logger.warning("Failed to read %s: %s", wav_path, exc)
632
- return
633
-
634
- if data.ndim == 2:
635
- data = data.mean(axis=1)
636
-
637
- # Resample if the file's rate doesn't match the output device.
638
- try:
639
- sr_out = int(reachy_mini.media.get_output_audio_samplerate() or 16000)
640
- except Exception:
641
- sr_out = 16000
642
- if sr != sr_out:
643
- try:
644
- from scipy.signal import resample
645
-
646
- num_samples = int(len(data) * sr_out / sr)
647
- data = resample(data, num_samples).astype(np.float32)
648
- sr = sr_out
649
- except Exception as exc:
650
- logger.warning("Resample failed: %s — playing at original rate", exc)
651
-
652
- reachy_mini.media.start_playing()
653
-
654
- samples_per_chunk = max(1, int(sr * chunk_duration))
655
- play_start = time.perf_counter()
656
- try:
657
- for offset in range(0, len(data), samples_per_chunk):
658
- if stop_event.is_set():
659
- break
660
- chunk = data[offset : offset + samples_per_chunk]
661
- reachy_mini.media.push_audio_sample(chunk)
662
- # Sleep slightly less than chunk duration to keep buffer fed
663
- stop_event.wait(chunk_duration * 0.8)
664
- else:
665
- # All chunks pushed — wait for the audio buffer to drain.
666
- # We pushed ~20% faster than real-time, so the playback
667
- # pipeline still has buffered audio to play out.
668
- audio_duration = len(data) / sr
669
- elapsed = time.perf_counter() - play_start
670
- remaining = audio_duration - elapsed
671
- if remaining > 0:
672
- stop_event.wait(remaining)
673
- finally:
674
- # stop_playing() can hang on some audio backends (GStreamer's
675
- # set_state(NULL) blocks when the device is slow or the MainLoop
676
- # is starved). Run it in a daemon thread with a timeout so we
677
- # never block the caller indefinitely.
678
- try:
679
- t = threading.Thread(
680
- target=reachy_mini.media.stop_playing, daemon=True
681
- )
682
- t.start()
683
- t.join(2.0)
684
- if t.is_alive():
685
- logger.warning(
686
- "stop_playing() did not complete within 2s — abandoning"
687
- )
688
- except Exception:
689
- pass
690
-
691
- @staticmethod
692
- def _preload_wav(wav_path: Path, target_sr: int | None = None) -> tuple[np.ndarray, int] | None:
693
- """Read a WAV file, returning (data, sample_rate) or None.
694
-
695
- If target_sr is given and differs from the file's rate, resample.
696
- """
697
- if sf is None:
698
- return None
699
- try:
700
- data, sr = sf.read(str(wav_path), dtype="float32")
701
- except Exception as exc:
702
- logger.warning("Failed to read %s: %s", wav_path, exc)
703
- return None
704
- if data.ndim == 2:
705
- data = data.mean(axis=1)
706
- if target_sr and sr != target_sr:
707
- try:
708
- from scipy.signal import resample
709
-
710
- num_samples = int(len(data) * target_sr / sr)
711
- data = resample(data, num_samples).astype(np.float32)
712
- sr = target_sr
713
- except Exception as exc:
714
- logger.warning("Resample failed: %s — using original rate", exc)
715
- return data, sr
716
-
717
- @staticmethod
718
- def _play_preloaded_wav(
719
- reachy_mini: ReachyMini,
720
- wav_data: tuple[np.ndarray, int],
721
- stop_event: threading.Event,
722
- chunk_duration: float = 0.02,
723
- pipeline_ready: bool = False,
724
- start_signal: threading.Event | None = None,
725
- ) -> None:
726
- """Push preloaded WAV data through the playback pipeline."""
727
- data, sr = wav_data
728
- if not pipeline_ready:
729
- reachy_mini.media.start_playing()
730
- # If a start_signal is provided, wait until the caller says go
731
- # (e.g. after the first motion command is sent).
732
- if start_signal is not None:
733
- start_signal.wait(timeout=5.0)
734
- samples_per_chunk = max(1, int(sr * chunk_duration))
735
- play_start = time.perf_counter()
736
- try:
737
- for offset in range(0, len(data), samples_per_chunk):
738
- if stop_event.is_set():
739
- break
740
- chunk = data[offset : offset + samples_per_chunk]
741
- reachy_mini.media.push_audio_sample(chunk)
742
- stop_event.wait(chunk_duration * 0.8)
743
- else:
744
- audio_duration = len(data) / sr
745
- elapsed = time.perf_counter() - play_start
746
- remaining = audio_duration - elapsed
747
- if remaining > 0:
748
- stop_event.wait(remaining)
749
- finally:
750
- try:
751
- t = threading.Thread(
752
- target=reachy_mini.media.stop_playing, daemon=True
753
- )
754
- t.start()
755
- t.join(2.0)
756
- if t.is_alive():
757
- logger.warning(
758
- "stop_playing() did not complete within 2s — abandoning"
759
- )
760
- except Exception:
761
- pass
762
-
763
- def _run_startup_animation(
764
- self, reachy_mini: ReachyMini, stop_event: threading.Event
765
- ) -> None:
766
- """Startup greeting with configurable welcome messages.
767
-
768
- Behaviour depends on self._welcome_messages (0, 1, or 2):
769
- 0 messages — head up, head down, release torque (silent)
770
- 1 message — head up, play intro, head down, release torque
771
- 2 messages — head up, play intro, head down, pause, head semi-up,
772
- play second message, release torque (full greeting)
773
-
774
- Uses stop_event.wait() instead of time.sleep() so the startup
775
- can be interrupted cleanly if the app is shutting down.
776
- """
777
- self._set_state(mode="starting_up", message="Starting up…", active_move=None)
778
- assets = Path(__file__).parent / "assets"
779
- n_messages = self._welcome_messages
780
- _anim_t0 = time.perf_counter()
781
-
782
- # Step 1: Head up (all modes)
783
- self._safe_enable_motors(reachy_mini)
784
- print(f"[BOOT] anim: enable_motors done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
785
- self._goto_pose_scaled(
786
- reachy_mini,
787
- INIT_HEAD_POSE,
788
- antennas=[0.0, 0.0],
789
- min_duration=0.2,
790
- )
791
- print(f"[BOOT] anim: head-up done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
792
-
793
- if stop_event.is_set():
794
- return
795
-
796
- # Step 2: Play intro sound (1 or 2 messages)
797
- if n_messages >= 1:
798
- intro_path = assets / "intro_marionette.wav"
799
- if intro_path.exists() and not stop_event.is_set():
800
- try:
801
- self._play_wav_chunked(reachy_mini, intro_path, stop_event)
802
- except Exception as exc:
803
- logger.warning("Failed to play intro sound: %s", exc)
804
- print(f"[BOOT] anim: intro sound done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
805
-
806
- if stop_event.is_set():
807
- return
808
-
809
- # Step 3: Head down + release torque (all modes)
810
- self._goto_sleep_and_release(reachy_mini)
811
-
812
- if stop_event.is_set():
813
- return
814
-
815
- # Step 4: Second message with semi-awaken pose (2 messages only)
816
- if n_messages >= 2:
817
- second_path = assets / "please.wav"
818
- if second_path.exists() and not stop_event.is_set():
819
- # Preload wav during the pause so playback starts immediately.
820
- wav_data = self._preload_wav(second_path)
821
- #stop_event.wait(0.5)
822
- if stop_event.is_set():
823
- return
824
- self._safe_enable_motors(reachy_mini)
825
- antenna_angle = np.deg2rad(15)
826
- # Start audio and head rise simultaneously.
827
- audio_stop = threading.Event()
828
- audio_thread: threading.Thread | None = None
829
- if wav_data is not None:
830
- audio_thread = threading.Thread(
831
- target=self._play_preloaded_wav,
832
- args=(reachy_mini, wav_data, audio_stop),
833
- daemon=True,
834
- )
835
- audio_thread.start()
836
- reachy_mini.goto_target(
837
- SEMI_AWAKEN_POSE,
838
- antennas=[-antenna_angle, antenna_angle],
839
- duration=0.3,
840
- )
841
- if audio_thread is not None:
842
- audio_thread.join(timeout=10.0)
843
- audio_stop.set()
844
- self._goto_sleep_and_release(reachy_mini)
845
-
846
- def _disable_mic_agc(self) -> None:
847
- """Disable the mic's automatic gain control for cleaner recordings.
848
-
849
- The XMOS XVF3800 AGC scales sensitivity to avoid saturation, but
850
- this causes the mic to "mute" when motors are loud. We disable
851
- AGC on startup and restore the original value on shutdown.
852
- """
853
- try:
854
- from reachy_mini.media.audio_control_utils import init_respeaker_usb
855
- except ImportError:
856
- logger.debug("audio_control_utils not available, skipping AGC config")
857
- return
858
-
859
- try:
860
- respeaker = init_respeaker_usb()
861
- if respeaker is None:
862
- logger.debug("No ReSpeaker USB device found, skipping AGC config")
863
- return
864
- self._original_agc = respeaker.read("PP_AGCONOFF")
865
- respeaker.write("PP_AGCONOFF", [0])
866
- respeaker.close()
867
- logger.info("Mic AGC disabled (was %s)", self._original_agc)
868
- except Exception as exc:
869
- logger.warning("Failed to disable mic AGC: %s", exc)
870
-
871
- def _restore_mic_agc(self) -> None:
872
- """Restore the mic AGC to its original value."""
873
- if not hasattr(self, "_original_agc") or self._original_agc is None:
874
- return
875
- try:
876
- from reachy_mini.media.audio_control_utils import init_respeaker_usb
877
-
878
- respeaker = init_respeaker_usb()
879
- if respeaker is None:
880
- return
881
- respeaker.write("PP_AGCONOFF", self._original_agc)
882
- respeaker.close()
883
- logger.info("Mic AGC restored to %s", self._original_agc)
884
- except Exception as exc:
885
- logger.warning("Failed to restore mic AGC: %s", exc)
886
-
887
- def run(self, reachy_mini: ReachyMini, stop_event: threading.Event) -> None:
888
- try:
889
- _run_t0 = time.perf_counter()
890
- print(f"[BOOT] run() entered at +{_run_t0 - _BOOT_T0:.2f}s after module import", flush=True)
891
- self._disable_mic_agc()
892
- print(f"[BOOT] _disable_mic_agc done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
893
- self._run_startup_animation(reachy_mini, stop_event)
894
- print(f"[BOOT] startup animation done at +{time.perf_counter() - _BOOT_T0:.2f}s", flush=True)
895
- self._set_idle_state()
896
-
897
- while not stop_event.is_set():
898
- job_type: str | None = None
899
- payload: RecordingRequest | str | None = None
900
-
901
- with self._state_lock:
902
- if self._pending_recording is not None:
903
- job_type = "record"
904
- payload = self._pending_recording
905
- self._pending_recording = None
906
- elif self._pending_playback is not None:
907
- job_type = "play"
908
- payload = self._pending_playback
909
- self._pending_playback = None
910
-
911
- if job_type == "record" and isinstance(payload, RecordingRequest):
912
- self._perform_recording(reachy_mini, stop_event, payload)
913
- self._set_idle_state()
914
- elif job_type == "play" and isinstance(payload, str):
915
- self._perform_playback(reachy_mini, payload)
916
- self._set_idle_state()
917
- self._align_head_and_release(reachy_mini)
918
- else:
919
- stop_event.wait(0.05)
920
- finally:
921
- self._restore_mic_agc()
922
- self._park_robot(reachy_mini)
923
-
924
- # ──────── Robot actions ────────────────────���────────────────────────
925
- def _perform_recording(
926
- self,
927
- reachy_mini: ReachyMini,
928
- stop_event: threading.Event,
929
- request: RecordingRequest,
930
- ) -> None:
931
- self._recording_cancel_event.clear()
932
- countdown_end = time.time() + COUNTDOWN_SECONDS
933
- self._set_state(
934
- mode="countdown",
935
- message=f"Recording {request.label} in {COUNTDOWN_SECONDS}s",
936
- active_move=request.label,
937
- countdown_ends_at=countdown_end,
938
- )
939
-
940
- while time.time() < countdown_end:
941
- if stop_event.wait(0.1) or self._recording_cancel_event.is_set():
942
- self._recording_cancel_event.clear()
943
- self._set_state(mode="idle", message="Recording cancelled", active_move=None)
944
- return
945
-
946
- with self._state_lock:
947
- self._recording_stats = None
948
-
949
- self._set_state(
950
- mode="recording",
951
- message=f"Recording {request.label}",
952
- active_move=request.move_id,
953
- recording_started_at=time.time(),
954
- recording_duration=request.duration,
955
- )
956
-
957
- try:
958
- self._run_capture_and_save(reachy_mini, stop_event, request)
959
- except Exception as exc:
960
- logger.error("Recording failed: %s", exc, exc_info=True)
961
- self._set_state(
962
- mode="error",
963
- message=f"Recording error: {exc}",
964
- active_move=None,
965
- )
966
-
967
- def _run_capture_and_save(
968
- self,
969
- reachy_mini: ReachyMini,
970
- stop_event: threading.Event,
971
- request: RecordingRequest,
972
- ) -> None:
973
- """Inner recording logic, wrapped by _perform_recording's safety net."""
974
- # For uploaded-audio sessions, preload and prime the playback pipeline,
975
- # then gate audio start on the exact capture start instant.
976
- audio_stop = threading.Event()
977
- audio_start = threading.Event()
978
- audio_thread: threading.Thread | None = None
979
- if request.uploaded_audio_path and request.uploaded_audio_path.exists():
980
- try:
981
- sr_out = int(reachy_mini.media.get_output_audio_samplerate() or 16000)
982
- except Exception:
983
- sr_out = 16000
984
- wav_data = self._preload_wav(request.uploaded_audio_path, target_sr=sr_out)
985
- if wav_data is not None:
986
- reachy_mini.media.start_playing()
987
- audio_thread = threading.Thread(
988
- target=self._play_preloaded_wav,
989
- args=(reachy_mini, wav_data, audio_stop),
990
- kwargs={"pipeline_ready": True, "start_signal": audio_start},
991
- daemon=True,
992
- )
993
- audio_thread.start()
994
-
995
- # When using uploaded audio, don't record from mic (audio comes from the file)
996
- should_record_mic = request.record_audio and not request.uploaded_audio_path
997
- try:
998
- timestamps, frames, audio_frames, audio_samplerate = self._capture_motion(
999
- reachy_mini, stop_event, request.duration, should_record_mic,
1000
- record_motion=request.record_motion,
1001
- on_capture_start=(audio_start.set if audio_thread is not None else None),
1002
- )
1003
- finally:
1004
- # Signal the audio thread to stop and wait briefly
1005
- audio_stop.set()
1006
- if audio_thread is not None:
1007
- audio_thread.join(timeout=3.0)
1008
-
1009
- # Check if recording was cancelled
1010
- was_cancelled = self._recording_cancel_event.is_set()
1011
- self._recording_cancel_event.clear()
1012
-
1013
- if not timestamps:
1014
- self._set_state(
1015
- mode="idle",
1016
- message="Recording cancelled" if was_cancelled else "No motion data captured.",
1017
- active_move=None,
1018
- )
1019
- return
1020
-
1021
- # Save the recording (even if partial due to early stop)
1022
- self._save_recording(request, timestamps, frames, audio_frames, audio_samplerate)
1023
- self._refresh_recordings()
1024
- pose_count = len(frames)
1025
- duration_elapsed = timestamps[-1] if timestamps else request.duration
1026
- duration_elapsed = max(duration_elapsed, 1e-6)
1027
- poses_per_sec = pose_count / duration_elapsed
1028
- with self._state_lock:
1029
- self._recording_stats = {
1030
- "poses": pose_count,
1031
- "duration": duration_elapsed,
1032
- "poses_per_second": poses_per_sec,
1033
- }
1034
- if not request.record_motion:
1035
- status_msg = f"Recorded audio: {request.label} ({duration_elapsed:.1f}s)"
1036
- if was_cancelled:
1037
- status_msg = f"Saved audio: {request.label} (stopped early, {duration_elapsed:.1f}s)"
1038
- else:
1039
- status_msg = f"Recorded {request.label} • {pose_count} poses ({poses_per_sec:.1f}/s)"
1040
- if was_cancelled:
1041
- status_msg = f"Saved {request.label} (stopped early) • {pose_count} poses"
1042
- self._set_state(
1043
- mode="idle",
1044
- message=status_msg,
1045
- active_move=None,
1046
- )
1047
- if not was_cancelled:
1048
- self._preferred_duration = request.duration
1049
-
1050
- def _perform_playback(self, reachy_mini: ReachyMini, move_id: str) -> None:
1051
- meta = self._recordings.get(move_id)
1052
- if not meta:
1053
- self._set_state(mode="error", message=f"Move {move_id} missing.", active_move=None)
1054
- return
1055
-
1056
- try:
1057
- move = self._load_move(meta.json_path)
1058
- except Exception as exc: # pragma: no cover - filesystem failure
1059
- self._set_state(
1060
- mode="error",
1061
- message=f"Failed to load {move_id}: {exc}",
1062
- active_move=None,
1063
- )
1064
- return
1065
-
1066
- # Audio-only moves: just play the audio, no motion replay
1067
- if meta.audio_only:
1068
- self._playback_cancel_event.clear()
1069
- self._set_state(
1070
- mode="playing",
1071
- message=f"Playing audio: {meta.label}",
1072
- active_move=meta.move_id,
1073
- )
1074
- if move.sound_path is not None:
1075
- try:
1076
- self._play_wav_chunked(
1077
- reachy_mini, move.sound_path, self._playback_cancel_event,
1078
- )
1079
- except Exception:
1080
- pass
1081
- cancelled = self._playback_cancel_event.is_set()
1082
- self._playback_cancel_event.clear()
1083
- msg = f"Playback stopped for {meta.label}" if cancelled else f"Finished playing {meta.label}"
1084
- self._set_state(mode="idle", message=msg, active_move=None)
1085
- return
1086
-
1087
- move_to_play = self._apply_motion_model(move)
1088
-
1089
- self._playback_cancel_event.clear()
1090
- self._set_state(
1091
- mode="playing",
1092
- message=f"Playing {meta.label}",
1093
- active_move=meta.move_id,
1094
- )
1095
-
1096
- self._safe_enable_motors(reachy_mini)
1097
- cancelled = False
1098
-
1099
- # Preload audio (including resample) while we go to the start pose,
1100
- # so playback can begin instantly when motion starts.
1101
- try:
1102
- sr_out = int(reachy_mini.media.get_output_audio_samplerate() or 16000)
1103
- except Exception:
1104
- sr_out = 16000
1105
- wav_data: tuple[np.ndarray, int] | None = None
1106
- if move_to_play.sound_path is not None:
1107
- wav_data = self._preload_wav(move_to_play.sound_path, target_sr=sr_out)
1108
-
1109
- # Move to the start pose BEFORE starting audio so they begin in sync.
1110
- try:
1111
- start_head_pose, start_antennas, start_body_yaw = move_to_play.evaluate(0.0)
1112
- except Exception:
1113
- self._set_state(mode="error", message="Failed to evaluate move start pose", active_move=None)
1114
- return
1115
- self._goto_pose_scaled(
1116
- reachy_mini,
1117
- start_head_pose,
1118
- antennas=list(start_antennas) if start_antennas is not None else None,
1119
- min_duration=0.2,
1120
- )
1121
- if start_body_yaw is not None:
1122
- reachy_mini.set_target_body_yaw(float(start_body_yaw))
1123
-
1124
- # Warm the audio pipeline, then let the audio thread wait for the
1125
- # motion loop to send its first command before pushing audio.
1126
- audio_stop = threading.Event()
1127
- audio_start = threading.Event()
1128
- audio_thread: threading.Thread | None = None
1129
- if wav_data is not None:
1130
- reachy_mini.media.start_playing()
1131
- audio_thread = threading.Thread(
1132
- target=self._play_preloaded_wav,
1133
- args=(reachy_mini, wav_data, audio_stop),
1134
- kwargs={"pipeline_ready": True, "start_signal": audio_start},
1135
- daemon=True,
1136
- )
1137
- audio_thread.start()
1138
- try:
1139
- cancelled = not self._stream_playback(
1140
- reachy_mini, move_to_play, start_signal=audio_start,
1141
- )
1142
- finally:
1143
- self._playback_cancel_event.clear()
1144
- audio_stop.set()
1145
- if audio_thread is not None:
1146
- audio_thread.join(timeout=3.0)
1147
-
1148
- if cancelled:
1149
- self._set_state(
1150
- mode="idle",
1151
- message=f"Playback stopped for {meta.label}",
1152
- active_move=None,
1153
- )
1154
- else:
1155
- self._set_state(
1156
- mode="idle",
1157
- message=f"Finished playing {meta.label}",
1158
- active_move=None,
1159
- )
1160
-
1161
- @staticmethod
1162
- def _call_with_timeout(fn: Callable, timeout: float = 2.0, retries: int = 2) -> bool:
1163
- """Call fn() in a daemon thread with timeout and retries.
1164
-
1165
- GStreamer's set_state() can hang when the audio device is slow or the
1166
- GLib MainLoop is starved. We run the call in a disposable daemon thread
1167
- and retry up to `retries` times — often the second attempt succeeds
1168
- because the MainLoop has had time to process pending bus messages.
1169
-
1170
- Returns True if the call completed within the timeout, False if all
1171
- attempts timed out (the last thread is abandoned as a daemon).
1172
- """
1173
- for attempt in range(1, retries + 1):
1174
- t = threading.Thread(target=fn, daemon=True)
1175
- t.start()
1176
- t.join(timeout)
1177
- if not t.is_alive():
1178
- return True
1179
- logger.warning(
1180
- "%s attempt %d/%d did not complete within %.1fs",
1181
- fn, attempt, retries, timeout,
1182
- )
1183
- logger.error("%s hung after %d attempts — abandoning", fn, retries)
1184
- return False
1185
-
1186
- # ──────── Capture helpers ───────────────────────────────────────────
1187
- def _capture_motion(
1188
- self,
1189
- reachy_mini: ReachyMini,
1190
- stop_event: threading.Event,
1191
- duration: float,
1192
- record_audio: bool,
1193
- record_motion: bool = True,
1194
- on_capture_start: Callable[[], None] | None = None,
1195
- ) -> tuple[list[float], list[dict[str, Any]], list[np.ndarray], int | None]:
1196
- timestamps: list[float] = []
1197
- frames: list[dict[str, Any]] = []
1198
- audio_frames: list[np.ndarray] = []
1199
- audio_samplerate: int | None = None
1200
- audio_active = False
1201
-
1202
- def _pull_audio_frames(max_seconds: float = 1.0) -> None:
1203
- if not audio_active:
1204
- return
1205
- deadline = time.perf_counter() + max_seconds
1206
- while time.perf_counter() < deadline:
1207
- sample = reachy_mini.media.get_audio_sample()
1208
- if sample is None:
1209
- break
1210
- audio_frames.append(sample)
1211
-
1212
- if record_audio and self._audio_available:
1213
- try:
1214
- reachy_mini.media.start_recording()
1215
- audio_active = True
1216
- reported_rate = reachy_mini.media.get_input_audio_samplerate()
1217
- audio_samplerate = int(reported_rate) if reported_rate else AUDIO_SAMPLE_RATE
1218
- except Exception as exc: # pragma: no cover - runtime audio failure
1219
- self._set_state(
1220
- mode="error",
1221
- message=f"Audio init failed: {exc}",
1222
- active_move=None,
1223
- )
1224
- record_audio = False
1225
- audio_active = False
1226
-
1227
- start = time.perf_counter()
1228
- # Update recording_started_at to the actual capture start (after audio init).
1229
- # Without this, the frontend progress bar is ahead by the audio init delay (~0.5s).
1230
- self._set_state(recording_started_at=time.time())
1231
- if on_capture_start is not None:
1232
- try:
1233
- on_capture_start()
1234
- except Exception as exc:
1235
- logger.warning("Capture start hook failed: %s", exc)
1236
-
1237
- # Hard time guard: if an SDK call hangs mid-iteration, bail out
1238
- # after duration + generous margin rather than blocking forever.
1239
- hard_deadline = start + duration + 10.0
1240
-
1241
- sample_period = 1.0 / MOTION_SAMPLE_RATE
1242
- try:
1243
- while not stop_event.is_set() and not self._recording_cancel_event.is_set():
1244
- elapsed = time.perf_counter() - start
1245
- if elapsed >= duration:
1246
- break
1247
- if time.perf_counter() > hard_deadline:
1248
- logger.error("Capture loop exceeded hard deadline — aborting")
1249
- break
1250
- if record_motion:
1251
- try:
1252
- head_pose = reachy_mini.get_current_head_pose()
1253
- head_joints, antennas = reachy_mini.get_current_joint_positions()
1254
- except Exception as exc:
1255
- logger.warning("SDK call failed during capture: %s", exc)
1256
- break
1257
- timestamps.append(elapsed)
1258
- frames.append(
1259
- {
1260
- "head": np.asarray(head_pose, dtype=float).tolist(),
1261
- "antennas": np.asarray(antennas, dtype=float).tolist(),
1262
- "body_yaw": float(head_joints[0]) if head_joints else 0.0,
1263
- "check_collision": False,
1264
- }
1265
- )
1266
- else:
1267
- timestamps.append(elapsed)
1268
- stop_event.wait(sample_period)
1269
- _pull_audio_frames()
1270
- finally:
1271
- if audio_active:
1272
- stopped_ok = False
1273
- try:
1274
- stopped_ok = self._call_with_timeout(
1275
- reachy_mini.media.stop_recording, timeout=3.0
1276
- )
1277
- except Exception as exc:
1278
- logger.warning("stop_recording() failed: %s", exc)
1279
- if stopped_ok:
1280
- _pull_audio_frames()
1281
- else:
1282
- logger.warning(
1283
- "Skipping audio drain — stop_recording did not complete"
1284
- )
1285
- return timestamps, frames, audio_frames, audio_samplerate
1286
-
1287
- def _stream_playback(
1288
- self,
1289
- reachy_mini: ReachyMini,
1290
- move: RecordedMove,
1291
- sample_hook: Callable[[], None] | None = None,
1292
- start_signal: threading.Event | None = None,
1293
- ) -> bool:
1294
- playback_freq = 100.0
1295
- sleep_period = 1.0 / playback_freq
1296
- t0 = time.perf_counter()
1297
- signalled = False
1298
- while True:
1299
- if self._playback_cancel_event.is_set():
1300
- return False
1301
- elapsed = time.perf_counter() - t0
1302
- if elapsed >= move.duration:
1303
- break
1304
- t = min(max(elapsed, 0.0), max(move.duration - 1e-3, 0.0))
1305
- head, antennas, body_yaw = move.evaluate(t)
1306
- reachy_mini.set_target_head_pose(head)
1307
- if body_yaw is not None:
1308
- reachy_mini.set_target_body_yaw(float(body_yaw))
1309
- if antennas is not None:
1310
- reachy_mini.set_target_antenna_joint_positions(list(antennas))
1311
- if not signalled and start_signal is not None:
1312
- start_signal.set()
1313
- signalled = True
1314
- if sample_hook is not None:
1315
- sample_hook()
1316
-
1317
- remaining = move.duration - elapsed
1318
- wait_time = max(0.001, min(sleep_period, remaining))
1319
- if self._playback_cancel_event.wait(wait_time):
1320
- return False
1321
-
1322
- return True
1323
-
1324
- def _apply_motion_model(self, move: RecordedMove) -> RecordedMove:
1325
- try:
1326
- return self._motion_model_registry.apply(move)
1327
- except Exception as exc:
1328
- logger.warning("Motion model %s failed: %s", self._motion_model_registry.active, exc)
1329
- return move
1330
-
1331
- def _save_recording(
1332
- self,
1333
- request: RecordingRequest,
1334
- timestamps: list[float],
1335
- frames: list[dict[str, Any]],
1336
- audio_frames: list[np.ndarray],
1337
- audio_samplerate: int | None,
1338
- ) -> None:
1339
- json_path = self._dataset_dir / f"{request.move_id}.json"
1340
- wav_path = json_path.with_suffix(".wav")
1341
-
1342
- data: dict[str, Any] = {
1343
- "description": request.description,
1344
- "time": timestamps,
1345
- "set_target_data": frames,
1346
- }
1347
- if not request.record_motion:
1348
- data["audio_only"] = True
1349
-
1350
- json_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
1351
-
1352
- # Handle audio: either copy uploaded file or save mic recording
1353
- if request.uploaded_audio_path and request.uploaded_audio_path.exists():
1354
- # Copy uploaded audio to move's wav path.
1355
- # Keep the temp file so the user can record again with the same upload.
1356
- shutil.copy2(str(request.uploaded_audio_path), str(wav_path))
1357
- elif request.record_audio and audio_frames and self._audio_available and audio_samplerate:
1358
- audio_data = np.concatenate(audio_frames, axis=0)
1359
- sf.write(str(wav_path), audio_data, audio_samplerate) # type: ignore[arg-type]
1360
-
1361
- def _load_move(self, json_path: Path, *, prefer_denoised: bool = True) -> RecordedMove:
1362
- move = json.loads(json_path.read_text(encoding="utf-8"))
1363
- sound_path = json_path.with_suffix(".wav")
1364
- if prefer_denoised:
1365
- denoised = json_path.with_suffix(DENOISED_SUFFIX)
1366
- if denoised.exists():
1367
- sound_path = denoised
1368
- return RecordedMove(move, sound_path if sound_path.exists() else None)
1369
-
1370
- def _build_recording_request(
1371
- self, payload: StartRecordingPayload, uploaded_audio_path: Path | None = None
1372
- ) -> RecordingRequest:
1373
- stamp = datetime.utcnow().strftime("%Y%m%d-%H%M%S")
1374
- base_name = payload.label or f"take-{stamp}"
1375
- label = payload.label or f"Take {stamp}"
1376
- slug = _slugify(base_name)
1377
- move_id = slug
1378
- index = 1
1379
- while (self._dataset_dir / f"{move_id}.json").exists():
1380
- move_id = f"{slug}-{index}"
1381
- index += 1
1382
-
1383
- description = payload.description or f"Recorded with Marionette on {stamp}"
1384
-
1385
- return RecordingRequest(
1386
- move_id=move_id,
1387
- label=label,
1388
- description=description,
1389
- duration=float(payload.duration),
1390
- record_audio=bool(payload.record_audio),
1391
- record_motion=bool(payload.record_motion),
1392
- uploaded_audio_path=uploaded_audio_path,
1393
- )
1394
-
1395
- def _refresh_recordings(self) -> None:
1396
- recordings: dict[str, RecordingMetadata] = {}
1397
- # Get uploaded move IDs for the active dataset
1398
- active_entry = self._datasets.get(self._active_dataset_id or "")
1399
- uploaded_ids = active_entry.uploaded_move_ids if active_entry else set()
1400
-
1401
- for json_path in sorted(self._dataset_dir.glob("*.json")):
1402
- try:
1403
- move = json.loads(json_path.read_text(encoding="utf-8"))
1404
- timestamps = move.get("time", [])
1405
- duration = (
1406
- float(timestamps[-1]) - float(timestamps[0])
1407
- if len(timestamps) >= 2
1408
- else float(timestamps[0]) if timestamps else 0.0
1409
- )
1410
- description = move.get("description", "")
1411
- original_audio = json_path.with_suffix(".wav")
1412
- denoised_audio = json_path.with_suffix(DENOISED_SUFFIX)
1413
- noise_profile = json_path.with_suffix(NOISE_SUFFIX)
1414
- has_denoised = denoised_audio.exists()
1415
- has_audio = has_denoised or original_audio.exists()
1416
- move_id = json_path.stem
1417
- recordings[move_id] = RecordingMetadata(
1418
- move_id=move_id,
1419
- label=move_id,
1420
- description=description,
1421
- duration=duration,
1422
- created_at=json_path.stat().st_mtime,
1423
- json_path=json_path,
1424
- has_audio=has_audio,
1425
- has_noise_profile=noise_profile.exists(),
1426
- has_denoised_audio=has_denoised,
1427
- is_uploaded=move_id in uploaded_ids,
1428
- audio_only=bool(move.get("audio_only", False)),
1429
- )
1430
- except Exception:
1431
- continue
1432
-
1433
- with self._state_lock:
1434
- self._recordings = recordings
1435
-
1436
- # ──────── Dataset management ───────────────────────────────────────
1437
- def _load_dataset_registry(self) -> None:
1438
- if self._registry_path.exists():
1439
- try:
1440
- raw = json.loads(self._registry_path.read_text(encoding="utf-8"))
1441
- except Exception:
1442
- raw = {}
1443
- else:
1444
- raw = {}
1445
-
1446
- raw_params = raw.get("motion_model_params", {})
1447
- if isinstance(raw_params, dict):
1448
- for model_name, params in raw_params.items():
1449
- if isinstance(params, dict):
1450
- try:
1451
- self._motion_model_registry.set_model_params(model_name, params)
1452
- except KeyError:
1453
- continue
1454
- self._preferred_duration = float(raw.get("preferred_duration", DEFAULT_DURATION))
1455
- self._welcome_messages = int(raw.get("welcome_messages", 2))
1456
-
1457
- dataset_entries = raw.get("datasets", [])
1458
- metadata_by_folder: dict[str, dict[str, Any]] = {}
1459
- fallback_root: Path | None = None
1460
- for entry in dataset_entries:
1461
- folder = entry.get("folder")
1462
- path_str = entry.get("path")
1463
- if folder:
1464
- metadata_by_folder[folder] = entry
1465
- if not fallback_root and path_str:
1466
- resolved = self._resolve_dataset_path(path_str)
1467
- fallback_root = resolved.parent
1468
- elif path_str and not fallback_root:
1469
- resolved = self._resolve_dataset_path(path_str)
1470
- fallback_root = resolved.parent
1471
-
1472
- raw_root = raw.get("root_path")
1473
- if self._dataset_root_override is not None:
1474
- root_path = self._dataset_root_override
1475
- elif raw_root:
1476
- root_path = self._resolve_dataset_path(raw_root)
1477
- elif fallback_root:
1478
- root_path = fallback_root
1479
- else:
1480
- root_path = self._default_dataset_root()
1481
- self._dataset_root = root_path
1482
- self._dataset_root.mkdir(parents=True, exist_ok=True)
1483
- logger.info("Using dataset root at %s", self._dataset_root)
1484
-
1485
- datasets: dict[str, DatasetEntry] = {}
1486
- used_ids: set[str] = set()
1487
- for folder_path in sorted(self._dataset_root.iterdir()):
1488
- if not folder_path.is_dir():
1489
- continue
1490
- folder = folder_path.name
1491
- meta = metadata_by_folder.get(folder, {})
1492
- dataset_id = meta.get("id")
1493
- if not dataset_id or dataset_id in used_ids:
1494
- dataset_id = self._next_dataset_id(folder, used_ids)
1495
- else:
1496
- used_ids.add(dataset_id)
1497
- label = meta.get("label") or folder
1498
- uploaded_ids = set(meta.get("uploaded_move_ids") or [])
1499
- origin = meta.get("origin", "local")
1500
- entry = self._entry_from_folder(folder, label, dataset_id, uploaded_ids, origin=origin)
1501
- datasets[dataset_id] = entry
1502
-
1503
- if not datasets:
1504
- default_id = "default" if "default" not in used_ids else self._next_dataset_id(DATASET_DIRNAME, used_ids)
1505
- entry = self._entry_from_folder(DATASET_DIRNAME, DEFAULT_DATASET_LABEL, default_id)
1506
- datasets[entry.dataset_id] = entry
1507
-
1508
- self._datasets = datasets
1509
- active_id = raw.get("active")
1510
- if active_id not in self._datasets:
1511
- active_id = next(iter(self._datasets))
1512
- self._active_dataset_id = active_id
1513
- self._dataset_dir = self._ensure_dataset_data_dir(self._datasets[active_id])
1514
- self._save_dataset_registry()
1515
-
1516
- def _save_dataset_registry(self) -> None:
1517
- data = {
1518
- "active": self._active_dataset_id,
1519
- "root_path": str(self._dataset_root),
1520
- "motion_model_params": {
1521
- "lead_compensation": self._motion_model_registry.get_model_params("lead_compensation"),
1522
- },
1523
- "preferred_duration": self._preferred_duration,
1524
- "welcome_messages": self._welcome_messages,
1525
- "datasets": [
1526
- {
1527
- "id": entry.dataset_id,
1528
- "label": entry.label,
1529
- "folder": entry.folder,
1530
- "path": str(entry.path),
1531
- "uploaded_move_ids": list(entry.uploaded_move_ids or []),
1532
- "origin": entry.origin,
1533
- }
1534
- for entry in self._datasets.values()
1535
- ],
1536
- }
1537
- self._registry_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
1538
-
1539
- def _resolve_dataset_path(self, raw_path: str | Path) -> Path:
1540
- path = Path(raw_path).expanduser()
1541
- try:
1542
- path = path.resolve(strict=False)
1543
- except Exception:
1544
- path = path.expanduser().absolute()
1545
- return path
1546
-
1547
- def _next_dataset_id(self, label: str, used: set[str]) -> str:
1548
- base = _slugify(label or "dataset")
1549
- candidate = base or "dataset"
1550
- suffix = 1
1551
- while candidate in used:
1552
- candidate = f"{base}-{suffix}"
1553
- suffix += 1
1554
- used.add(candidate)
1555
- return candidate
1556
-
1557
- def _entry_from_folder(
1558
- self,
1559
- folder: str,
1560
- label: str,
1561
- dataset_id: str,
1562
- uploaded_move_ids: set[str] | None = None,
1563
- origin: str = "local",
1564
- ) -> DatasetEntry:
1565
- path = self._dataset_root / folder
1566
- entry = DatasetEntry(
1567
- dataset_id=dataset_id,
1568
- label=label,
1569
- folder=folder,
1570
- path=path,
1571
- uploaded_move_ids=uploaded_move_ids or set(),
1572
- origin=origin,
1573
- )
1574
- self._ensure_dataset_data_dir(entry)
1575
- return entry
1576
-
1577
- def _default_dataset_root(self) -> Path:
1578
- system = platform.system().lower()
1579
- home = Path.home()
1580
- if "windows" in system:
1581
- base = home / "Documents" / "ReachyMini" / "datasets"
1582
- elif "darwin" in system:
1583
- base = home / "Library" / "Application Support" / "ReachyMini" / "datasets"
1584
- else:
1585
- base = home / "reachy_mini_datasets"
1586
- return base
1587
-
1588
- def _ensure_dataset_data_dir(self, entry: DatasetEntry) -> Path:
1589
- """Ensure the dataset layout (root + data subdir) exists and migrate legacy files."""
1590
- entry.path.mkdir(parents=True, exist_ok=True)
1591
- data_dir = entry.path / DATASET_DATA_SUBDIR
1592
- data_dir.mkdir(parents=True, exist_ok=True)
1593
- legacy_moves = list(entry.path.glob("*.json"))
1594
- for json_path in legacy_moves:
1595
- target = data_dir / json_path.name
1596
- if not target.exists():
1597
- shutil.move(str(json_path), str(target))
1598
- for suffix in (".wav", NOISE_SUFFIX, DENOISED_SUFFIX):
1599
- source = json_path.with_suffix(suffix)
1600
- if source.exists():
1601
- shutil.move(str(source), str(data_dir / source.name))
1602
- return data_dir
1603
-
1604
- def _create_dataset_internal(
1605
- self,
1606
- folder: str,
1607
- label: str,
1608
- *,
1609
- dataset_id: str | None = None,
1610
- save: bool = True,
1611
- origin: str = "local",
1612
- ) -> DatasetEntry:
1613
- folder_slug = _slugify(folder or label or "dataset")
1614
- if any(entry.folder == folder_slug for entry in self._datasets.values()):
1615
- if dataset_id is None:
1616
- raise HTTPException(status_code=409, detail=f"Dataset folder '{folder_slug}' already exists.")
1617
- path = self._dataset_root / folder_slug
1618
- path.mkdir(parents=True, exist_ok=True)
1619
-
1620
- if dataset_id is None:
1621
- base = _slugify(label or folder_slug or "dataset")
1622
- dataset_id = base or "dataset"
1623
- suffix = 1
1624
- while dataset_id in self._datasets:
1625
- dataset_id = f"{base}-{suffix}"
1626
- suffix += 1
1627
-
1628
- entry = DatasetEntry(
1629
- dataset_id=dataset_id, label=label or dataset_id, folder=folder_slug, path=path, origin=origin
1630
- )
1631
- self._datasets[entry.dataset_id] = entry
1632
- self._ensure_dataset_data_dir(entry)
1633
- if save:
1634
- self._save_dataset_registry()
1635
- return entry
1636
-
1637
- def _create_dataset(self, name: str, label: str | None) -> DatasetEntry:
1638
- if not name:
1639
- raise HTTPException(status_code=400, detail="Dataset name is required.")
1640
- folder = _slugify(name)
1641
- if not folder:
1642
- raise HTTPException(status_code=400, detail="Dataset name is invalid.")
1643
- if any(entry.folder == folder for entry in self._datasets.values()):
1644
- raise HTTPException(status_code=409, detail="Dataset name already used.")
1645
- label_to_use = label or name or DEFAULT_DATASET_LABEL
1646
- entry = self._create_dataset_internal(folder, label_to_use)
1647
- return entry
1648
-
1649
- def _select_dataset(self, dataset_id: str) -> None:
1650
- entry = self._datasets.get(dataset_id)
1651
- if entry is None:
1652
- raise HTTPException(status_code=404, detail=f"Dataset {dataset_id} not found.")
1653
- self._active_dataset_id = dataset_id
1654
- self._dataset_dir = self._ensure_dataset_data_dir(entry)
1655
- self._save_dataset_registry()
1656
-
1657
- def _datasets_payload(self) -> dict[str, Any]:
1658
- entries = [entry.to_payload() for entry in self._datasets.values()]
1659
- return {
1660
- "active_id": self._active_dataset_id,
1661
- "root_path": str(self._dataset_root),
1662
- "entries": entries,
1663
- }
1664
-
1665
- def _set_dataset_root(self, raw_path: str) -> None:
1666
- new_root = self._resolve_dataset_path(raw_path)
1667
- new_root.mkdir(parents=True, exist_ok=True)
1668
- self._dataset_root = new_root
1669
- logger.info("Dataset root updated to %s", self._dataset_root)
1670
- for entry in self._datasets.values():
1671
- entry.path = self._dataset_root / entry.folder
1672
- self._ensure_dataset_data_dir(entry)
1673
- if self._active_dataset_id not in self._datasets and self._datasets:
1674
- self._active_dataset_id = next(iter(self._datasets))
1675
- if self._active_dataset_id:
1676
- self._dataset_dir = self._ensure_dataset_data_dir(self._datasets[self._active_dataset_id])
1677
- else:
1678
- default = self._create_dataset_internal(DATASET_DIRNAME, DEFAULT_DATASET_LABEL)
1679
- self._active_dataset_id = default.dataset_id
1680
- self._dataset_dir = self._ensure_dataset_data_dir(default)
1681
- self._save_dataset_registry()
1682
-
1683
- def _check_hf_login(self) -> str | None:
1684
- """Return the logged-in Hugging Face username, or None.
1685
-
1686
- Uses a two-tier check:
1687
- 1. Local: hf_get_token() reads ~/.cache/huggingface/token (no network).
1688
- If no token exists, we're definitely not logged in.
1689
- 2. Network: hf_whoami() fetches the username from the HF API.
1690
- If this fails (timeout, rate limit, network down), we still report
1691
- as logged in (token exists) — just without a username.
1692
-
1693
- Results are cached so we don't hit the API on every poll.
1694
- """
1695
- if self._hf_checked:
1696
- return self._hf_username
1697
- self._hf_checked = True
1698
- # Tier 1: local token check (no network)
1699
- if hf_get_token is None or not hf_get_token():
1700
- self._hf_username = None
1701
- return None
1702
- # Tier 2: try to get the username (network call)
1703
- if hf_whoami is not None:
1704
- try:
1705
- info = hf_whoami()
1706
- self._hf_username = info.get("name") if isinstance(info, dict) else None
1707
- except Exception:
1708
- # Network failed but token exists — report as logged in
1709
- self._hf_username = "(token saved)"
1710
- else:
1711
- self._hf_username = "(token saved)"
1712
- return self._hf_username
1713
-
1714
- def _ensure_hf_backend(self, purpose: str) -> None:
1715
- global HfApi, DatasetFilter, snapshot_download
1716
- if HfApi is not None and snapshot_download is not None:
1717
- return
1718
- try:
1719
- from huggingface_hub import HfApi as _HfApi, snapshot_download as _snapshot_download
1720
- try:
1721
- from huggingface_hub import DatasetFilter as _DatasetFilter
1722
- except Exception:
1723
- _DatasetFilter = DatasetFilter # keep existing (possibly None)
1724
- except Exception as exc: # pragma: no cover - optional dependency
1725
- logger.warning("huggingface_hub unavailable for %s: %s", purpose, exc)
1726
- raise HTTPException(
1727
- status_code=500,
1728
- detail=f"huggingface_hub is not installed. Install it to {purpose}",
1729
- ) from exc
1730
- DatasetFilter = _DatasetFilter
1731
- HfApi = _HfApi
1732
- snapshot_download = _snapshot_download
1733
-
1734
- def _sync_dataset(self, hf_username: str, move_ids: list[str], dataset_slug: str | None) -> dict[str, Any]:
1735
- self._ensure_hf_backend("publish datasets.")
1736
- if not move_ids:
1737
- raise HTTPException(status_code=400, detail="No moves selected for synchronization.")
1738
-
1739
- self._refresh_recordings()
1740
- missing = [move_id for move_id in move_ids if move_id not in self._recordings]
1741
- if missing:
1742
- raise HTTPException(
1743
- status_code=404,
1744
- detail=f"Moves not found in active dataset: {', '.join(missing)}",
1745
- )
1746
-
1747
- entry = self._datasets.get(self._active_dataset_id or "")
1748
- if entry is None:
1749
- raise HTTPException(status_code=404, detail="No active dataset.")
1750
-
1751
- repo_basename = dataset_slug or _slugify(entry.label or entry.dataset_id)
1752
- repo_id = f"{hf_username}/{repo_basename}"
1753
- exported_bytes = 0
1754
- selected_meta = [self._recordings[move_id] for move_id in move_ids]
1755
-
1756
- with TemporaryDirectory(prefix="marionette-sync-") as tmpdir:
1757
- export_root = Path(tmpdir)
1758
- data_dir = export_root / "data"
1759
- data_dir.mkdir(parents=True, exist_ok=True)
1760
- for meta in selected_meta:
1761
- src_json = meta.json_path
1762
- dest_json = data_dir / src_json.name
1763
- shutil.copy2(src_json, dest_json)
1764
- exported_bytes += dest_json.stat().st_size
1765
- wav_path = src_json.with_suffix(".wav")
1766
- if meta.has_audio and wav_path.exists():
1767
- dest_wav = data_dir / wav_path.name
1768
- shutil.copy2(wav_path, dest_wav)
1769
- exported_bytes += dest_wav.stat().st_size
1770
-
1771
- readme_path = export_root / "README.md"
1772
- readme_path.write_text(
1773
- self._build_hf_readme(entry, selected_meta, repo_id, exported_bytes),
1774
- encoding="utf-8",
1775
- )
1776
-
1777
- api = HfApi()
1778
- try:
1779
- api.create_repo(
1780
- repo_id=repo_id,
1781
- repo_type="dataset",
1782
- exist_ok=True,
1783
- private=False,
1784
- )
1785
- api.upload_folder(
1786
- folder_path=str(export_root),
1787
- repo_id=repo_id,
1788
- repo_type="dataset",
1789
- )
1790
- except Exception as exc: # pragma: no cover - network failure
1791
- raise HTTPException(status_code=502, detail=f"Failed to upload dataset: {exc}") from exc
1792
-
1793
- # Mark moves as uploaded
1794
- if entry.uploaded_move_ids is None:
1795
- entry.uploaded_move_ids = set()
1796
- for meta in selected_meta:
1797
- entry.uploaded_move_ids.add(meta.move_id)
1798
- self._save_dataset_registry()
1799
- self._refresh_recordings()
1800
-
1801
- url = f"https://huggingface.co/datasets/{repo_id}"
1802
- return {
1803
- "status": "synced",
1804
- "repo_id": repo_id,
1805
- "uploaded_moves": len(selected_meta),
1806
- "url": url,
1807
- }
1808
-
1809
- def _download_community_dataset(self, payload: DownloadDatasetPayload) -> DatasetEntry:
1810
- self._ensure_hf_backend("download datasets.")
1811
- repo_id = payload.repo_id.strip()
1812
- logger.info("Downloading community dataset %s", repo_id)
1813
- if "/" not in repo_id:
1814
- raise HTTPException(status_code=400, detail="repo_id must include the username (e.g. user/name).")
1815
- owner, repo_name = repo_id.split("/", 1)
1816
- folder_base = payload.name or f"{owner}-{repo_name}"
1817
- folder = _slugify(folder_base)
1818
- if not folder:
1819
- folder = _slugify(repo_id.replace("/", "-"))
1820
- if any(entry.folder == folder for entry in self._datasets.values()):
1821
- raise HTTPException(status_code=409, detail=f"Dataset '{folder}' already exists locally. Delete it first to re-download.")
1822
- target_path = self._dataset_root / folder
1823
- target_path.mkdir(parents=True, exist_ok=True)
1824
-
1825
- try:
1826
- snapshot_download(
1827
- repo_id=repo_id,
1828
- repo_type="dataset",
1829
- local_dir=str(target_path),
1830
- )
1831
- except Exception as exc: # pragma: no cover - network failure
1832
- logger.exception("Failed to download dataset %s", repo_id)
1833
- raise HTTPException(status_code=502, detail=f"Failed to download dataset: {exc}") from exc
1834
-
1835
- label = payload.label or repo_id
1836
- entry = self._create_dataset_internal(folder=folder, label=label, origin="downloaded")
1837
- self._select_dataset(entry.dataset_id)
1838
- logger.info("Dataset %s downloaded into %s", repo_id, entry.path)
1839
- return entry
1840
-
1841
- def _list_community_datasets(self) -> list[dict[str, Any]]:
1842
- datasets: list[Any] = self._fetch_community_datasets_http()
1843
- if not datasets and HfApi is not None:
1844
- dataset_filter = None
1845
- if DatasetFilter is not None:
1846
- try:
1847
- dataset_filter = DatasetFilter(tags=[COMMUNITY_DATASET_TAG])
1848
- except Exception: # pragma: no cover - incompatible hub version
1849
- logger.debug("DatasetFilter unavailable, falling back to text search.")
1850
- dataset_filter = None
1851
- search_query = None if dataset_filter else COMMUNITY_DATASET_TAG
1852
- logger.info("Listing community datasets via HfApi for tag %s", COMMUNITY_DATASET_TAG)
1853
- try:
1854
- api = HfApi()
1855
- datasets = api.list_datasets(
1856
- filter=dataset_filter,
1857
- search=search_query,
1858
- limit=MAX_COMMUNITY_DATASETS,
1859
- full=True,
1860
- )
1861
- except Exception as exc: # pragma: no cover - network failure
1862
- logger.warning("HfApi dataset listing failed: %s", exc)
1863
- datasets = []
1864
- if not datasets:
1865
- raise HTTPException(status_code=502, detail="Unable to list community datasets from Hugging Face.")
1866
-
1867
- filtered_items: list[Any] = []
1868
- for item in datasets:
1869
- tags = getattr(item, "tags", None) or item.get("tags") if isinstance(item, dict) else []
1870
- if tags and COMMUNITY_DATASET_TAG in tags:
1871
- filtered_items.append(item)
1872
- if not filtered_items:
1873
- filtered_items = datasets
1874
-
1875
- results = []
1876
- for item in filtered_items:
1877
- repo_id = (
1878
- getattr(item, "id", None)
1879
- or getattr(item, "repo_id", None)
1880
- or (item.get("id") if isinstance(item, dict) else None)
1881
- or (item.get("repo_id") if isinstance(item, dict) else None)
1882
- )
1883
- if not repo_id:
1884
- continue
1885
- card = (
1886
- getattr(item, "cardData", None)
1887
- or getattr(item, "card_data", None)
1888
- or (item.get("cardData") if isinstance(item, dict) else None)
1889
- or {}
1890
- )
1891
- pretty = card.get("pretty_name") or repo_id
1892
- description = card.get("short_description") or card.get("description") or item.get("description", "")
1893
- updated = getattr(item, "lastModified", None) or item.get("lastModified")
1894
- if isinstance(updated, datetime):
1895
- updated_str = updated.isoformat()
1896
- else:
1897
- updated_str = str(updated) if updated else None
1898
-
1899
- results.append(
1900
- {
1901
- "repo_id": repo_id,
1902
- "pretty_name": pretty,
1903
- "description": description,
1904
- "author": getattr(item, "author", None) or item.get("author"),
1905
- "likes": getattr(item, "likes", None) or item.get("likes"),
1906
- "downloads": getattr(item, "downloads", None) or item.get("downloads"),
1907
- "last_modified": updated_str,
1908
- "tags": getattr(item, "tags", None) or item.get("tags"),
1909
- }
1910
- )
1911
- return results
1912
-
1913
- def _fetch_community_datasets_http(self) -> list[dict[str, Any]]:
1914
- base_params = {
1915
- "limit": MAX_COMMUNITY_DATASETS,
1916
- "full": "true",
1917
- "sort": "downloads",
1918
- "direction": "-1",
1919
- }
1920
- attempts = [
1921
- ("tag_search", {"search": f"tag:{COMMUNITY_DATASET_TAG}"}),
1922
- ("keyword_search", {"search": COMMUNITY_DATASET_TAG}),
1923
- ("legacy_filter", {"filter": COMMUNITY_DATASET_TAG}),
1924
- ]
1925
- for label, extra_params in attempts:
1926
- params = base_params.copy()
1927
- params.update(extra_params)
1928
- try:
1929
- import requests # lazy import — only needed for community datasets
1930
- logger.debug("HTTP dataset listing (%s) with params %s", label, params)
1931
- resp = requests.get(
1932
- HF_DATASETS_API_URL,
1933
- params=params,
1934
- timeout=30,
1935
- )
1936
- resp.raise_for_status()
1937
- data = resp.json()
1938
- if isinstance(data, list) and data:
1939
- logger.info("Fetched %d community datasets via HTTP (%s)", len(data), label)
1940
- return data
1941
- if isinstance(data, list):
1942
- logger.debug("HTTP dataset listing (%s) returned zero results.", label)
1943
- except Exception as exc:
1944
- logger.warning("HTTP dataset listing (%s) failed: %s", label, exc)
1945
- return []
1946
-
1947
- def _build_hf_readme(
1948
- self,
1949
- dataset_entry: DatasetEntry,
1950
- selected_meta: list[RecordingMetadata],
1951
- repo_id: str,
1952
- exported_bytes: int,
1953
- ) -> str:
1954
- pretty_name = f"{dataset_entry.label} • Reachy Mini Moves"
1955
- num_examples = len(selected_meta)
1956
- total_duration = sum(meta.duration for meta in selected_meta)
1957
- audio_count = sum(1 for meta in selected_meta if meta.has_audio)
1958
- latest_timestamp = max((meta.created_at for meta in selected_meta), default=time.time())
1959
- created_iso = datetime.utcfromtimestamp(latest_timestamp).strftime("%Y-%m-%dT%H:%M:%SZ")
1960
- move_rows = []
1961
- for meta in selected_meta:
1962
- recorded_at = datetime.utcfromtimestamp(meta.created_at).strftime("%Y-%m-%d %H:%M")
1963
- audio_label = "Yes" if meta.has_audio else "No"
1964
- move_rows.append(
1965
- f"| `{meta.move_id}` | {meta.duration:.1f}s | {audio_label} | {recorded_at} |"
1966
- )
1967
- moves_table = "\n".join(move_rows) if move_rows else "| – | – | – | – |"
1968
- front_matter = dedent(
1969
- f"""\
1970
- ---
1971
- dataset_info:
1972
- features:
1973
- - name: move_id
1974
- dtype: string
1975
- - name: description
1976
- dtype: string
1977
- - name: duration_seconds
1978
- dtype: float64
1979
- - name: has_audio
1980
- dtype: bool
1981
- splits:
1982
- - name: train
1983
- num_examples: {num_examples}
1984
- num_bytes: {exported_bytes}
1985
- download_size: {exported_bytes}
1986
- dataset_size: {exported_bytes}
1987
- configs:
1988
- - config_name: default
1989
- data_files:
1990
- - split: train
1991
- path: data/*.json
1992
- task_categories:
1993
- - robotics
1994
- language:
1995
- - en
1996
- tags:
1997
- - reachy_mini_community_moves
1998
- pretty_name: {pretty_name}
1999
- license: apache-2.0
2000
- ---
2001
- """
2002
- ).strip()
2003
-
2004
- body = dedent(
2005
- f"""
2006
- # {pretty_name}
2007
-
2008
- Community-contributed Marionette recordings captured on Reachy Mini.
2009
-
2010
- - **Moves uploaded:** {num_examples}
2011
- - **Total motion time:** {total_duration:.1f} seconds
2012
- - **Audio tracks:** {audio_count}
2013
- - **Last updated:** {created_iso}
2014
-
2015
- Files live under `data/` — each move ships as a JSON trajectory (Reachy Mini emotions schema) plus an optional WAV recorded directly from the robot.
2016
-
2017
- ## How this dataset was produced
2018
-
2019
- These takes were recorded with the Marionette Reachy Mini app. Pick the moves to share, set your Hugging Face username, run `huggingface-cli login` once locally, then hit **Synchronize to Hugging Face dataset** inside Marionette. The app packages the selected files, generates this README, and uploads them to `{repo_id}`.
2020
-
2021
- ## Selected moves
2022
-
2023
- | Move | Duration | Audio | Recorded at (UTC) |
2024
- | --- | --- | --- | --- |
2025
- {moves_table}
2026
-
2027
- ## Reuse
2028
-
2029
- - Cite this dataset as `{repo_id}`.
2030
- - Keep the `reachy_mini_community_moves` tag when sharing derivatives so the community can discover related sets.
2031
- """
2032
- ).strip()
2033
- return f"{front_matter}\n\n{body}\n"
2034
-
2035
- def _scaled_duration(
2036
- self, reachy_mini: ReachyMini, target_head_pose: np.ndarray, *, min_duration: float = 0.05
2037
- ) -> float:
2038
- _, _, magic_distance = distance_between_poses(
2039
- reachy_mini.get_current_head_pose(),
2040
- target_head_pose,
2041
- )
2042
- duration = magic_distance * 20 / 1000
2043
- return max(min_duration, duration)
2044
-
2045
- def _goto_pose_scaled(
2046
- self,
2047
- reachy_mini: ReachyMini,
2048
- head_pose: np.ndarray,
2049
- *,
2050
- antennas: list[float] | np.ndarray | None = None,
2051
- min_duration: float = 0.05,
2052
- ) -> None:
2053
- """Go to pose with duration scaled by distance. goto_target is blocking."""
2054
- duration = self._scaled_duration(reachy_mini, head_pose, min_duration=min_duration)
2055
- antennas_payload = list(antennas) if antennas is not None else None
2056
- reachy_mini.goto_target(
2057
- head=head_pose,
2058
- antennas=antennas_payload,
2059
- duration=duration,
2060
- )
2061
-
2062
- def _goto_current_pose(self, reachy_mini: ReachyMini, duration: float = 0.05) -> None:
2063
- head_pose = reachy_mini.get_current_head_pose()
2064
- _, antennas = reachy_mini.get_current_joint_positions()
2065
- reachy_mini.goto_target(
2066
- head=head_pose,
2067
- antennas=list(antennas) if antennas is not None else None,
2068
- duration=max(0.02, duration),
2069
- )
2070
-
2071
- def _safe_enable_motors(self, reachy_mini: ReachyMini) -> None:
2072
- self._goto_current_pose(reachy_mini, duration=0.05)
2073
- reachy_mini.enable_motors()
2074
- time.sleep(0.1)
2075
-
2076
- def _park_robot(self, reachy_mini: ReachyMini) -> None:
2077
- self._safe_enable_motors(reachy_mini)
2078
- self._goto_pose_scaled(
2079
- reachy_mini,
2080
- INIT_HEAD_POSE,
2081
- antennas=[0.0, 0.0],
2082
- min_duration=0.2,
2083
- )
2084
-
2085
- def _align_head_and_release(self, reachy_mini: ReachyMini) -> None:
2086
- self._safe_enable_motors(reachy_mini)
2087
- self._goto_pose_scaled(
2088
- reachy_mini,
2089
- INIT_HEAD_POSE,
2090
- antennas=[0.0, 0.0],
2091
- min_duration=0.2,
2092
- )
2093
- self._goto_sleep_and_release(reachy_mini)
2094
-
2095
- def _goto_sleep_and_release(self, reachy_mini: ReachyMini) -> None:
2096
- """Move head to sleep pose then disable motors (graceful torque-off)."""
2097
- # Use 15 degrees for antennas for style points
2098
- antenna_angle = np.deg2rad(15)
2099
- reachy_mini.goto_target(
2100
- SLEEP_HEAD_POSE,
2101
- antennas=[-antenna_angle, antenna_angle],
2102
- duration=1.0,
2103
- )
2104
- reachy_mini.disable_motors()
2105
-
2106
- def _delete_move_files(self, move_id: str) -> None:
2107
- json_path = self._dataset_dir / f"{move_id}.json"
2108
- if not json_path.exists():
2109
- raise HTTPException(status_code=404, detail=f"Move {move_id} not found.")
2110
- try:
2111
- json_path.unlink()
2112
- wav_path = json_path.with_suffix(".wav")
2113
- if wav_path.exists():
2114
- wav_path.unlink()
2115
- except OSError as exc:
2116
- raise HTTPException(status_code=500, detail=f"Failed to delete {move_id}: {exc}") from exc
2117
-
2118
- # ──────── State helpers ─────────────────────────────────────────────
2119
- def _serialize_state(self) -> dict[str, Any]:
2120
- with self._state_lock:
2121
- moves = sorted(
2122
- (meta.to_payload() for meta in self._recordings.values()),
2123
- key=lambda entry: entry["created_at"],
2124
- reverse=True,
2125
- )
2126
- # Unified timing: phase_start_at / phase_end_at
2127
- phase_start_at: float | None = None
2128
- phase_end_at: float | None = None
2129
- if self._mode == "countdown" and self._countdown_ends_at is not None:
2130
- phase_start_at = self._countdown_ends_at - COUNTDOWN_SECONDS
2131
- phase_end_at = self._countdown_ends_at
2132
- elif self._mode == "recording" and self._active_record_started_at is not None:
2133
- phase_start_at = self._active_record_started_at
2134
- if self._active_record_duration is not None:
2135
- phase_end_at = self._active_record_started_at + self._active_record_duration
2136
-
2137
- return {
2138
- "server_time": time.time(),
2139
- "mode": self._mode,
2140
- "message": self._message,
2141
- "active_move": self._active_move,
2142
- "phase_start_at": phase_start_at,
2143
- "phase_end_at": phase_end_at,
2144
- "countdown_ends_at": self._countdown_ends_at,
2145
- "recording_started_at": self._active_record_started_at,
2146
- "recording_duration": self._active_record_duration,
2147
- "recording_stats": self._recording_stats,
2148
- "pending_recording": self._pending_recording.label if self._pending_recording else None,
2149
- "pending_playback": self._pending_playback,
2150
- "moves": moves,
2151
- "config": {
2152
- "default_duration": DEFAULT_DURATION,
2153
- "preferred_duration": self._preferred_duration,
2154
- "countdown_seconds": COUNTDOWN_SECONDS,
2155
- "motion_sample_rate": MOTION_SAMPLE_RATE,
2156
- "audio_available": self._audio_available,
2157
- "active_dataset_path": str(self._dataset_dir),
2158
- "dataset_root_path": str(self._dataset_root),
2159
- "hf_username": self._check_hf_login(),
2160
- "motion_models": self._motion_model_registry.to_payload(),
2161
- "welcome_messages": self._welcome_messages,
2162
- },
2163
- "datasets": self._datasets_payload(),
2164
- }
2165
-
2166
- _UNSET = object()
2167
-
2168
- def _set_state(
2169
- self,
2170
- *,
2171
- mode: str | None = None,
2172
- message: str | None = None,
2173
- active_move: object = _UNSET,
2174
- countdown_ends_at: object = _UNSET,
2175
- recording_started_at: object = _UNSET,
2176
- recording_duration: object = _UNSET,
2177
- ) -> None:
2178
- with self._state_lock:
2179
- if mode is not None:
2180
- self._mode = mode
2181
- if message is not None:
2182
- self._message = message
2183
- if active_move is not self._UNSET:
2184
- self._active_move = active_move
2185
- if countdown_ends_at is not self._UNSET:
2186
- self._countdown_ends_at = countdown_ends_at
2187
- if recording_started_at is not self._UNSET:
2188
- self._active_record_started_at = recording_started_at
2189
- if recording_duration is not self._UNSET:
2190
- self._active_record_duration = recording_duration
2191
-
2192
- def _set_idle_state(self) -> None:
2193
- self._set_state(
2194
- mode="idle",
2195
- message="Ready to capture moves",
2196
- active_move=None,
2197
- countdown_ends_at=None,
2198
- recording_started_at=None,
2199
- recording_duration=None,
2200
- )
2201
-
2202
-
2203
- def create_app(
2204
- registry_path: Path | None = None,
2205
- dataset_root: Path | None = None,
2206
- ) -> tuple["FastAPI", "Marionette"]:
2207
- """Create a Marionette instance and its FastAPI app.
2208
-
2209
- Useful for testing with FastAPI's TestClient:
2210
- app, marionette = create_app(registry_path=tmp / "reg.json", dataset_root=tmp / "ds")
2211
- client = TestClient(app)
2212
- """
2213
- from fastapi import FastAPI as _FastAPI
2214
-
2215
- marionette = Marionette(
2216
- registry_path=registry_path,
2217
- dataset_root=dataset_root,
2218
- )
2219
- assert marionette.settings_app is not None
2220
- return marionette.settings_app, marionette
2221
-
2222
 
 
 
2223
  if __name__ == "__main__":
2224
  app = Marionette()
2225
  try:
 
1
+ """Marionette re-export hub for backward compatibility.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
 
3
+ All public names are available via ``from marionette.main import ...`` so
4
+ existing tests and entry points continue to work unchanged after the
5
+ backend was split into focused submodules.
6
+ """
7
+ from __future__ import annotations
 
 
 
 
 
 
 
 
 
8
 
9
+ # Re-export all public types, constants, and utilities from submodules.
10
+ from marionette.models import ( # noqa: F401
11
+ AUDIO_SAMPLE_RATE,
12
+ COMMUNITY_DATASET_TAG,
13
+ COUNTDOWN_SECONDS,
14
+ DATASET_DATA_SUBDIR,
15
+ DATASET_DIRNAME,
16
+ DATASET_REGISTRY_FILENAME,
17
+ DEFAULT_DATASET_LABEL,
18
+ DEFAULT_DURATION,
19
+ HF_DATASETS_API_URL,
20
+ MAX_COMMUNITY_DATASETS,
21
+ MOTION_SAMPLE_RATE,
22
+ SEMI_AWAKEN_POSE,
23
+ CreateDatasetPayload,
24
+ DatasetEntry,
25
+ DownloadDatasetPayload,
26
+ HfTokenPayload,
27
+ PlayMovePayload,
28
+ RecordingMetadata,
29
+ RecordingRequest,
30
+ RobotAudioSelectPayload,
31
+ SelectDatasetPayload,
32
+ StartRecordingPayload,
33
+ SyncDatasetPayload,
34
+ UpdateDatasetRootPayload,
35
+ UpdateExperimentsPayload,
36
+ UpdateLeadCompensationPayload,
37
+ _slugify,
38
  )
39
 
40
+ from marionette.audio import ( # noqa: F401
41
+ call_with_timeout,
42
+ get_audio_duration,
43
+ play_preloaded_wav,
44
+ play_wav_chunked,
45
+ preload_wav,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
  )
47
 
48
+ from marionette.state import StateMixin # noqa: F401
49
+ from marionette.recording import RecordingMixin # noqa: F401
50
+ from marionette.datasets import ( # noqa: F401
51
+ DatasetMixin,
52
+ hf_whoami,
53
+ hf_login,
54
+ hf_logout,
55
+ hf_get_token,
56
+ HfApi,
57
+ DatasetFilter,
58
+ snapshot_download,
59
+ )
60
+ from marionette.routes import register_routes # noqa: F401
61
+ from marionette.app import Marionette, create_app, __version__ # noqa: F401
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
 
63
+ # The daemon runs `python -u -m marionette.main` to launch the app.
64
+ # Without this block, the module just imports and exits immediately.
65
  if __name__ == "__main__":
66
  app = Marionette()
67
  try:
marionette/models.py ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Data models, constants, and utility functions for the Marionette app.
2
+
3
+ This module contains all the shared types (dataclasses, Pydantic models),
4
+ configuration constants, and small utility functions used across the
5
+ application. It has no dependencies on other marionette submodules.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import re
10
+ from dataclasses import dataclass
11
+ from pathlib import Path
12
+ from typing import Any
13
+
14
+ import numpy as np
15
+ from pydantic import BaseModel, Field
16
+
17
+ # ──────── Configuration constants ─────────────────────────────────────
18
+
19
+ AUDIO_SAMPLE_RATE = 44_100 # Default sample rate for mic recording (Hz)
20
+ MOTION_SAMPLE_RATE = 100.0 # How often we sample the robot's pose (Hz)
21
+ COUNTDOWN_SECONDS = 3 # Seconds of countdown before recording starts
22
+ # Compensation for push_audio_sample pipeline latency during playback.
23
+ # Audio is started this many ms before the first motion command so that
24
+ # both arrive at the speaker/motors at the same time.
25
+ # Measured via tests/test_marionette_sync.py: 320ms = 245ms GStreamer
26
+ # pipeline latency + ~75ms thread-scheduling overhead.
27
+ AUDIO_LEAD_MS = 320
28
+ DEFAULT_DURATION = 5.0 # Default recording duration (seconds)
29
+ DATASET_DIRNAME = "local_dataset" # Folder name for the default dataset
30
+ DATASET_REGISTRY_FILENAME = "dataset_registry.json" # Persists dataset config
31
+ DEFAULT_DATASET_LABEL = "Local dataset"
32
+ COMMUNITY_DATASET_TAG = "reachy_mini_community_moves" # HF tag for discovery
33
+ MAX_COMMUNITY_DATASETS = 50 # Cap on HF API results
34
+ HF_DATASETS_API_URL = "https://huggingface.co/api/datasets"
35
+ DATASET_DATA_SUBDIR = "data" # Subdirectory within each dataset folder
36
+
37
+ # Semi-awaken pose: identity rotation, same XY as sleep, Z raised 1.5cm
38
+ SEMI_AWAKEN_POSE = np.array(
39
+ [
40
+ [1.0, 0.0, 0.0, -0.021],
41
+ [0.0, 1.0, 0.0, 0.001],
42
+ [0.0, 0.0, 1.0, -0.029],
43
+ [0.0, 0.0, 0.0, 1.0],
44
+ ]
45
+ )
46
+
47
+
48
+ # ──────── Utility functions ───────────────────────────────────────────
49
+
50
+ def _slugify(value: str) -> str:
51
+ """Convert a human-readable string to a filesystem-safe slug.
52
+
53
+ Used for move IDs and dataset folder names. Falls back to "take"
54
+ if the input contains no alphanumeric characters.
55
+ """
56
+ slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
57
+ return slug or "take"
58
+
59
+
60
+ # ──────── Dataclasses ─────────────────────────────────────────────────
61
+
62
+ @dataclass
63
+ class RecordingMetadata:
64
+ """In-memory representation of a saved recording (one JSON file on disk).
65
+
66
+ Populated by _refresh_recordings() which scans the active dataset directory.
67
+ """
68
+ move_id: str # Unique identifier, same as the JSON filename stem
69
+ label: str # Human-readable name shown in the UI
70
+ description: str # User-provided or auto-generated description
71
+ duration: float # Duration in seconds (computed from timestamps)
72
+ created_at: float # Unix timestamp of file modification time
73
+ json_path: Path # Absolute path to the .json trajectory file
74
+ has_audio: bool # Whether a matching .wav file exists
75
+ is_uploaded: bool = False # Whether this move was synced to HF
76
+ audio_only: bool = False # True for mic-only recordings (no motion)
77
+
78
+ def to_payload(self) -> dict[str, Any]:
79
+ return {
80
+ "id": self.move_id,
81
+ "label": self.label,
82
+ "duration": self.duration,
83
+ "created_at": self.created_at,
84
+ "has_audio": self.has_audio,
85
+ "description": self.description,
86
+ "is_uploaded": self.is_uploaded,
87
+ "audio_only": self.audio_only,
88
+ }
89
+
90
+
91
+ @dataclass
92
+ class RecordingRequest:
93
+ """Parameters for a pending recording, built from StartRecordingPayload.
94
+
95
+ Created by _build_recording_request() and consumed by _perform_recording().
96
+ Passed from the HTTP thread to the robot thread via _pending_recording.
97
+ """
98
+ move_id: str # Generated unique slug
99
+ label: str # Display name
100
+ description: str # Description text
101
+ duration: float # Requested duration in seconds
102
+ record_audio: bool # Whether to capture from the mic
103
+ record_motion: bool = True # False = mic-only (no pose capture)
104
+ uploaded_audio_path: Path | None = None # Pre-uploaded audio to play during recording
105
+
106
+
107
+ @dataclass
108
+ class DatasetEntry:
109
+ """Represents one dataset (a folder containing recordings).
110
+
111
+ A dataset is a directory under the dataset root. It has a unique ID
112
+ (for API use), a human label, and tracks which moves have been synced
113
+ to Hugging Face.
114
+ """
115
+ dataset_id: str # Unique identifier for API calls
116
+ label: str # Human-readable name
117
+ folder: str # Filesystem folder name under dataset root
118
+ path: Path # Absolute path to the dataset folder
119
+ uploaded_move_ids: set[str] | None = None # Move IDs that have been synced to HF
120
+ origin: str = "local" # "local" (user-created) or "downloaded" (from HF)
121
+
122
+ def __post_init__(self) -> None:
123
+ if self.uploaded_move_ids is None:
124
+ self.uploaded_move_ids = set()
125
+
126
+ def to_payload(self) -> dict[str, Any]:
127
+ return {
128
+ "id": self.dataset_id,
129
+ "label": self.label,
130
+ "path": str(self.path),
131
+ "folder": self.folder,
132
+ "origin": self.origin,
133
+ }
134
+
135
+
136
+ # ──────── Pydantic request models ────────────────────────────────────
137
+
138
+ class StartRecordingPayload(BaseModel):
139
+ duration: float = Field(DEFAULT_DURATION, gt=0.5, le=300.0)
140
+ record_audio: bool = Field(default=True)
141
+ record_motion: bool = Field(default=True, description="When False, record audio only (no motion capture)")
142
+ label: str | None = Field(default=None, max_length=80)
143
+ description: str | None = Field(default=None, max_length=500)
144
+ uploaded_audio_id: str | None = Field(default=None, description="ID of previously uploaded audio file")
145
+
146
+
147
+ class PlayMovePayload(BaseModel):
148
+ move_id: str = Field(..., description="Move identifier (filename stem)")
149
+
150
+
151
+ class CreateDatasetPayload(BaseModel):
152
+ name: str = Field(..., description="Folder name for the dataset", min_length=1, max_length=80)
153
+ label: str | None = Field(default=None, max_length=80)
154
+
155
+
156
+ class SelectDatasetPayload(BaseModel):
157
+ dataset_id: str
158
+
159
+
160
+ class SyncDatasetPayload(BaseModel):
161
+ move_ids: list[str] = Field(..., min_length=1, description="Subset of moves to publish")
162
+ hf_username: str | None = Field(default=None, min_length=2, max_length=80)
163
+ dataset_slug: str | None = Field(
164
+ default=None, description="Optional override for the Hugging Face dataset slug"
165
+ )
166
+
167
+
168
+ class UpdateDatasetRootPayload(BaseModel):
169
+ path: str = Field(..., description="Filesystem directory containing all datasets")
170
+
171
+
172
+ class DownloadDatasetPayload(BaseModel):
173
+ repo_id: str = Field(..., description="Hugging Face dataset repository, e.g. user/name")
174
+ label: str | None = Field(default=None, max_length=80)
175
+ name: str | None = Field(
176
+ default=None,
177
+ description="Optional folder name; defaults to the dataset slug",
178
+ max_length=80,
179
+ )
180
+
181
+
182
+ class UpdateLeadCompensationPayload(BaseModel):
183
+ lead_frames_head: int | None = Field(
184
+ default=None, ge=0, le=2000, description="Look-ahead for head/body in 100Hz frames"
185
+ )
186
+ lead_frames_antennas: int | None = Field(
187
+ default=None, ge=0, le=2000, description="Look-ahead for antennas in 100Hz frames"
188
+ )
189
+
190
+
191
+ class UpdateExperimentsPayload(BaseModel):
192
+ duration_seconds: float | None = Field(default=None, gt=0.5, le=300.0)
193
+ welcome_messages: int | None = Field(default=None, ge=0, le=2, description="Number of welcome messages at startup (0, 1, or 2)")
194
+
195
+
196
+ class HfTokenPayload(BaseModel):
197
+ token: str = Field(..., min_length=5, description="Hugging Face access token (starts with hf_)")
198
+
199
+
200
+
201
+ class RobotAudioSelectPayload(BaseModel):
202
+ path: str = Field(..., description="Absolute path to a WAV file on the robot")
marionette/motion_models.py CHANGED
@@ -3,7 +3,7 @@
3
  from __future__ import annotations
4
 
5
  import copy
6
- from typing import Any, Dict
7
 
8
  try:
9
  from reachy_mini.motion.recorded_move import RecordedMove
@@ -11,7 +11,7 @@ except Exception: # pragma: no cover - reachable at runtime
11
  RecordedMove = None # type: ignore
12
 
13
 
14
- def _deepcopy_move(move: Dict[str, Any]) -> Dict[str, Any]:
15
  """Copy a move dict without mutating the source."""
16
  return copy.deepcopy(move)
17
 
@@ -31,7 +31,7 @@ class LeadCompensationModel:
31
  self.lead_frames_antennas = max(0, int(lead_frames_antennas))
32
  self.lead_frames_head = max(0, int(lead_frames_head))
33
 
34
- def transform(self, move: Dict[str, Any]) -> Dict[str, Any]:
35
  patched = _deepcopy_move(move)
36
  data = patched.get("set_target_data", [])
37
  if not data:
@@ -59,7 +59,7 @@ class MotionModelRegistry:
59
  def active(self) -> str:
60
  return self._active
61
 
62
- def set_model_params(self, name: str, params: Dict[str, Any]) -> None:
63
  if name != LeadCompensationModel.name:
64
  raise KeyError(name)
65
  head = params.get("lead_frames_head")
@@ -69,7 +69,7 @@ class MotionModelRegistry:
69
  if antennas is not None:
70
  self._model.lead_frames_antennas = max(0, int(antennas))
71
 
72
- def get_model_params(self, name: str) -> Dict[str, Any]:
73
  if name != LeadCompensationModel.name:
74
  raise KeyError(name)
75
  return {
@@ -77,7 +77,7 @@ class MotionModelRegistry:
77
  "lead_frames_antennas": int(self._model.lead_frames_antennas),
78
  }
79
 
80
- def transform_move(self, move_data: Dict[str, Any]) -> Dict[str, Any]:
81
  return self._model.transform(move_data)
82
 
83
  def apply(self, recorded_move: RecordedMove) -> RecordedMove:
@@ -86,7 +86,7 @@ class MotionModelRegistry:
86
  mutated = self.transform_move(recorded_move.move)
87
  return RecordedMove(mutated, recorded_move.sound_path)
88
 
89
- def to_payload(self) -> Dict[str, Any]:
90
  return {
91
  "active": self._active,
92
  "models": [
 
3
  from __future__ import annotations
4
 
5
  import copy
6
+ from typing import Any
7
 
8
  try:
9
  from reachy_mini.motion.recorded_move import RecordedMove
 
11
  RecordedMove = None # type: ignore
12
 
13
 
14
+ def _deepcopy_move(move: dict[str, Any]) -> dict[str, Any]:
15
  """Copy a move dict without mutating the source."""
16
  return copy.deepcopy(move)
17
 
 
31
  self.lead_frames_antennas = max(0, int(lead_frames_antennas))
32
  self.lead_frames_head = max(0, int(lead_frames_head))
33
 
34
+ def transform(self, move: dict[str, Any]) -> dict[str, Any]:
35
  patched = _deepcopy_move(move)
36
  data = patched.get("set_target_data", [])
37
  if not data:
 
59
  def active(self) -> str:
60
  return self._active
61
 
62
+ def set_model_params(self, name: str, params: dict[str, Any]) -> None:
63
  if name != LeadCompensationModel.name:
64
  raise KeyError(name)
65
  head = params.get("lead_frames_head")
 
69
  if antennas is not None:
70
  self._model.lead_frames_antennas = max(0, int(antennas))
71
 
72
+ def get_model_params(self, name: str) -> dict[str, Any]:
73
  if name != LeadCompensationModel.name:
74
  raise KeyError(name)
75
  return {
 
77
  "lead_frames_antennas": int(self._model.lead_frames_antennas),
78
  }
79
 
80
+ def transform_move(self, move_data: dict[str, Any]) -> dict[str, Any]:
81
  return self._model.transform(move_data)
82
 
83
  def apply(self, recorded_move: RecordedMove) -> RecordedMove:
 
86
  mutated = self.transform_move(recorded_move.move)
87
  return RecordedMove(mutated, recorded_move.sound_path)
88
 
89
+ def to_payload(self) -> dict[str, Any]:
90
  return {
91
  "active": self._active,
92
  "models": [
marionette/recording.py ADDED
@@ -0,0 +1,678 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Recording, capture, and playback mixin for the Marionette app.
2
+
3
+ Contains all methods related to motion capture, audio recording during
4
+ capture, move playback, and recording file I/O (save/load/refresh).
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import json
9
+ import logging
10
+ import shutil
11
+ import threading
12
+ import time
13
+ from datetime import datetime
14
+ from pathlib import Path
15
+ from typing import Any, Callable
16
+
17
+ import numpy as np
18
+ from reachy_mini.motion.recorded_move import RecordedMove
19
+
20
+ from marionette.audio import (
21
+ call_with_timeout,
22
+ generate_beep,
23
+ play_preloaded_wav,
24
+ play_wav_chunked,
25
+ preload_wav,
26
+ )
27
+ from marionette.models import (
28
+ AUDIO_LEAD_MS,
29
+ AUDIO_SAMPLE_RATE,
30
+ COUNTDOWN_SECONDS,
31
+ MOTION_SAMPLE_RATE,
32
+ RecordingMetadata,
33
+ RecordingRequest,
34
+ StartRecordingPayload,
35
+ _slugify,
36
+ )
37
+
38
+ try:
39
+ import soundfile as sf
40
+ except Exception: # pragma: no cover
41
+ sf = None
42
+
43
+ logger = logging.getLogger(__name__)
44
+
45
+
46
+ class RecordingMixin:
47
+ """Mixin providing recording, capture, and playback methods.
48
+
49
+ Expects the host class to have: _state_lock, _recording_cancel_event,
50
+ _playback_cancel_event, _recordings, _dataset_dir, _audio_available,
51
+ _preferred_duration, _motion_model_registry, _set_state(),
52
+ _set_idle_state(), _safe_enable_motors(), _goto_pose_scaled(),
53
+ _goto_sleep_and_release().
54
+ """
55
+
56
+ def _perform_recording(
57
+ self,
58
+ reachy_mini: object,
59
+ stop_event: threading.Event,
60
+ request: RecordingRequest,
61
+ ) -> None:
62
+ self._recording_cancel_event.clear()
63
+
64
+ # ── Pre-countdown: preload audio + warm pipeline ──────────────
65
+ # All audio prep (resample, GStreamer warm-up) happens BEFORE the
66
+ # countdown so there's zero lag at the critical sync point.
67
+ audio_stop = threading.Event()
68
+ audio_start = threading.Event()
69
+ audio_thread: threading.Thread | None = None
70
+ pipeline_warmed = False
71
+
72
+ try:
73
+ sr_out = int(reachy_mini.media.get_output_audio_samplerate() or 16000)
74
+ except Exception:
75
+ sr_out = 16000
76
+
77
+ if request.uploaded_audio_path and request.uploaded_audio_path.exists():
78
+ self._set_state(
79
+ mode="preparing",
80
+ message=f"Preparing audio for {request.label}",
81
+ active_move=request.label,
82
+ )
83
+ wav_data = preload_wav(request.uploaded_audio_path, target_sr=sr_out)
84
+ if wav_data is not None:
85
+ self._warm_audio_pipeline(reachy_mini)
86
+ pipeline_warmed = True
87
+ audio_thread = threading.Thread(
88
+ target=play_preloaded_wav,
89
+ args=(reachy_mini, wav_data, audio_stop),
90
+ kwargs={"pipeline_ready": True, "start_signal": audio_start},
91
+ daemon=True,
92
+ )
93
+ audio_thread.start()
94
+
95
+ # Warm pipeline for countdown beeps (even without uploaded audio)
96
+ if not pipeline_warmed:
97
+ self._warm_audio_pipeline(reachy_mini)
98
+ pipeline_warmed = True
99
+
100
+ # ── Countdown with beeps ──────────────────────────────────────
101
+ countdown_end = time.perf_counter() + COUNTDOWN_SECONDS
102
+ self._set_state(
103
+ mode="countdown",
104
+ message=f"Recording {request.label} in {COUNTDOWN_SECONDS}s",
105
+ active_move=request.label,
106
+ countdown_ends_at=time.time() + COUNTDOWN_SECONDS,
107
+ )
108
+
109
+ beeped = set()
110
+ beep_samples = generate_beep(freq=440, duration=0.1, sr=sr_out)
111
+ go_beep_samples = generate_beep(freq=880, duration=0.1, sr=sr_out)
112
+ while time.perf_counter() < countdown_end:
113
+ # Play beeps at each second boundary
114
+ elapsed = COUNTDOWN_SECONDS - (countdown_end - time.perf_counter())
115
+ second = int(elapsed)
116
+ if second not in beeped and 0 <= second < COUNTDOWN_SECONDS:
117
+ beeped.add(second)
118
+ try:
119
+ reachy_mini.media.push_audio_sample(beep_samples)
120
+ except Exception:
121
+ pass
122
+ if stop_event.wait(0.02) or self._recording_cancel_event.is_set():
123
+ self._recording_cancel_event.clear()
124
+ self._cleanup_audio(audio_thread, audio_start, audio_stop, pipeline_warmed, reachy_mini, has_uploaded_audio=audio_thread is not None)
125
+ self._set_state(mode="idle", message="Recording cancelled", active_move=None)
126
+ return
127
+
128
+ # 4th beep: GO! (higher pitch, fires at capture start)
129
+ try:
130
+ reachy_mini.media.push_audio_sample(go_beep_samples)
131
+ except Exception:
132
+ pass
133
+
134
+ # Stop pipeline if we only warmed it for beeps (no uploaded audio to play)
135
+ if pipeline_warmed and audio_thread is None:
136
+ try:
137
+ call_with_timeout(reachy_mini.media.stop_playing)
138
+ except Exception:
139
+ pass
140
+
141
+ with self._state_lock:
142
+ self._recording_stats = None
143
+
144
+ self._set_state(
145
+ mode="recording",
146
+ message=f"Recording {request.label}",
147
+ active_move=request.move_id,
148
+ recording_started_at=time.time(),
149
+ recording_duration=request.duration,
150
+ )
151
+
152
+ try:
153
+ self._run_capture_and_save(
154
+ reachy_mini, stop_event, request,
155
+ audio_thread=audio_thread,
156
+ audio_start=audio_start,
157
+ audio_stop=audio_stop,
158
+ )
159
+ except Exception as exc:
160
+ logger.error("Recording failed: %s", exc, exc_info=True)
161
+ self._cleanup_audio(audio_thread, audio_start, audio_stop, False, reachy_mini, has_uploaded_audio=audio_thread is not None)
162
+ self._set_state(
163
+ mode="error",
164
+ message=f"Recording error: {exc}",
165
+ active_move=None,
166
+ )
167
+
168
+ @staticmethod
169
+ def _warm_audio_pipeline(reachy_mini: object) -> None:
170
+ """Start GStreamer and push a silent buffer to avoid cold-start lag."""
171
+ reachy_mini.media.start_playing()
172
+ try:
173
+ reachy_mini.media.push_audio_sample(
174
+ np.zeros(160, dtype=np.float32)
175
+ )
176
+ except Exception:
177
+ pass
178
+
179
+ @staticmethod
180
+ def _cleanup_audio(
181
+ audio_thread: threading.Thread | None,
182
+ audio_start: threading.Event,
183
+ audio_stop: threading.Event,
184
+ pipeline_warmed: bool,
185
+ reachy_mini: object,
186
+ has_uploaded_audio: bool,
187
+ ) -> None:
188
+ """Clean up audio thread and pipeline on cancellation or error."""
189
+ audio_start.set()
190
+ audio_stop.set()
191
+ if audio_thread is not None:
192
+ audio_thread.join(timeout=3.0)
193
+ if pipeline_warmed and not has_uploaded_audio:
194
+ try:
195
+ call_with_timeout(reachy_mini.media.stop_playing)
196
+ except Exception:
197
+ pass
198
+
199
+ def _run_capture_and_save(
200
+ self,
201
+ reachy_mini: object,
202
+ stop_event: threading.Event,
203
+ request: RecordingRequest,
204
+ audio_thread: threading.Thread | None = None,
205
+ audio_start: threading.Event | None = None,
206
+ audio_stop: threading.Event | None = None,
207
+ ) -> None:
208
+ """Inner recording logic, wrapped by _perform_recording's safety net.
209
+
210
+ Audio preload/resample/pipeline-warm is done in _perform_recording
211
+ before the countdown. This method receives the pre-built audio thread
212
+ and events, and only needs to fire audio_start at capture start.
213
+ """
214
+ if audio_start is None:
215
+ audio_start = threading.Event()
216
+ if audio_stop is None:
217
+ audio_stop = threading.Event()
218
+
219
+ # When using uploaded audio, don't record from mic (audio comes from the file)
220
+ should_record_mic = request.record_audio and not request.uploaded_audio_path
221
+ try:
222
+ timestamps, frames, audio_frames, audio_samplerate = self._capture_motion(
223
+ reachy_mini, stop_event, request.duration, should_record_mic,
224
+ record_motion=request.record_motion,
225
+ on_capture_start=(audio_start.set if audio_thread is not None else None),
226
+ )
227
+ finally:
228
+ # Unblock the audio thread's start_signal.wait() first, then
229
+ # signal stop. Without this, an early capture failure would
230
+ # leave the audio thread stranded in wait() for up to 5s.
231
+ audio_start.set()
232
+ audio_stop.set()
233
+ if audio_thread is not None:
234
+ audio_thread.join(timeout=3.0)
235
+
236
+ # Check if recording was cancelled
237
+ was_cancelled = self._recording_cancel_event.is_set()
238
+ self._recording_cancel_event.clear()
239
+
240
+ if not timestamps:
241
+ self._set_state(
242
+ mode="idle",
243
+ message="Recording cancelled" if was_cancelled else "No motion data captured.",
244
+ active_move=None,
245
+ )
246
+ return
247
+
248
+ # Save the recording (even if partial due to early stop)
249
+ self._save_recording(request, timestamps, frames, audio_frames, audio_samplerate)
250
+ self._refresh_recordings()
251
+ pose_count = len(frames)
252
+ duration_elapsed = timestamps[-1] if timestamps else request.duration
253
+ duration_elapsed = max(duration_elapsed, 1e-6)
254
+ poses_per_sec = pose_count / duration_elapsed
255
+ with self._state_lock:
256
+ self._recording_stats = {
257
+ "poses": pose_count,
258
+ "duration": duration_elapsed,
259
+ "poses_per_second": poses_per_sec,
260
+ }
261
+ if not request.record_motion:
262
+ status_msg = f"Recorded audio: {request.label} ({duration_elapsed:.1f}s)"
263
+ if was_cancelled:
264
+ status_msg = f"Saved audio: {request.label} (stopped early, {duration_elapsed:.1f}s)"
265
+ else:
266
+ status_msg = f"Recorded {request.label} • {pose_count} poses ({poses_per_sec:.1f}/s)"
267
+ if was_cancelled:
268
+ status_msg = f"Saved {request.label} (stopped early) • {pose_count} poses"
269
+ self._set_state(
270
+ mode="idle",
271
+ message=status_msg,
272
+ active_move=None,
273
+ )
274
+ if not was_cancelled:
275
+ self._preferred_duration = request.duration
276
+
277
+ def _perform_playback(self, reachy_mini: object, move_id: str) -> None:
278
+ """Replay a recorded move with optional audio.
279
+
280
+ Playback flow for motion moves:
281
+ 1. Load JSON trajectory and apply lead compensation
282
+ 2. Preload WAV and resample (overlapped with goto start pose)
283
+ 3. Warm GStreamer pipeline, spawn audio thread waiting on start_signal
284
+ 4. _stream_playback sends first motion command then fires start_signal
285
+ 5. Audio and motion run in sync until done or cancelled
286
+ """
287
+ meta = self._recordings.get(move_id)
288
+ if not meta:
289
+ self._set_state(mode="error", message=f"Move {move_id} missing.", active_move=None)
290
+ return
291
+
292
+ try:
293
+ move = self._load_move(meta.json_path)
294
+ except Exception as exc: # pragma: no cover - filesystem failure
295
+ self._set_state(
296
+ mode="error",
297
+ message=f"Failed to load {move_id}: {exc}",
298
+ active_move=None,
299
+ )
300
+ return
301
+
302
+ # Audio-only moves: just play the audio, no motion replay
303
+ if meta.audio_only:
304
+ self._playback_cancel_event.clear()
305
+ self._set_state(
306
+ mode="playing",
307
+ message=f"Playing audio: {meta.label}",
308
+ active_move=meta.move_id,
309
+ )
310
+ if move.sound_path is not None:
311
+ try:
312
+ play_wav_chunked(
313
+ reachy_mini, move.sound_path, self._playback_cancel_event,
314
+ )
315
+ except Exception:
316
+ pass
317
+ cancelled = self._playback_cancel_event.is_set()
318
+ self._playback_cancel_event.clear()
319
+ msg = f"Playback stopped for {meta.label}" if cancelled else f"Finished playing {meta.label}"
320
+ self._set_state(mode="idle", message=msg, active_move=None)
321
+ return
322
+
323
+ move_to_play = self._apply_motion_model(move)
324
+
325
+ self._playback_cancel_event.clear()
326
+ self._set_state(
327
+ mode="playing",
328
+ message=f"Playing {meta.label}",
329
+ active_move=meta.move_id,
330
+ )
331
+
332
+ self._safe_enable_motors(reachy_mini)
333
+ cancelled = False
334
+
335
+ # Preload audio (including resample) while we go to the start pose,
336
+ # so playback can begin instantly when motion starts.
337
+ try:
338
+ sr_out = int(reachy_mini.media.get_output_audio_samplerate() or 16000)
339
+ except Exception:
340
+ sr_out = 16000
341
+ wav_data: tuple[np.ndarray, int] | None = None
342
+ if move_to_play.sound_path is not None:
343
+ wav_data = preload_wav(move_to_play.sound_path, target_sr=sr_out)
344
+
345
+ # Move to the start pose BEFORE starting audio so they begin in sync.
346
+ try:
347
+ start_head_pose, start_antennas, start_body_yaw = move_to_play.evaluate(0.0)
348
+ except Exception:
349
+ self._set_state(mode="error", message="Failed to evaluate move start pose", active_move=None)
350
+ return
351
+ self._goto_pose_scaled(
352
+ reachy_mini,
353
+ start_head_pose,
354
+ antennas=list(start_antennas) if start_antennas is not None else None,
355
+ min_duration=0.2,
356
+ )
357
+ if start_body_yaw is not None:
358
+ reachy_mini.set_target_body_yaw(float(start_body_yaw))
359
+
360
+ # Warm the audio pipeline and start audio early to compensate for
361
+ # push_audio_sample pipeline latency (see AUDIO_LEAD_MS).
362
+ audio_stop = threading.Event()
363
+ audio_start = threading.Event()
364
+ audio_thread: threading.Thread | None = None
365
+ if wav_data is not None:
366
+ reachy_mini.media.start_playing()
367
+ # Warm the GStreamer pipeline (see recording path for explanation).
368
+ try:
369
+ reachy_mini.media.push_audio_sample(
370
+ np.zeros(160, dtype=np.float32)
371
+ )
372
+ except Exception:
373
+ pass
374
+ audio_thread = threading.Thread(
375
+ target=play_preloaded_wav,
376
+ args=(reachy_mini, wav_data, audio_stop),
377
+ kwargs={"pipeline_ready": True, "start_signal": audio_start},
378
+ daemon=True,
379
+ )
380
+ audio_thread.start()
381
+ # Start audio BEFORE motion so that by the time chunks traverse
382
+ # the GStreamer pipeline, the first audible output coincides with
383
+ # the first physical movement.
384
+ audio_start.set()
385
+ time.sleep(AUDIO_LEAD_MS / 1000.0)
386
+ try:
387
+ cancelled = not self._stream_playback(
388
+ reachy_mini, move_to_play,
389
+ )
390
+ finally:
391
+ self._playback_cancel_event.clear()
392
+ if audio_thread is not None:
393
+ if cancelled:
394
+ # Force-stop audio immediately on user cancellation.
395
+ audio_stop.set()
396
+ audio_thread.join(timeout=3.0)
397
+ else:
398
+ # Normal completion: let the audio thread finish naturally
399
+ # (it may still be draining its buffer since chunks are
400
+ # pushed ~20% faster than real-time).
401
+ audio_thread.join(timeout=max(10.0, move_to_play.duration))
402
+ audio_stop.set() # cleanup: ensure thread exits
403
+
404
+ if cancelled:
405
+ self._set_state(
406
+ mode="idle",
407
+ message=f"Playback stopped for {meta.label}",
408
+ active_move=None,
409
+ )
410
+ else:
411
+ self._set_state(
412
+ mode="idle",
413
+ message=f"Finished playing {meta.label}",
414
+ active_move=None,
415
+ )
416
+
417
+ # ──────── Capture helpers ─────────────────────────────────────────
418
+
419
+ def _capture_motion(
420
+ self,
421
+ reachy_mini: object,
422
+ stop_event: threading.Event,
423
+ duration: float,
424
+ record_audio: bool,
425
+ record_motion: bool = True,
426
+ on_capture_start: Callable[[], None] | None = None,
427
+ ) -> tuple[list[float], list[dict[str, Any]], list[np.ndarray], int | None]:
428
+ timestamps: list[float] = []
429
+ frames: list[dict[str, Any]] = []
430
+ audio_frames: list[np.ndarray] = []
431
+ audio_samplerate: int | None = None
432
+ audio_active = False
433
+
434
+ def _pull_audio_frames(max_seconds: float = 1.0) -> None:
435
+ if not audio_active:
436
+ return
437
+ deadline = time.perf_counter() + max_seconds
438
+ while time.perf_counter() < deadline:
439
+ sample = reachy_mini.media.get_audio_sample()
440
+ if sample is None:
441
+ break
442
+ audio_frames.append(sample)
443
+
444
+ if record_audio and self._audio_available:
445
+ try:
446
+ reachy_mini.media.start_recording()
447
+ audio_active = True
448
+ reported_rate = reachy_mini.media.get_input_audio_samplerate()
449
+ audio_samplerate = int(reported_rate) if reported_rate else AUDIO_SAMPLE_RATE
450
+ except Exception as exc: # pragma: no cover - runtime audio failure
451
+ self._set_state(
452
+ mode="error",
453
+ message=f"Audio init failed: {exc}",
454
+ active_move=None,
455
+ )
456
+ record_audio = False
457
+ audio_active = False
458
+
459
+ start = time.perf_counter()
460
+ # Update recording_started_at to the actual capture start (after audio init).
461
+ # Without this, the frontend progress bar is ahead by the audio init delay (~0.5s).
462
+ self._set_state(recording_started_at=time.time())
463
+ if on_capture_start is not None:
464
+ try:
465
+ on_capture_start()
466
+ except Exception as exc:
467
+ logger.warning("Capture start hook failed: %s", exc)
468
+
469
+ # Hard time guard: if an SDK call hangs mid-iteration, bail out
470
+ # after duration + generous margin rather than blocking forever.
471
+ hard_deadline = start + duration + 10.0
472
+
473
+ # Capture loop: read pose at MOTION_SAMPLE_RATE (100 Hz).
474
+ # We use perf_counter for timestamps (monotonic, high-resolution)
475
+ # while time.time() is used for server_time (wall clock for frontend sync).
476
+ sample_period = 1.0 / MOTION_SAMPLE_RATE
477
+ try:
478
+ while not stop_event.is_set() and not self._recording_cancel_event.is_set():
479
+ elapsed = time.perf_counter() - start
480
+ if elapsed >= duration:
481
+ break
482
+ if time.perf_counter() > hard_deadline:
483
+ logger.error("Capture loop exceeded hard deadline — aborting")
484
+ break
485
+ if record_motion:
486
+ try:
487
+ head_pose = reachy_mini.get_current_head_pose()
488
+ head_joints, antennas = reachy_mini.get_current_joint_positions()
489
+ except Exception as exc:
490
+ logger.warning("SDK call failed during capture: %s", exc)
491
+ break
492
+ timestamps.append(elapsed)
493
+ frames.append(
494
+ {
495
+ "head": np.asarray(head_pose, dtype=float).tolist(),
496
+ "antennas": np.asarray(antennas, dtype=float).tolist(),
497
+ "body_yaw": float(head_joints[0]) if head_joints else 0.0,
498
+ "check_collision": False,
499
+ }
500
+ )
501
+ else:
502
+ timestamps.append(elapsed)
503
+ stop_event.wait(sample_period)
504
+ _pull_audio_frames()
505
+ finally:
506
+ if audio_active:
507
+ stopped_ok = False
508
+ try:
509
+ stopped_ok = call_with_timeout(
510
+ reachy_mini.media.stop_recording, timeout=3.0
511
+ )
512
+ except Exception as exc:
513
+ logger.warning("stop_recording() failed: %s", exc)
514
+ if stopped_ok:
515
+ _pull_audio_frames()
516
+ else:
517
+ logger.warning(
518
+ "Skipping audio drain — stop_recording did not complete"
519
+ )
520
+ return timestamps, frames, audio_frames, audio_samplerate
521
+
522
+ def _stream_playback(
523
+ self,
524
+ reachy_mini: object,
525
+ move: RecordedMove,
526
+ sample_hook: Callable[[], None] | None = None,
527
+ start_signal: threading.Event | None = None,
528
+ ) -> bool:
529
+ """Stream a RecordedMove to the robot at 100 Hz.
530
+
531
+ Returns True if playback completed normally, False if cancelled.
532
+ Fires start_signal after the first motion command so audio can begin
533
+ in sync with the physical movement.
534
+ """
535
+ sleep_period = 1.0 / MOTION_SAMPLE_RATE
536
+ t0 = time.perf_counter()
537
+ signalled = False
538
+ while True:
539
+ if self._playback_cancel_event.is_set():
540
+ return False
541
+ elapsed = time.perf_counter() - t0
542
+ if elapsed >= move.duration:
543
+ break
544
+ t = min(max(elapsed, 0.0), max(move.duration - 1e-3, 0.0))
545
+ try:
546
+ head, antennas, body_yaw = move.evaluate(t)
547
+ reachy_mini.set_target_head_pose(head)
548
+ if body_yaw is not None:
549
+ reachy_mini.set_target_body_yaw(float(body_yaw))
550
+ if antennas is not None:
551
+ reachy_mini.set_target_antenna_joint_positions(list(antennas))
552
+ except Exception as exc:
553
+ logger.error("Playback command failed at t=%.2f: %s", t, exc)
554
+ break
555
+ if not signalled and start_signal is not None:
556
+ start_signal.set()
557
+ signalled = True
558
+ if sample_hook is not None:
559
+ sample_hook()
560
+
561
+ remaining = move.duration - elapsed
562
+ wait_time = max(0.001, min(sleep_period, remaining))
563
+ if self._playback_cancel_event.wait(wait_time):
564
+ return False
565
+
566
+ # Ensure start_signal fires even if the loop exited early, so
567
+ # the audio thread isn't left waiting on start_signal.wait().
568
+ if not signalled and start_signal is not None:
569
+ start_signal.set()
570
+ return True
571
+
572
+ def _apply_motion_model(self, move: RecordedMove) -> RecordedMove:
573
+ try:
574
+ return self._motion_model_registry.apply(move)
575
+ except Exception as exc:
576
+ logger.warning("Motion model %s failed: %s", self._motion_model_registry.active, exc)
577
+ return move
578
+
579
+ def _save_recording(
580
+ self,
581
+ request: RecordingRequest,
582
+ timestamps: list[float],
583
+ frames: list[dict[str, Any]],
584
+ audio_frames: list[np.ndarray],
585
+ audio_samplerate: int | None,
586
+ ) -> None:
587
+ json_path = self._dataset_dir / f"{request.move_id}.json"
588
+ wav_path = json_path.with_suffix(".wav")
589
+
590
+ data: dict[str, Any] = {
591
+ "description": request.description,
592
+ "time": timestamps,
593
+ "set_target_data": frames,
594
+ }
595
+ if not request.record_motion:
596
+ data["audio_only"] = True
597
+
598
+ json_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
599
+
600
+ # Handle audio: either copy uploaded file or save mic recording
601
+ if request.uploaded_audio_path and request.uploaded_audio_path.exists():
602
+ # Copy uploaded audio to move's wav path.
603
+ # Keep the temp file so the user can record again with the same upload.
604
+ shutil.copy2(str(request.uploaded_audio_path), str(wav_path))
605
+ elif request.record_audio and audio_frames and self._audio_available and audio_samplerate:
606
+ audio_data = np.concatenate(audio_frames, axis=0)
607
+ sf.write(str(wav_path), audio_data, audio_samplerate) # type: ignore[arg-type]
608
+
609
+ def _load_move(self, json_path: Path) -> RecordedMove:
610
+ move = json.loads(json_path.read_text(encoding="utf-8"))
611
+ sound_path = json_path.with_suffix(".wav")
612
+ return RecordedMove(move, sound_path if sound_path.exists() else None)
613
+
614
+ def _build_recording_request(
615
+ self, payload: StartRecordingPayload, uploaded_audio_path: Path | None = None
616
+ ) -> RecordingRequest:
617
+ stamp = datetime.utcnow().strftime("%Y%m%d-%H%M%S")
618
+ base_name = payload.label or f"take-{stamp}"
619
+ label = payload.label or f"Take {stamp}"
620
+ slug = _slugify(base_name)
621
+ move_id = slug
622
+ index = 1
623
+ while (self._dataset_dir / f"{move_id}.json").exists():
624
+ move_id = f"{slug}-{index}"
625
+ index += 1
626
+
627
+ description = payload.description or f"Recorded with Marionette on {stamp}"
628
+
629
+ return RecordingRequest(
630
+ move_id=move_id,
631
+ label=label,
632
+ description=description,
633
+ duration=float(payload.duration),
634
+ record_audio=bool(payload.record_audio),
635
+ record_motion=bool(payload.record_motion),
636
+ uploaded_audio_path=uploaded_audio_path,
637
+ )
638
+
639
+ def _refresh_recordings(self) -> None:
640
+ """Scan the active dataset's data/ dir and rebuild the in-memory recordings dict.
641
+
642
+ Called after any change that affects the moves list (record, delete,
643
+ dataset switch). The result is protected by _state_lock so the HTTP
644
+ thread always sees a consistent snapshot.
645
+ """
646
+ recordings: dict[str, RecordingMetadata] = {}
647
+ # Get uploaded move IDs for the active dataset
648
+ active_entry = self._datasets.get(self._active_dataset_id or "")
649
+ uploaded_ids = active_entry.uploaded_move_ids if active_entry else set()
650
+
651
+ for json_path in sorted(self._dataset_dir.glob("*.json")):
652
+ try:
653
+ move = json.loads(json_path.read_text(encoding="utf-8"))
654
+ timestamps = move.get("time", [])
655
+ duration = (
656
+ float(timestamps[-1]) - float(timestamps[0])
657
+ if len(timestamps) >= 2
658
+ else float(timestamps[0]) if timestamps else 0.0
659
+ )
660
+ description = move.get("description", "")
661
+ has_audio = json_path.with_suffix(".wav").exists()
662
+ move_id = json_path.stem
663
+ recordings[move_id] = RecordingMetadata(
664
+ move_id=move_id,
665
+ label=move_id,
666
+ description=description,
667
+ duration=duration,
668
+ created_at=json_path.stat().st_mtime,
669
+ json_path=json_path,
670
+ has_audio=has_audio,
671
+ is_uploaded=move_id in uploaded_ids,
672
+ audio_only=bool(move.get("audio_only", False)),
673
+ )
674
+ except Exception:
675
+ continue
676
+
677
+ with self._state_lock:
678
+ self._recordings = recordings
marionette/routes.py ADDED
@@ -0,0 +1,430 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """FastAPI route registration for the Marionette app.
2
+
3
+ All HTTP endpoints are defined here as closures that capture a reference
4
+ to the Marionette instance. This keeps routes thin — they validate input,
5
+ delegate to the appropriate method, and return the result.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import logging
10
+ import platform
11
+ import uuid
12
+ from pathlib import Path
13
+ from typing import TYPE_CHECKING, Any
14
+
15
+ from fastapi import HTTPException, UploadFile, File
16
+
17
+ from marionette.audio import get_audio_duration
18
+ from marionette.models import (
19
+ DATASET_DATA_SUBDIR,
20
+ CreateDatasetPayload,
21
+ DownloadDatasetPayload,
22
+ HfTokenPayload,
23
+ PlayMovePayload,
24
+ RobotAudioSelectPayload,
25
+ SelectDatasetPayload,
26
+ StartRecordingPayload,
27
+ SyncDatasetPayload,
28
+ UpdateDatasetRootPayload,
29
+ UpdateExperimentsPayload,
30
+ UpdateLeadCompensationPayload,
31
+ )
32
+
33
+ try:
34
+ import soundfile as sf
35
+ except Exception: # pragma: no cover
36
+ sf = None
37
+
38
+ if TYPE_CHECKING:
39
+ from marionette.app import Marionette
40
+
41
+ logger = logging.getLogger(__name__)
42
+
43
+ # Shared temp directory for uploaded and downloaded audio files.
44
+ _TEMP_UPLOADS_DIR = Path(__file__).resolve().parent.parent / "temp_uploads"
45
+
46
+
47
+ def register_routes(app: Marionette, *, version: str) -> None:
48
+ """Register all FastAPI endpoints on *app*.settings_app."""
49
+ # Import the datasets MODULE (not individual names) so that references
50
+ # like _ds.hf_login are resolved at call time. This is critical for
51
+ # testing: tests monkeypatch marionette.datasets.hf_login, and if we
52
+ # had imported `from marionette.datasets import hf_login`, the closure
53
+ # would capture the original function, not the monkeypatched one.
54
+ import marionette.datasets as _ds
55
+
56
+ assert app.settings_app is not None
57
+
58
+ @app.settings_app.get("/api/state")
59
+ def get_state() -> dict[str, Any]:
60
+ return app._serialize_state()
61
+
62
+ @app.settings_app.get("/api/version")
63
+ def get_version() -> dict[str, str]:
64
+ return {"version": version, "platform": platform.system(), "hostname": platform.node()}
65
+
66
+ # Dummy endpoint to silence 404 spam from external tools
67
+ @app.settings_app.get("/sensor_data")
68
+ def sensor_data() -> dict[str, Any]:
69
+ return {}
70
+
71
+ @app.settings_app.post("/api/upload-audio")
72
+ async def upload_audio(file: UploadFile = File(...)) -> dict[str, Any]:
73
+ if not file.filename:
74
+ raise HTTPException(status_code=400, detail="No file provided.")
75
+ ext = Path(file.filename).suffix.lower()
76
+ if ext not in {".wav", ".mp3"}:
77
+ raise HTTPException(
78
+ status_code=400,
79
+ detail=f"Unsupported audio format '{ext}'. Use .wav or .mp3.",
80
+ )
81
+ upload_id = str(uuid.uuid4())
82
+ temp_dir = _TEMP_UPLOADS_DIR
83
+ temp_dir.mkdir(parents=True, exist_ok=True)
84
+ temp_path = temp_dir / f"{upload_id}{ext}"
85
+ try:
86
+ content = await file.read()
87
+ temp_path.write_bytes(content)
88
+ except Exception as exc:
89
+ raise HTTPException(status_code=500, detail=f"Failed to save upload: {exc}") from exc
90
+
91
+ # Convert MP3 to WAV for better playback quality
92
+ duration: float | None = None
93
+ if ext == ".mp3" and sf is not None:
94
+ try:
95
+ data, samplerate = sf.read(str(temp_path))
96
+ wav_path = temp_dir / f"{upload_id}.wav"
97
+ # Resample to 48kHz for better quality (common output rate)
98
+ target_rate = 48000
99
+ if samplerate != target_rate:
100
+ try:
101
+ from scipy import signal
102
+ original_rate = samplerate
103
+ num_samples = int(len(data) * target_rate / samplerate)
104
+ if data.ndim == 1:
105
+ data = signal.resample(data, num_samples)
106
+ else:
107
+ data = signal.resample(data, num_samples, axis=0)
108
+ samplerate = target_rate
109
+ logger.info("Resampled audio from %d to %d Hz", original_rate, target_rate)
110
+ except ImportError:
111
+ logger.warning("scipy not available, skipping resample")
112
+ # Write as 16-bit PCM WAV for maximum compatibility
113
+ sf.write(str(wav_path), data, samplerate, subtype='PCM_16')
114
+ duration = len(data) / samplerate
115
+ # Remove original MP3 and use WAV
116
+ temp_path.unlink()
117
+ temp_path = wav_path
118
+ logger.info("Converted MP3 to WAV: %s", wav_path.name)
119
+ except Exception as exc:
120
+ logger.warning("MP3 to WAV conversion failed, using original: %s", exc)
121
+
122
+ # Get audio duration if not already determined
123
+ if duration is None:
124
+ duration = get_audio_duration(temp_path, fallback=None)
125
+
126
+ app._uploaded_audio[upload_id] = temp_path
127
+ return {"upload_id": upload_id, "filename": file.filename, "duration": duration}
128
+
129
+ @app.settings_app.post("/api/record")
130
+ def start_recording(payload: StartRecordingPayload) -> dict[str, Any]:
131
+ uploaded_audio_path: Path | None = None
132
+ if payload.uploaded_audio_id:
133
+ uploaded_audio_path = app._uploaded_audio.get(payload.uploaded_audio_id)
134
+ if not uploaded_audio_path or not uploaded_audio_path.exists():
135
+ raise HTTPException(
136
+ status_code=400,
137
+ detail="Uploaded audio file not found. Please re-upload.",
138
+ )
139
+ elif payload.record_audio and not app._audio_available:
140
+ raise HTTPException(
141
+ status_code=400,
142
+ detail="Audio capture backend unavailable.",
143
+ )
144
+
145
+ active_entry = app._datasets.get(app._active_dataset_id or "")
146
+ if active_entry and active_entry.origin == "downloaded":
147
+ raise HTTPException(
148
+ status_code=409,
149
+ detail="Cannot record into a downloaded dataset. Switch to a local dataset or create a new one.",
150
+ )
151
+
152
+ request = app._build_recording_request(payload, uploaded_audio_path)
153
+
154
+ with app._state_lock:
155
+ if (
156
+ app._mode not in {"idle", "queued"}
157
+ or app._pending_recording
158
+ or app._pending_playback
159
+ ):
160
+ raise HTTPException(status_code=409, detail="Robot is busy.")
161
+ app._preferred_duration = float(request.duration)
162
+ app._pending_recording = request
163
+ app._mode = "queued"
164
+ app._message = "Recording scheduled"
165
+ app._countdown_ends_at = None
166
+ app._active_record_started_at = None
167
+ app._active_record_duration = None
168
+ # Persist outside the lock — never do I/O while holding _state_lock
169
+ app._save_dataset_registry()
170
+
171
+ return {
172
+ "accepted": True,
173
+ "move_id": request.move_id,
174
+ "label": request.label,
175
+ }
176
+
177
+ @app.settings_app.post("/api/play")
178
+ def play_move(payload: PlayMovePayload) -> dict[str, Any]:
179
+ if payload.move_id not in app._recordings:
180
+ raise HTTPException(status_code=404, detail="Move not found.")
181
+
182
+ with app._state_lock:
183
+ if (
184
+ app._mode != "idle"
185
+ or app._pending_recording
186
+ or app._pending_playback
187
+ ):
188
+ raise HTTPException(status_code=409, detail="Robot is busy.")
189
+ app._pending_playback = payload.move_id
190
+ app._mode = "queued"
191
+ app._message = f"Playback queued for {payload.move_id}"
192
+
193
+ return {"accepted": True, "move_id": payload.move_id}
194
+
195
+ @app.settings_app.post("/api/play/stop")
196
+ def stop_playback() -> dict[str, Any]:
197
+ with app._state_lock:
198
+ if app._mode == "queued" and app._pending_playback is not None:
199
+ app._pending_playback = None
200
+ app._mode = "idle"
201
+ app._message = "Playback cancelled"
202
+ return {"stopped": True}
203
+ if app._mode != "playing":
204
+ return {"stopped": False, "reason": "not_playing"}
205
+ app._playback_cancel_event.set()
206
+ return {"stopped": True}
207
+
208
+ @app.settings_app.post("/api/record/stop")
209
+ def stop_recording() -> dict[str, Any]:
210
+ with app._state_lock:
211
+ if app._mode not in {"recording", "countdown", "queued", "preparing"}:
212
+ return {"stopped": False, "reason": "not_recording"}
213
+ if app._mode == "queued":
214
+ app._pending_recording = None
215
+ app._mode = "idle"
216
+ app._message = "Recording cancelled"
217
+ return {"stopped": True}
218
+ app._recording_cancel_event.set()
219
+ return {"stopped": True}
220
+
221
+ @app.settings_app.delete("/api/moves/{move_id}")
222
+ def delete_move(move_id: str) -> dict[str, Any]:
223
+ app._delete_move_files(move_id)
224
+ app._refresh_recordings()
225
+ return {"status": "deleted", "move_id": move_id}
226
+
227
+ @app.settings_app.get("/api/datasets")
228
+ def list_datasets() -> dict[str, Any]:
229
+ return app._datasets_payload()
230
+
231
+ @app.settings_app.post("/api/datasets")
232
+ def create_dataset(payload: CreateDatasetPayload) -> dict[str, Any]:
233
+ with app._state_lock:
234
+ if (
235
+ app._mode not in {"idle", "queued"}
236
+ or app._pending_recording
237
+ or app._pending_playback
238
+ ):
239
+ raise HTTPException(status_code=409, detail="Robot is busy.")
240
+ entry = app._create_dataset(payload.name, payload.label)
241
+ app._select_dataset(entry.dataset_id)
242
+ app._refresh_recordings()
243
+ return {"status": "created", "dataset": entry.to_payload()}
244
+
245
+ @app.settings_app.post("/api/datasets/select")
246
+ def select_dataset(payload: SelectDatasetPayload) -> dict[str, Any]:
247
+ with app._state_lock:
248
+ if (
249
+ app._mode not in {"idle", "queued"}
250
+ or app._pending_recording
251
+ or app._pending_playback
252
+ ):
253
+ raise HTTPException(status_code=409, detail="Robot is busy.")
254
+ app._select_dataset(payload.dataset_id)
255
+ app._refresh_recordings()
256
+ return {"status": "selected", "active_id": payload.dataset_id}
257
+
258
+ @app.settings_app.post("/api/datasets/sync")
259
+ def sync_dataset(payload: SyncDatasetPayload) -> dict[str, Any]:
260
+ with app._state_lock:
261
+ if (
262
+ app._mode not in {"idle", "queued"}
263
+ or app._pending_recording
264
+ or app._pending_playback
265
+ ):
266
+ raise HTTPException(status_code=409, detail="Robot is busy.")
267
+ username = payload.hf_username or app._check_hf_login()
268
+ if not username:
269
+ raise HTTPException(
270
+ status_code=400,
271
+ detail="No Hugging Face username provided and not logged in via CLI.",
272
+ )
273
+ result = app._sync_dataset(username, payload.move_ids, payload.dataset_slug)
274
+ return result
275
+
276
+ @app.settings_app.post("/api/datasets/root")
277
+ def update_dataset_root(payload: UpdateDatasetRootPayload) -> dict[str, Any]:
278
+ with app._state_lock:
279
+ if (
280
+ app._mode not in {"idle", "queued"}
281
+ or app._pending_recording
282
+ or app._pending_playback
283
+ ):
284
+ raise HTTPException(status_code=409, detail="Robot is busy.")
285
+ app._set_dataset_root(payload.path)
286
+ app._refresh_recordings()
287
+ return {"status": "updated", "root_path": str(app._dataset_root)}
288
+
289
+ @app.settings_app.get("/api/datasets/community")
290
+ def community_datasets() -> dict[str, Any]:
291
+ datasets = app._list_community_datasets()
292
+ return {"datasets": datasets}
293
+
294
+ @app.settings_app.post("/api/datasets/download")
295
+ def download_dataset(payload: DownloadDatasetPayload) -> dict[str, Any]:
296
+ with app._state_lock:
297
+ if app._mode not in {"idle", "queued"} or app._pending_recording or app._pending_playback:
298
+ raise HTTPException(status_code=409, detail="Robot is busy.")
299
+ entry = app._download_community_dataset(payload)
300
+ app._refresh_recordings()
301
+ return {"status": "downloaded", "dataset": entry.to_payload()}
302
+
303
+ @app.settings_app.post("/api/experiments")
304
+ def update_experiments(payload: UpdateExperimentsPayload) -> dict[str, Any]:
305
+ updates = payload.model_dump(exclude_none=True)
306
+ if not updates:
307
+ return {"status": "unchanged"}
308
+ with app._state_lock:
309
+ for key, value in updates.items():
310
+ if key == "duration_seconds":
311
+ app._preferred_duration = float(value)
312
+ elif key == "welcome_messages":
313
+ app._welcome_messages = max(0, min(2, int(value)))
314
+ app._save_dataset_registry()
315
+ return {
316
+ "status": "updated",
317
+ "preferred_duration": app._preferred_duration,
318
+ "motion_models": app._motion_model_registry.to_payload(),
319
+ }
320
+
321
+ @app.settings_app.post("/api/motion-model/lead")
322
+ def update_motion_model_lead(payload: UpdateLeadCompensationPayload) -> dict[str, Any]:
323
+ params = payload.model_dump(exclude_none=True)
324
+ if not params:
325
+ return {
326
+ "status": "unchanged",
327
+ "active": app._motion_model_registry.active,
328
+ "params": app._motion_model_registry.get_model_params("lead_compensation"),
329
+ }
330
+ try:
331
+ app._motion_model_registry.set_model_params("lead_compensation", params)
332
+ except KeyError as exc:
333
+ raise HTTPException(status_code=404, detail="Lead compensation model unavailable.") from exc
334
+ app._save_dataset_registry()
335
+ return {
336
+ "status": "updated",
337
+ "active": app._motion_model_registry.active,
338
+ "params": app._motion_model_registry.get_model_params("lead_compensation"),
339
+ }
340
+
341
+ @app.settings_app.post("/api/hf-auth/save-token")
342
+ def save_hf_token(payload: HfTokenPayload) -> dict[str, Any]:
343
+ if _ds.hf_login is None:
344
+ raise HTTPException(status_code=500, detail="huggingface_hub is not installed.")
345
+ token = payload.token.strip()
346
+ if not token.startswith("hf_"):
347
+ raise HTTPException(status_code=422, detail="Token must start with 'hf_'.")
348
+ try:
349
+ _ds.hf_login(token=token, add_to_git_credential=False)
350
+ except Exception as exc:
351
+ raise HTTPException(status_code=401, detail=f"Invalid token: {exc}") from exc
352
+ # Verify the token works
353
+ app._hf_checked = False
354
+ username = app._check_hf_login()
355
+ if not username:
356
+ raise HTTPException(status_code=401, detail="Token saved but could not verify identity.")
357
+ return {"status": "logged_in", "username": username}
358
+
359
+ @app.settings_app.delete("/api/hf-auth/token")
360
+ def delete_hf_token() -> dict[str, Any]:
361
+ if _ds.hf_logout is None:
362
+ raise HTTPException(status_code=500, detail="huggingface_hub is not installed.")
363
+ try:
364
+ _ds.hf_logout()
365
+ except Exception as exc:
366
+ logger.warning("HF logout error: %s", exc)
367
+ app._hf_username = None
368
+ app._hf_checked = False
369
+ return {"status": "logged_out"}
370
+
371
+ # ── Robot audio file listing ──────────────────────────────────
372
+
373
+ @app.settings_app.get("/api/robot-audio")
374
+ def list_robot_audio() -> dict[str, Any]:
375
+ """List WAV files from audio-only recordings (Record Audio button)."""
376
+ import json as _json
377
+
378
+ files: list[dict[str, Any]] = []
379
+ seen: set[str] = set()
380
+
381
+ for entry in app._datasets.values():
382
+ data_dir = entry.path / DATASET_DATA_SUBDIR
383
+ if not data_dir.is_dir():
384
+ continue
385
+ for json_path in data_dir.glob("*.json"):
386
+ try:
387
+ meta = _json.loads(json_path.read_text(encoding="utf-8"))
388
+ except Exception:
389
+ continue
390
+ if not meta.get("audio_only", False):
391
+ continue
392
+ wav_path = json_path.with_suffix(".wav")
393
+ if not wav_path.exists():
394
+ continue
395
+ real = str(wav_path.resolve())
396
+ if real not in seen:
397
+ seen.add(real)
398
+ files.append({
399
+ "name": wav_path.stem,
400
+ "path": real,
401
+ "duration_seconds": get_audio_duration(wav_path, fallback=None),
402
+ })
403
+
404
+ # Sort newest first so auto-select picks the latest recording
405
+ files.sort(key=lambda f: Path(f["path"]).stat().st_mtime, reverse=True)
406
+ return {"files": files}
407
+
408
+ @app.settings_app.post("/api/robot-audio/select")
409
+ def select_robot_audio(payload: RobotAudioSelectPayload) -> dict[str, Any]:
410
+ """Register a robot-side WAV file for use as recording audio."""
411
+ path = Path(payload.path).resolve()
412
+
413
+ # Security: only allow files within dataset dirs or temp_uploads
414
+ allowed_roots: list[Path] = [_TEMP_UPLOADS_DIR]
415
+ for entry in app._datasets.values():
416
+ allowed_roots.append(entry.path.resolve())
417
+
418
+ if not any(path == root or root in path.parents for root in allowed_roots):
419
+ raise HTTPException(
420
+ status_code=403,
421
+ detail="Access denied: file is outside allowed directories.",
422
+ )
423
+ if not path.exists() or not path.is_file():
424
+ raise HTTPException(status_code=404, detail="File not found.")
425
+ if path.suffix.lower() != ".wav":
426
+ raise HTTPException(status_code=400, detail="Only WAV files are supported.")
427
+
428
+ upload_id = str(uuid.uuid4())
429
+ app._uploaded_audio[upload_id] = path
430
+ return {"upload_id": upload_id, "filename": path.name, "duration": get_audio_duration(path, fallback=None)}
marionette/state.py ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """State management mixin for the Marionette app.
2
+
3
+ Provides thread-safe state serialization and mutation methods.
4
+ The _state_lock (a threading.Lock) is defined on the Marionette instance;
5
+ this mixin supplies the methods that read/write the protected fields.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import time
10
+ from typing import Any
11
+
12
+ from marionette.models import COUNTDOWN_SECONDS, DEFAULT_DURATION, MOTION_SAMPLE_RATE
13
+
14
+
15
+ class StateMixin:
16
+ """Mixin providing state serialization and mutation helpers.
17
+
18
+ Expects the host class to have: _state_lock, _mode, _message,
19
+ _active_move, _countdown_ends_at, _active_record_started_at,
20
+ _active_record_duration, _recordings, _recording_stats,
21
+ _pending_recording, _pending_playback, _preferred_duration,
22
+ _audio_available, _dataset_dir, _dataset_root, _welcome_messages,
23
+ _motion_model_registry, _check_hf_login(), _datasets_payload().
24
+ """
25
+
26
+ # Sentinel value to distinguish "not passed" from None in _set_state().
27
+ # We can't use None as default because None is a valid value for
28
+ # active_move, countdown_ends_at, etc.
29
+ _UNSET = object()
30
+
31
+ def _serialize_state(self) -> dict[str, Any]:
32
+ # Resolve HF username BEFORE acquiring the lock — _check_hf_login()
33
+ # may call hf_whoami() (a network round-trip) on first invocation,
34
+ # and we must never hold the lock during I/O.
35
+ hf_username = self._check_hf_login()
36
+
37
+ with self._state_lock:
38
+ moves = sorted(
39
+ (meta.to_payload() for meta in self._recordings.values()),
40
+ key=lambda entry: entry["created_at"],
41
+ reverse=True,
42
+ )
43
+ # Unified timing sent to the frontend for overlay animations.
44
+ # The frontend uses (phase_start_at, phase_end_at) + clock offset
45
+ # to render a smooth countdown or recording progress bar.
46
+ phase_start_at: float | None = None
47
+ phase_end_at: float | None = None
48
+ if self._mode == "countdown" and self._countdown_ends_at is not None:
49
+ phase_start_at = self._countdown_ends_at - COUNTDOWN_SECONDS
50
+ phase_end_at = self._countdown_ends_at
51
+ elif self._mode == "recording" and self._active_record_started_at is not None:
52
+ phase_start_at = self._active_record_started_at
53
+ if self._active_record_duration is not None:
54
+ phase_end_at = self._active_record_started_at + self._active_record_duration
55
+
56
+ return {
57
+ "server_time": time.time(),
58
+ "mode": self._mode,
59
+ "message": self._message,
60
+ "active_move": self._active_move,
61
+ "phase_start_at": phase_start_at,
62
+ "phase_end_at": phase_end_at,
63
+ "countdown_ends_at": self._countdown_ends_at,
64
+ "recording_started_at": self._active_record_started_at,
65
+ "recording_duration": self._active_record_duration,
66
+ "recording_stats": self._recording_stats,
67
+ "pending_recording": self._pending_recording.label if self._pending_recording else None,
68
+ "pending_playback": self._pending_playback,
69
+ "moves": moves,
70
+ "config": {
71
+ "default_duration": DEFAULT_DURATION,
72
+ "preferred_duration": self._preferred_duration,
73
+ "countdown_seconds": COUNTDOWN_SECONDS,
74
+ "motion_sample_rate": MOTION_SAMPLE_RATE,
75
+ "audio_available": self._audio_available,
76
+ "active_dataset_path": str(self._dataset_dir),
77
+ "dataset_root_path": str(self._dataset_root),
78
+ "hf_username": hf_username,
79
+ "motion_models": self._motion_model_registry.to_payload(),
80
+ "welcome_messages": self._welcome_messages,
81
+ },
82
+ "datasets": self._datasets_payload(),
83
+ }
84
+
85
+ def _set_state(
86
+ self,
87
+ *,
88
+ mode: str | None = None,
89
+ message: str | None = None,
90
+ active_move: object = _UNSET,
91
+ countdown_ends_at: object = _UNSET,
92
+ recording_started_at: object = _UNSET,
93
+ recording_duration: object = _UNSET,
94
+ ) -> None:
95
+ with self._state_lock:
96
+ if mode is not None:
97
+ self._mode = mode
98
+ if message is not None:
99
+ self._message = message
100
+ if active_move is not self._UNSET:
101
+ self._active_move = active_move
102
+ if countdown_ends_at is not self._UNSET:
103
+ self._countdown_ends_at = countdown_ends_at
104
+ if recording_started_at is not self._UNSET:
105
+ self._active_record_started_at = recording_started_at
106
+ if recording_duration is not self._UNSET:
107
+ self._active_record_duration = recording_duration
108
+
109
+ def _set_idle_state(self) -> None:
110
+ self._set_state(
111
+ mode="idle",
112
+ message="Ready to capture moves",
113
+ active_move=None,
114
+ countdown_ends_at=None,
115
+ recording_started_at=None,
116
+ recording_duration=None,
117
+ )
marionette/static/index.html CHANGED
@@ -26,17 +26,70 @@
26
  </div>
27
  </header>
28
 
29
- <!-- ════════ RECORD HERO ════════ -->
30
  <section class="record-hero fade-in fade-in-1" id="record-hero">
31
  <p class="record-hero-tagline">
32
  Move Reachy Mini's head with your hands to create animated movements with sound
33
  </p>
34
- <div class="record-ring">
35
- <button class="record-btn" id="record-btn" title="Start Recording">
36
- <span class="rec-icon"></span>
37
- <span class="rec-label">Record</span>
38
- </button>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  </div>
 
 
40
  <div class="record-fields">
41
  <div class="record-field">
42
  <label for="rec-name">Name</label>
@@ -47,6 +100,7 @@
47
  <input type="number" id="rec-duration" min="0.5" max="300" step="any" placeholder="5.0"/>
48
  </div>
49
  </div>
 
50
  <p class="record-hint">3 seconds to prepare before recording starts</p>
51
  <p class="downloaded-warning" id="downloaded-warning">
52
  Switch to a local dataset to record new moves
@@ -55,14 +109,17 @@
55
 
56
  <!-- ════════ SECTION TABS ════════ -->
57
  <nav class="section-tabs fade-in fade-in-2">
58
- <button class="section-tab active" data-tab="moves">
59
- Moves <span class="tab-count" id="moves-count">0</span>
 
 
 
60
  </button>
61
  <button class="section-tab" data-tab="community">Community</button>
62
  </nav>
63
 
64
- <!-- ════════ MOVES TAB ════════ -->
65
- <div class="tab-panel active" id="tab-moves">
66
  <div class="moves-toolbar">
67
  <div class="moves-toolbar-left">
68
  <div class="dataset-bar">
@@ -89,12 +146,26 @@
89
  </div>
90
  </div>
91
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
  <!-- ════════ COMMUNITY TAB ════════ -->
93
  <div class="tab-panel" id="tab-community">
94
  <div class="community-toolbar">
95
- <button class="community-fetch-btn" id="fetch-community-btn">
96
- Fetch community datasets
97
- </button>
98
  <button class="community-download-btn" id="download-community-btn" disabled>
99
  Download selected
100
  </button>
@@ -122,32 +193,6 @@
122
  <button class="settings-close" id="settings-close">&times;</button>
123
  </div>
124
 
125
- <!-- Audio source: mic (default), upload file, or silent -->
126
- <div class="settings-section">
127
- <h3>Audio Source</h3>
128
- <div class="settings-radio-group">
129
- <label class="settings-radio">
130
- <input type="radio" name="audio-src" value="mic" checked/> Mic
131
- </label>
132
- <label class="settings-radio">
133
- <input type="radio" name="audio-src" value="upload"/> Upload
134
- </label>
135
- <label class="settings-radio">
136
- <input type="radio" name="audio-src" value="none"/> Silent
137
- </label>
138
- <label class="settings-radio">
139
- <input type="radio" name="audio-src" value="mic-only"/> Mic only (no motion)
140
- </label>
141
- </div>
142
- <div id="audio-upload-section" style="display:none">
143
- <div class="audio-upload-zone" id="audio-upload-zone">
144
- Drop a WAV or MP3 here, or click to browse
145
- <input type="file" id="audio-file-input" accept=".wav,.mp3,audio/wav,audio/mpeg"/>
146
- </div>
147
- <p class="audio-upload-status" id="audio-upload-status"></p>
148
- </div>
149
- </div>
150
-
151
  <!-- Hugging Face login -->
152
  <div class="settings-section">
153
  <h3>Hugging Face</h3>
 
26
  </div>
27
  </header>
28
 
29
+ <!-- ════════ RECORD HERO — Two buttons: Record Move + Record Audio ════════ -->
30
  <section class="record-hero fade-in fade-in-1" id="record-hero">
31
  <p class="record-hero-tagline">
32
  Move Reachy Mini's head with your hands to create animated movements with sound
33
  </p>
34
+
35
+ <div class="record-actions">
36
+ <!-- PRIMARY: Record a motion (with optional audio source) -->
37
+ <div class="record-action-block">
38
+ <div class="record-ring">
39
+ <button class="record-btn" id="record-btn" title="Record Move">
40
+ <span class="rec-icon"></span>
41
+ <span class="rec-label">Record Move</span>
42
+ </button>
43
+ </div>
44
+
45
+ <!-- Audio source: inline below Record Move button -->
46
+ <div class="audio-source-panel" id="audio-source-panel">
47
+ <span class="audio-source-label">Audio source</span>
48
+ <p class="audio-source-hint">Plays during recording so you can sync your movements</p>
49
+ <div class="audio-source-tabs">
50
+ <label class="audio-tab">
51
+ <input type="radio" name="audio-src" value="silent" checked/> Silent
52
+ </label>
53
+ <label class="audio-tab">
54
+ <input type="radio" name="audio-src" value="upload"/> Upload
55
+ </label>
56
+ <label class="audio-tab">
57
+ <input type="radio" name="audio-src" value="robot"/> Robot file
58
+ </label>
59
+ </div>
60
+
61
+ <!-- Upload zone (visible when 'upload' is selected) -->
62
+ <div id="audio-upload-section" style="display:none">
63
+ <div class="audio-upload-zone" id="audio-upload-zone">
64
+ Drop a WAV or MP3 here, or click to browse
65
+ <input type="file" id="audio-file-input" accept=".wav,.mp3,audio/wav,audio/mpeg"/>
66
+ </div>
67
+ <p class="audio-upload-status" id="audio-upload-status"></p>
68
+ </div>
69
+
70
+ <!-- Robot file (visible when 'robot' is selected) -->
71
+ <div id="robot-audio-section" style="display:none">
72
+ <select class="dataset-select" id="robot-audio-select">
73
+ <option value="">Select audio file...</option>
74
+ </select>
75
+ </div>
76
+ </div>
77
+ </div>
78
+
79
+ <!-- SECONDARY: Record audio only (mic, no motion) -->
80
+ <div class="record-action-block">
81
+ <div class="record-ring record-ring-audio">
82
+ <button class="record-btn record-btn-audio" id="record-audio-btn" title="Record Audio">
83
+ <span class="rec-icon-audio">&#127908;</span>
84
+ <span class="rec-label">Record Audio</span>
85
+ </button>
86
+ </div>
87
+ <p class="record-hint-small">Mic only, no motion</p>
88
+ <p class="record-audio-hint">Audio recordings appear in your moves list and under &ldquo;Robot file&rdquo; for use as a soundtrack during motion recording.</p>
89
+ </div>
90
  </div>
91
+
92
+ <!-- Shared name + duration fields -->
93
  <div class="record-fields">
94
  <div class="record-field">
95
  <label for="rec-name">Name</label>
 
100
  <input type="number" id="rec-duration" min="0.5" max="300" step="any" placeholder="5.0"/>
101
  </div>
102
  </div>
103
+ <p class="audio-selection-status" id="audio-selection-status">Audio: Silent</p>
104
  <p class="record-hint">3 seconds to prepare before recording starts</p>
105
  <p class="downloaded-warning" id="downloaded-warning">
106
  Switch to a local dataset to record new moves
 
109
 
110
  <!-- ════════ SECTION TABS ════════ -->
111
  <nav class="section-tabs fade-in fade-in-2">
112
+ <button class="section-tab active" data-tab="my-moves">
113
+ My Moves <span class="tab-count" id="my-moves-count">0</span>
114
+ </button>
115
+ <button class="section-tab" data-tab="library">
116
+ Library <span class="tab-count" id="library-count">0</span>
117
  </button>
118
  <button class="section-tab" data-tab="community">Community</button>
119
  </nav>
120
 
121
+ <!-- ════════ MY MOVES TAB ════════ -->
122
+ <div class="tab-panel active" id="tab-my-moves">
123
  <div class="moves-toolbar">
124
  <div class="moves-toolbar-left">
125
  <div class="dataset-bar">
 
146
  </div>
147
  </div>
148
 
149
+ <!-- ════════ LIBRARY TAB ════════ -->
150
+ <div class="tab-panel" id="tab-library">
151
+ <div class="moves-toolbar">
152
+ <div class="moves-toolbar-left">
153
+ <div class="dataset-bar">
154
+ <select class="dataset-select" id="library-dataset-select"></select>
155
+ </div>
156
+ </div>
157
+ </div>
158
+ <div id="library-moves-list">
159
+ <div class="moves-empty">
160
+ <div class="moves-empty-icon">&#11088;</div>
161
+ Download datasets from the Community tab to see them here
162
+ </div>
163
+ </div>
164
+ </div>
165
+
166
  <!-- ════════ COMMUNITY TAB ════════ -->
167
  <div class="tab-panel" id="tab-community">
168
  <div class="community-toolbar">
 
 
 
169
  <button class="community-download-btn" id="download-community-btn" disabled>
170
  Download selected
171
  </button>
 
193
  <button class="settings-close" id="settings-close">&times;</button>
194
  </div>
195
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
196
  <!-- Hugging Face login -->
197
  <div class="settings-section">
198
  <h3>Hugging Face</h3>
marionette/static/main.js CHANGED
@@ -1,21 +1,26 @@
1
  /* ═══════════════════════════════════════════════════════════════════════
2
  MARIONETTE FRONTEND — JavaScript
3
 
4
- Architecture overview:
5
- ────────────────────
6
- 1. STATE POLLING: fetchState() runs every 1500ms (200ms during active phases),
7
- calls GET /api/state, and passes the response to updateUI().
8
-
9
- 2. UI UPDATE: updateUI() is the single function that synchronizes all
10
- DOM elements with the backend state. It's called after every poll
11
- and after every user action.
 
 
 
 
12
 
13
  3. PHASE ANIMATION: A requestAnimationFrame loop continuously updates
14
- the countdown/recording overlay based on phase_start_at / phase_end_at
15
- timestamps from the backend.
16
 
17
- 4. EVENT HANDLERS: All user interactions (record, play, delete, etc.)
18
- send the appropriate API call and then trigger a state refresh.
 
19
  ═══════════════════════════════════════════════════════════════════════ */
20
 
21
 
@@ -23,119 +28,129 @@
23
  // SECTION 1: APP STATE & ELEMENT REFERENCES
24
  // ═══════════════════════════════════════════════════════════════════════
25
 
26
- let lastState = null; // Last received state from backend
27
- let busy = false; // True when mode is not idle/queued
28
- let clockOffset = 0; // Diff between server clock and local clock (seconds, NTP-style)
29
- let clockOffsetSamples = 0; // Number of clock offset measurements taken
30
- let bestRtt = Infinity; // Lowest RTT seen (best clock sync sample)
31
- let phaseMode = 'idle'; // Current phase for overlay: 'idle', 'countdown', 'recording'
32
- let phaseStartAt = null; // Server timestamp: when current phase started
33
- let phaseEndAt = null; // Server timestamp: when current phase ends
34
- let countdownEndAt = null; // Server timestamp: when countdown ends (= recording starts)
35
- let recordingEndAt = null; // Server timestamp: when recording ends
36
- let requestedDuration = null; // Duration the user requested (seconds), for predicting recording end
37
- let selectedMoves = new Set(); // Move IDs selected for HF upload
38
- let communitySelection = new Set(); // Community dataset repo_ids selected for download
39
- let uploadedAudioId = null; // ID of uploaded audio file (from /api/upload-audio)
40
- let uploadedAudioFilename = null; // Original filename of uploaded audio
41
- let stopping = false; // True while a stop request is in flight (debounce)
42
- let datasetSwitching = false; // True while a dataset select is in flight (prevents dropdown snap-back)
43
- let pollingHandle = null; // setTimeout handle for adaptive state polling
44
- let stateSeq = 0; // Monotonic counter to ignore out-of-order responses
45
- let latestSeq = 0; // Highest seq we've processed
46
- let hfUsername = null; // Auto-detected HF username (from backend)
 
 
 
 
 
 
 
 
47
 
48
  const STORAGE_KEYS = {
49
  moveLabel: 'marionette.last_move_label',
50
  };
51
 
52
- /* DOM element references — gathered once at startup */
53
- const $modeBadge = document.getElementById('mode-badge');
54
- const $recordBtn = document.getElementById('record-btn');
55
- const $recName = document.getElementById('rec-name');
56
- const $recDuration = document.getElementById('rec-duration');
 
 
57
  const $downloadedWarning = document.getElementById('downloaded-warning');
58
- const $movesList = document.getElementById('moves-list');
59
- const $movesCount = document.getElementById('moves-count');
60
- const $uploadBar = document.getElementById('upload-bar');
61
- const $uploadCount = document.getElementById('upload-count');
62
- const $uploadBtn = document.getElementById('upload-btn');
63
- const $datasetSelect = document.getElementById('dataset-select');
64
- const $newDatasetBtn = document.getElementById('new-dataset-btn');
 
 
 
65
  const $newDatasetInline = document.getElementById('new-dataset-inline');
66
- const $newDatasetInput = document.getElementById('new-dataset-input');
67
  const $createDatasetBtn = document.getElementById('create-dataset-btn');
68
  const $cancelDatasetBtn = document.getElementById('cancel-dataset-btn');
69
- const $settingsBtn = document.getElementById('settings-btn');
70
  const $settingsBackdrop = document.getElementById('settings-backdrop');
71
- const $settingsDrawer = document.getElementById('settings-drawer');
72
- const $settingsClose = document.getElementById('settings-close');
73
- const $phaseOverlay = document.getElementById('phase-overlay');
74
- const $phaseNumber = document.getElementById('phase-number');
75
- const $phaseSublabel = document.getElementById('phase-sublabel');
76
- const $phaseFill = document.getElementById('phase-fill');
77
- const $phaseStopBtn = document.getElementById('phase-stop-btn');
78
- const $hfStatus = document.getElementById('hf-status');
79
- const $hfHint = document.getElementById('hf-hint');
80
- const $hfLoginForm = document.getElementById('hf-login-form');
81
- const $hfTokenInput = document.getElementById('hf-token-input');
82
- const $hfLoginBtn = document.getElementById('hf-login-btn');
83
- const $hfLogoutSection = document.getElementById('hf-logout-section');
84
- const $hfLogoutBtn = document.getElementById('hf-logout-btn');
85
  const $datasetRootInput = document.getElementById('dataset-root-input');
86
- const $datasetRootHint = document.getElementById('dataset-root-hint');
87
- const $updateRootBtn = document.getElementById('update-root-btn');
88
- const $leadFramesHead = document.getElementById('lead-frames-head');
89
  const $leadFramesAntennas = document.getElementById('lead-frames-antennas');
90
- const $fetchCommunityBtn = document.getElementById('fetch-community-btn');
91
  const $downloadCommunityBtn = document.getElementById('download-community-btn');
92
- const $communityStatus = document.getElementById('community-status');
93
- const $communityList = document.getElementById('community-list');
94
- const $audioFileInput = document.getElementById('audio-file-input');
95
- const $audioUploadZone = document.getElementById('audio-upload-zone');
96
  const $audioUploadSection = document.getElementById('audio-upload-section');
97
  const $audioUploadStatus = document.getElementById('audio-upload-status');
 
 
 
98
 
99
 
100
  // ═══════════════════════════════════════════════════════════════════════
101
- // SECTION 2: STATE POLLING & CLOCK SYNCHRONIZATION
102
- //
103
- // CLOCK SYNC: We use an NTP-style algorithm to estimate the offset
104
- // between the server clock and the local clock. On each poll:
105
- // 1. Record local time BEFORE the request (t1)
106
- // 2. Server generates server_time during request handling (ts)
107
- // 3. Record local time AFTER the response arrives (t2)
108
- // 4. Estimate one-way latency as (t2 - t1) / 2
109
- // 5. offset = ts - (t1 + one_way_latency) = ts - (t1 + t2) / 2
110
  //
111
- // We keep a running average of the best (lowest-latency) samples
112
- // for stability. This gives ~5-20ms accuracy on LAN/WiFi.
 
 
 
 
113
  //
114
- // ADAPTIVE POLLING: During countdown/recording, we poll every 200ms
115
- // (instead of 1500ms) to get backend timestamps sooner. In idle
116
- // mode we poll every 1500ms to save resources.
117
  // ═══════════════════════════════════════════════════════════════════════
118
 
119
- const POLL_IDLE_MS = 1500; // Polling interval when idle
120
- const POLL_ACTIVE_MS = 200; // Polling interval during countdown/recording
121
 
122
  async function fetchState() {
123
  const id = ++stateSeq;
124
- const t1 = Date.now() / 1000; // Local Unix time before request (seconds)
125
  try {
126
  const r = await fetch('/api/state', { cache: 'no-store' });
127
  if (!r.ok) { scheduleNextPoll('idle'); return; }
128
  const data = await r.json();
129
- const t2 = Date.now() / 1000; // Local Unix time after response (seconds)
130
- if (id < latestSeq) return; // Ignore stale responses
131
  latestSeq = id;
132
 
133
- // NTP-style clock offset calculation.
134
  if (data.server_time) {
135
  const rtt = t2 - t1;
136
  const localMidpoint = (t1 + t2) / 2;
137
  const newOffset = data.server_time - localMidpoint;
138
-
139
  if (rtt < 0.5) {
140
  if (clockOffsetSamples === 0 || rtt < bestRtt) {
141
  clockOffset = newOffset;
@@ -149,8 +164,6 @@ async function fetchState() {
149
 
150
  updateUI(data);
151
  lastState = data;
152
-
153
- // Adaptive polling: faster during active phases
154
  scheduleNextPoll(data.mode);
155
  } catch (e) {
156
  console.error('State poll failed:', e);
@@ -160,117 +173,143 @@ async function fetchState() {
160
 
161
  function scheduleNextPoll(mode) {
162
  clearTimeout(pollingHandle);
163
- const isActive = ['countdown', 'recording', 'playing'].includes(mode);
164
  pollingHandle = setTimeout(fetchState, isActive ? POLL_ACTIVE_MS : POLL_IDLE_MS);
165
  }
166
 
167
- function startPolling() {
168
- fetchState(); // First poll immediately
169
- }
170
 
171
 
172
  // ═══════════════════════════════════════════════════════════════════════
173
- // SECTION 3: UI UPDATE — Single function that syncs all DOM with state
 
 
 
 
 
 
 
 
 
 
 
 
174
  // ═══════════════════════════════════════════════════════════════════════
175
 
176
  function updateUI(s) {
177
- // ── Mode badge ──
178
- $modeBadge.textContent = s.mode.toUpperCase();
179
- $modeBadge.dataset.mode = s.mode;
 
 
180
 
181
- // ── Busy flag — disables interactive controls ──
182
  busy = !['idle', 'queued'].includes(s.mode);
183
 
184
- // ── Record button state ──
185
  const activeEntry = s.datasets?.entries?.find(e => e.id === s.datasets?.active_id);
186
  const isDownloaded = activeEntry?.origin === 'downloaded';
187
  $recordBtn.disabled = busy || isDownloaded;
188
-
189
- // Show warning when active dataset is downloaded (can't record into it)
190
  $downloadedWarning.classList.toggle('visible', isDownloaded && !busy);
191
-
192
- // Visual state of record button
193
  $recordBtn.classList.toggle('is-recording', s.mode === 'recording');
194
  $recordBtn.classList.toggle('is-countdown', s.mode === 'countdown');
195
 
196
- // ── Config (from state.config) ──
197
- if (s.config) {
198
- // Duration field placeholder from preferred duration
199
- const dur = s.config.preferred_duration || s.config.default_duration || 5;
200
- $recDuration.placeholder = dur.toFixed(1);
201
- if (!$recDuration.value) $recDuration.value = dur.toFixed(1);
202
-
203
- // Hugging Face login status — auto-detected, NOT editable
204
- if (s.config.hf_username) {
205
- hfUsername = s.config.hf_username;
206
- $hfStatus.textContent = `Logged in as ${s.config.hf_username}`;
207
- $hfStatus.className = 'hf-status logged-in';
208
- $hfHint.textContent = '';
209
- $hfLoginForm.style.display = 'none';
210
- $hfLogoutSection.style.display = '';
211
- } else {
212
- hfUsername = null;
213
- $hfStatus.textContent = 'Not logged in';
214
- $hfStatus.className = 'hf-status logged-out';
215
- $hfHint.textContent = '';
216
- $hfLoginForm.style.display = '';
217
- $hfLogoutSection.style.display = 'none';
218
- }
219
 
220
- // Dataset root
221
- if (s.config.dataset_root_path) {
222
- if (!$datasetRootInput.value) $datasetRootInput.value = s.config.dataset_root_path;
223
- if ($datasetRootHint) $datasetRootHint.textContent =
224
- `Datasets folder: ${s.config.dataset_root_path}`;
225
- }
226
 
227
- // Audio availability disable mic options if not available
228
- const micRadio = document.querySelector('input[name="audio-src"][value="mic"]');
229
- const micOnlyRadio = document.querySelector('input[name="audio-src"][value="mic-only"]');
230
- if (micRadio) {
231
- micRadio.disabled = !s.config.audio_available;
232
- if (!s.config.audio_available && micRadio.checked) {
233
- document.querySelector('input[name="audio-src"][value="none"]').checked = true;
234
- }
235
- }
236
- if (micOnlyRadio) {
237
- micOnlyRadio.disabled = !s.config.audio_available;
238
- if (!s.config.audio_available && micOnlyRadio.checked) {
239
- document.querySelector('input[name="audio-src"][value="none"]').checked = true;
240
- }
241
- }
242
 
243
- // Welcome messages radio sync
244
- if (s.config.welcome_messages != null) {
245
- const radio = document.querySelector(`input[name="welcome-msgs"][value="${s.config.welcome_messages}"]`);
246
- if (radio && !radio.checked) radio.checked = true;
 
 
 
 
 
 
247
  }
 
 
 
 
 
 
 
 
248
 
249
- // Lead compensation
250
- updateLeadCompUI(s.config);
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
251
  }
252
 
253
- // ── Datasets ──
254
- if (s.datasets) updateDatasetUI(s.datasets);
 
 
 
 
255
 
256
- // ── Phase timing (for overlay animation) ──
257
- updatePhase(s);
 
 
 
 
 
258
 
259
- // ── Moves list ──
260
- const moves = s.moves || [];
261
- reconcileMoves(moves);
262
- renderMoves(moves, s.mode, s.active_move);
263
- $movesCount.textContent = moves.length;
264
- updateUploadBar();
265
  }
266
 
267
 
268
  // ═══════════════════════════════════════════════════════════════════════
269
- // SECTION 4: PHASE OVERLAY — Continuous countdown → recording display
 
 
 
 
 
 
 
 
 
 
 
270
  // ═══════════════════════════════════════════════════════════════════════
271
 
272
  function updatePhase(s) {
273
- if (s.mode === 'countdown' && s.phase_start_at != null && s.phase_end_at != null) {
 
 
274
  phaseMode = 'countdown';
275
  phaseStartAt = s.phase_start_at;
276
  phaseEndAt = s.phase_end_at;
@@ -292,7 +331,16 @@ function updatePhase(s) {
292
  }
293
 
294
  function updatePhaseDisplay() {
295
- // Hide overlay when no active phase
 
 
 
 
 
 
 
 
 
296
  if (phaseStartAt == null || phaseEndAt == null) {
297
  $phaseOverlay.className = 'phase-overlay';
298
  return;
@@ -300,9 +348,8 @@ function updatePhaseDisplay() {
300
 
301
  const now = Date.now() / 1000 + clockOffset;
302
 
303
- // Determine the effective display phase. When phaseMode is still
304
- // 'countdown' (poll hasn't caught up yet) but the countdown timer has
305
- // expired, we PREDICT the recording phase locally.
306
  let displayPhase = phaseMode;
307
  if (phaseMode === 'countdown' && now >= phaseEndAt) {
308
  displayPhase = 'recording';
@@ -311,7 +358,6 @@ function updatePhaseDisplay() {
311
  if (displayPhase === 'countdown') {
312
  const remaining = phaseEndAt - now;
313
  const secs = Math.ceil(remaining);
314
-
315
  $phaseOverlay.className = 'phase-overlay active phase-countdown';
316
  $phaseFill.style.width = '100%';
317
  $phaseNumber.textContent = Math.max(1, secs);
@@ -327,21 +373,18 @@ function updatePhaseDisplay() {
327
  recStart = countdownEndAt || phaseEndAt;
328
  recEnd = recStart + (requestedDuration || 5);
329
  }
330
-
331
  const recTotal = recEnd - recStart;
332
  const recRemaining = Math.max(0, recEnd - now);
333
  const ratio = recTotal > 0 ? Math.min(1, Math.max(0, recRemaining / recTotal)) : 0;
334
 
335
  $phaseOverlay.className = 'phase-overlay active phase-recording';
336
  $phaseFill.style.width = (ratio * 100) + '%';
337
-
338
  $phaseNumber.innerHTML = '<span class="rec-dot"></span>Recording';
339
  $phaseSublabel.textContent = recRemaining.toFixed(1) + 's remaining';
340
  $phaseStopBtn.textContent = 'Stop Recording';
341
  }
342
  }
343
 
344
- /* Animation loop — runs continuously via requestAnimationFrame */
345
  function animationLoop() {
346
  updatePhaseDisplay();
347
  requestAnimationFrame(animationLoop);
@@ -354,14 +397,50 @@ animationLoop();
354
  // ═══════════════════════════════════════════════════════════════════════
355
 
356
  function updateDatasetUI(ds) {
 
 
 
 
 
357
  $datasetSelect.innerHTML = '';
358
- (ds.entries || []).forEach(e => {
359
  const opt = document.createElement('option');
360
  opt.value = e.id;
361
- opt.textContent = (e.origin === 'downloaded' ? '\u2B07 ' : '') + (e.label || e.id);
362
  $datasetSelect.appendChild(opt);
363
  });
364
- if (!datasetSwitching) $datasetSelect.value = ds.active_id || '';
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
365
  }
366
 
367
 
@@ -376,30 +455,39 @@ function reconcileMoves(moves) {
376
  }
377
  }
378
 
379
- function renderMoves(moves, mode, activeMove) {
 
380
  if (!moves.length) {
381
- $movesList.innerHTML = `
 
 
 
 
382
  <div class="moves-empty">
383
- <div class="moves-empty-icon">&#9898;</div>
384
- Record your first move to see it here
385
  </div>`;
386
  return;
387
  }
388
 
389
- $movesList.innerHTML = '';
390
  moves.forEach(m => {
391
  const card = document.createElement('div');
392
  card.className = 'move-card' +
393
  (activeMove === m.id && mode === 'playing' ? ' is-playing' : '');
394
-
395
- const cb = document.createElement('input');
396
- cb.type = 'checkbox';
397
- cb.className = 'move-check';
398
- cb.checked = selectedMoves.has(m.id);
399
- cb.onchange = () => {
400
- cb.checked ? selectedMoves.add(m.id) : selectedMoves.delete(m.id);
401
- updateUploadBar();
402
- };
 
 
 
 
403
 
404
  const info = document.createElement('div');
405
  info.className = 'move-info';
@@ -415,7 +503,9 @@ function renderMoves(moves, mode, activeMove) {
415
  dot.className = 'audio-dot ' + (m.has_audio ? 'has' : 'none');
416
  meta.appendChild(dot);
417
  const dur = m.duration != null ? m.duration.toFixed(1) + 's' : '--';
418
- const date = new Date(m.created_at * 1000).toLocaleDateString();
 
 
419
  meta.appendChild(document.createTextNode(dur + ' \u00B7 ' + date));
420
  info.appendChild(meta);
421
 
@@ -440,23 +530,43 @@ function renderMoves(moves, mode, activeMove) {
440
  playBtn.textContent = (mode === 'playing' && activeMove === m.id) ? 'Stop' : 'Play';
441
  playBtn.disabled = busy && !(mode === 'playing' && activeMove === m.id);
442
  playBtn.onclick = () => {
443
- if (mode === 'playing' && activeMove === m.id) stopPlayback();
444
  else queuePlayback(m.id);
445
  };
446
-
447
- const delBtn = document.createElement('button');
448
- delBtn.className = 'move-action delete-action';
449
- delBtn.textContent = 'Delete';
450
- delBtn.disabled = busy;
451
- delBtn.onclick = () => deleteMove(m.id);
452
-
453
  actions.appendChild(playBtn);
454
- actions.appendChild(delBtn);
455
 
456
- card.appendChild(cb);
 
 
 
 
 
 
 
 
 
457
  card.appendChild(info);
458
  card.appendChild(actions);
459
- $movesList.appendChild(card);
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
460
  });
461
  }
462
 
@@ -474,24 +584,39 @@ function updateUploadBar() {
474
 
475
  // ═══════════════════════════════════════════════════════════════════════
476
  // SECTION 7: USER ACTIONS — API calls triggered by user interactions
 
 
 
 
 
 
477
  // ═══════════════════════════════════════════════════════════════════════
478
 
 
 
 
479
  async function queueRecording() {
480
  if (busy) return;
481
  const dur = parseFloat($recDuration.value) || parseFloat($recDuration.placeholder) || 5;
482
  const label = $recName.value.trim() || null;
483
- const audioSrc = document.querySelector('input[name="audio-src"]:checked')?.value || 'mic';
484
- const isMicOnly = audioSrc === 'mic-only';
485
- const recordAudio = audioSrc !== 'none';
486
- const recordMotion = !isMicOnly;
487
- const audioId = audioSrc === 'upload' ? uploadedAudioId : null;
488
 
489
- if (audioSrc === 'upload' && !uploadedAudioId) return;
 
 
 
 
490
 
 
491
  requestedDuration = dur;
492
  $recordBtn.disabled = true;
493
  try {
494
- const payload = { duration: dur, record_audio: recordAudio, record_motion: recordMotion, label };
 
 
 
 
 
495
  if (audioId) payload.uploaded_audio_id = audioId;
496
  const r = await fetch('/api/record', {
497
  method: 'POST',
@@ -499,11 +624,10 @@ async function queueRecording() {
499
  body: JSON.stringify(payload),
500
  });
501
  if (!r.ok) throw new Error(await r.text());
 
502
  await fetchState();
503
- // Preserve label for next recording
504
  const savedLabel = window.localStorage.getItem(STORAGE_KEYS.moveLabel);
505
  $recName.value = savedLabel || '';
506
- // Keep uploadedAudioId so the user can record again with the same file
507
  } catch (e) {
508
  console.error('Record error:', e);
509
  } finally {
@@ -511,6 +635,49 @@ async function queueRecording() {
511
  }
512
  }
513
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
514
  async function queuePlayback(id) {
515
  if (busy) return;
516
  try {
@@ -529,7 +696,7 @@ async function stopPlayback() {
529
  stopping = true;
530
  try {
531
  const ctrl = new AbortController();
532
- const tid = setTimeout(() => ctrl.abort(), 10000);
533
  await fetch('/api/play/stop', { method: 'POST', signal: ctrl.signal });
534
  clearTimeout(tid);
535
  await fetchState();
@@ -543,9 +710,10 @@ async function stopRecording() {
543
  $phaseStopBtn.disabled = true;
544
  try {
545
  const ctrl = new AbortController();
546
- const tid = setTimeout(() => ctrl.abort(), 10000);
547
  await fetch('/api/record/stop', { method: 'POST', signal: ctrl.signal });
548
  clearTimeout(tid);
 
549
  await fetchState();
550
  } catch (e) { console.error(e); }
551
  finally { stopping = false; $phaseStopBtn.disabled = false; }
@@ -558,6 +726,7 @@ async function deleteMove(id) {
558
  const r = await fetch('/api/moves/' + encodeURIComponent(id), { method: 'DELETE' });
559
  if (!r.ok) throw new Error(await r.text());
560
  selectedMoves.delete(id);
 
561
  await fetchState();
562
  } catch (e) { console.error('Delete error:', e); }
563
  }
@@ -572,6 +741,8 @@ async function selectDataset(id) {
572
  });
573
  if (!r.ok) throw new Error(await r.text());
574
  selectedMoves.clear();
 
 
575
  await fetchState();
576
  } catch (e) { console.error(e); }
577
  finally { datasetSwitching = false; }
@@ -589,6 +760,8 @@ async function createDataset(name) {
589
  const d = await r.json().catch(() => ({}));
590
  throw new Error(d.detail || r.statusText);
591
  }
 
 
592
  await fetchState();
593
  $newDatasetInline.classList.remove('active');
594
  $newDatasetBtn.style.display = '';
@@ -608,6 +781,7 @@ async function syncToHF() {
608
  });
609
  if (!r.ok) throw new Error(await r.text());
610
  selectedMoves.clear();
 
611
  await fetchState();
612
  } catch (e) {
613
  console.error(e);
@@ -629,6 +803,8 @@ async function updateDatasetRoot() {
629
  body: JSON.stringify({ path }),
630
  });
631
  if (!r.ok) throw new Error(await r.text());
 
 
632
  await fetchState();
633
  } catch (e) { console.error(e); }
634
  }
@@ -644,15 +820,80 @@ async function uploadAudioFile(file) {
644
  const data = await r.json();
645
  uploadedAudioId = data.upload_id;
646
  uploadedAudioFilename = data.filename;
647
- $audioUploadStatus.textContent = `Ready: ${data.filename}` + (data.duration ? ` (${data.duration.toFixed(1)}s)` : '');
648
- if (data.duration && $recDuration) $recDuration.value = (Math.round(data.duration * 10) / 10).toFixed(1);
649
- // Auto-close settings drawer after successful upload
650
- $settingsBackdrop.classList.remove('open');
651
- $settingsDrawer.classList.remove('open');
652
  } catch (e) {
653
  $audioUploadStatus.textContent = 'Failed: ' + e.message;
654
  uploadedAudioId = null;
655
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
656
  }
657
 
658
 
@@ -727,6 +968,8 @@ async function downloadCommunity() {
727
  }
728
  communitySelection.clear();
729
  $communityStatus.textContent = 'Download complete.';
 
 
730
  await fetchState();
731
  } catch (e) {
732
  $communityStatus.textContent = 'Error: ' + e.message;
@@ -745,14 +988,12 @@ function updateLeadCompUI(config) {
745
  const modelsPayload = config.motion_models;
746
  if (!modelsPayload) return;
747
  const params = modelsPayload.params?.lead_compensation || {};
748
- if ($leadFramesHead && document.activeElement !== $leadFramesHead && Number.isFinite(params.lead_frames_head)) {
 
749
  $leadFramesHead.value = String(params.lead_frames_head);
750
  }
751
- if (
752
- $leadFramesAntennas
753
- && document.activeElement !== $leadFramesAntennas
754
- && Number.isFinite(params.lead_frames_antennas)
755
- ) {
756
  $leadFramesAntennas.value = String(params.lead_frames_antennas);
757
  }
758
  }
@@ -762,22 +1003,42 @@ function updateLeadCompUI(config) {
762
  // SECTION 10: EVENT LISTENERS
763
  // ═══════════════════════════════════════════════════════════════════════
764
 
765
- /* Record button — either starts recording or stops it */
766
  $recordBtn.addEventListener('click', () => {
767
- if (busy && (phaseMode === 'recording' || phaseMode === 'countdown')) {
768
  stopRecording();
769
  } else {
770
  queueRecording();
771
  }
772
  });
773
 
774
- /* Phase overlay stop button */
775
- $phaseStopBtn.addEventListener('click', stopRecording);
 
 
 
 
 
 
 
 
 
 
 
 
 
 
776
 
777
- /* Dataset selector */
778
  $datasetSelect.addEventListener('change', e => {
779
  if (e.target.value) selectDataset(e.target.value);
780
  });
 
 
 
 
 
 
781
 
782
  /* New dataset creation (inline form) */
783
  $newDatasetBtn.addEventListener('click', () => {
@@ -795,7 +1056,6 @@ $newDatasetInput.addEventListener('keydown', e => {
795
  if (e.key === 'Enter') { e.preventDefault(); createDataset($newDatasetInput.value.trim()); }
796
  if (e.key === 'Escape') { $newDatasetInline.classList.remove('active'); $newDatasetBtn.style.display = ''; }
797
  });
798
- /* Auto-lowercase and replace spaces with underscores */
799
  $newDatasetInput.addEventListener('input', e => {
800
  e.target.value = e.target.value.toLowerCase().replace(/\s+/g, '_');
801
  });
@@ -820,37 +1080,54 @@ $settingsClose.addEventListener('click', () => {
820
  /* Dataset root update */
821
  $updateRootBtn.addEventListener('click', updateDatasetRoot);
822
 
823
- /* Audio source radio buttons */
824
  document.querySelectorAll('input[name="audio-src"]').forEach(r => {
825
  r.addEventListener('change', () => {
826
- $audioUploadSection.style.display = (r.value === 'upload' && r.checked) ? '' : 'none';
827
- if (r.value !== 'upload') {
828
- uploadedAudioId = null;
829
- uploadedAudioFilename = null;
830
- $audioUploadStatus.textContent = '';
831
- }
 
 
 
 
 
 
832
  });
833
  });
834
 
835
  /* Audio upload zone — click and drag-and-drop */
836
- $audioUploadZone.addEventListener('click', () => $audioFileInput.click());
837
- $audioFileInput.addEventListener('change', e => {
838
- if (e.target.files?.[0]) uploadAudioFile(e.target.files[0]);
839
- });
840
- $audioUploadZone.addEventListener('dragover', e => {
841
- e.preventDefault();
842
- $audioUploadZone.classList.add('drag-over');
843
- });
844
- $audioUploadZone.addEventListener('dragleave', e => {
845
- e.preventDefault();
846
- $audioUploadZone.classList.remove('drag-over');
847
- });
848
- $audioUploadZone.addEventListener('drop', e => {
849
- e.preventDefault();
850
- $audioUploadZone.classList.remove('drag-over');
851
- const f = e.dataTransfer?.files?.[0];
852
- if (f && /\.(wav|mp3)$/i.test(f.name)) uploadAudioFile(f);
853
- });
 
 
 
 
 
 
 
 
 
 
 
854
 
855
  /* HF login/logout */
856
  $hfLoginBtn.addEventListener('click', async () => {
@@ -932,18 +1209,43 @@ $recDuration.addEventListener('change', () => {
932
  }, 500);
933
  });
934
 
935
- /* Tab switching */
936
  document.querySelectorAll('.section-tab').forEach(tab => {
937
  tab.addEventListener('click', () => {
938
  document.querySelectorAll('.section-tab').forEach(t => t.classList.remove('active'));
939
  document.querySelectorAll('.tab-panel').forEach(p => p.classList.remove('active'));
940
  tab.classList.add('active');
941
  document.getElementById('tab-' + tab.dataset.tab)?.classList.add('active');
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
942
  });
943
  });
944
 
945
  /* Community */
946
- $fetchCommunityBtn.addEventListener('click', fetchCommunity);
947
  $downloadCommunityBtn.addEventListener('click', downloadCommunity);
948
 
949
  /* Name field — persist to localStorage */
@@ -965,5 +1267,4 @@ if ($recName) {
965
  // SECTION 11: INIT
966
  // ═══════════════════════════════════════════════════════════════════════
967
 
968
- /* Start polling */
969
  startPolling();
 
1
  /* ═══════════════════════════════════════════════════════════════════════
2
  MARIONETTE FRONTEND — JavaScript
3
 
4
+ Architecture overview
5
+ ────────────────────
6
+ 1. STATE POLLING: fetchState() runs every 1500 ms (200 ms during active
7
+ phases), calls GET /api/state, and passes the response to updateUI().
8
+
9
+ 2. UI UPDATE: updateUI() synchronises lightweight DOM elements with the
10
+ backend state: the mode badge, the busy flag, the record-button
11
+ appearance, and the phase-overlay timing variables. Heavy DOM work
12
+ (moves list, dataset dropdown) is ONLY done when a "dirty" flag is
13
+ set — which happens exclusively after user actions that change data
14
+ (record, delete, dataset switch …). This prevents the 1.5 s polling
15
+ loop from rebuilding DOM and causing visual flickering.
16
 
17
  3. PHASE ANIMATION: A requestAnimationFrame loop continuously updates
18
+ the countdown / recording overlay based on phase_start_at /
19
+ phase_end_at timestamps from the backend.
20
 
21
+ 4. EVENT HANDLERS: All user interactions (record, play, delete ) send
22
+ the appropriate API call, set dirty flags, then trigger a state
23
+ refresh via fetchState().
24
  ═══════════════════════════════════════════════════════════════════════ */
25
 
26
 
 
28
  // SECTION 1: APP STATE & ELEMENT REFERENCES
29
  // ═══════════════════════════════════════════════════════════════════════
30
 
31
+ let lastState = null; // Last received state from backend
32
+ let busy = false; // True when mode is not idle/queued
33
+ let clockOffset = 0; // Diff between server clock and local clock (seconds)
34
+ let clockOffsetSamples = 0; // Number of clock-offset measurements taken
35
+ let bestRtt = Infinity; // Lowest round-trip time seen (best clock sample)
36
+ let phaseMode = 'idle'; // 'idle' | 'countdown' | 'recording' | …
37
+ let phaseStartAt = null; // Server timestamp: current phase started
38
+ let phaseEndAt = null; // Server timestamp: current phase ends
39
+ let countdownEndAt = null; // Server timestamp: countdown ends (= recording starts)
40
+ let recordingEndAt = null; // Server timestamp: recording ends
41
+ let requestedDuration = null; // Duration the user requested (for prediction)
42
+ let selectedMoves = new Set(); // Move IDs checked for HF upload
43
+ let communitySelection = new Set(); // Community dataset repo_ids selected
44
+ let uploadedAudioId = null; // ID from /api/upload-audio or /api/robot-audio/select
45
+ let uploadedAudioFilename = null;
46
+ let stopping = false; // Debounce: true while a stop request is in flight
47
+ let datasetSwitching = false; // True while dataset select is in flight
48
+ let pollingHandle = null; // setTimeout handle for adaptive polling
49
+ let stateSeq = 0; // Monotonic counter ignore out-of-order responses
50
+ let latestSeq = 0; // Highest seq processed
51
+ let hfUsername = null; // Auto-detected HF username
52
+
53
+ // ── Dirty flags ──
54
+ // When true, the next updateUI() call will do a full DOM rebuild for that
55
+ // section. They start as true so the first poll populates everything.
56
+ let movesListDirty = true;
57
+ let datasetsDirty = true;
58
+ let libraryDirty = true;
59
+ let lastLocalDatasetId = null;
60
 
61
  const STORAGE_KEYS = {
62
  moveLabel: 'marionette.last_move_label',
63
  };
64
 
65
+ /* DOM element references — gathered once at startup.
66
+ We use the convention $name for DOM elements. */
67
+ const $modeBadge = document.getElementById('mode-badge');
68
+ const $recordBtn = document.getElementById('record-btn');
69
+ const $recordAudioBtn = document.getElementById('record-audio-btn');
70
+ const $recName = document.getElementById('rec-name');
71
+ const $recDuration = document.getElementById('rec-duration');
72
  const $downloadedWarning = document.getElementById('downloaded-warning');
73
+ const $movesList = document.getElementById('moves-list');
74
+ const $myMovesCount = document.getElementById('my-moves-count');
75
+ const $libraryCount = document.getElementById('library-count');
76
+ const $libraryDatasetSelect = document.getElementById('library-dataset-select');
77
+ const $libraryMovesList = document.getElementById('library-moves-list');
78
+ const $uploadBar = document.getElementById('upload-bar');
79
+ const $uploadCount = document.getElementById('upload-count');
80
+ const $uploadBtn = document.getElementById('upload-btn');
81
+ const $datasetSelect = document.getElementById('dataset-select');
82
+ const $newDatasetBtn = document.getElementById('new-dataset-btn');
83
  const $newDatasetInline = document.getElementById('new-dataset-inline');
84
+ const $newDatasetInput = document.getElementById('new-dataset-input');
85
  const $createDatasetBtn = document.getElementById('create-dataset-btn');
86
  const $cancelDatasetBtn = document.getElementById('cancel-dataset-btn');
87
+ const $settingsBtn = document.getElementById('settings-btn');
88
  const $settingsBackdrop = document.getElementById('settings-backdrop');
89
+ const $settingsDrawer = document.getElementById('settings-drawer');
90
+ const $settingsClose = document.getElementById('settings-close');
91
+ const $phaseOverlay = document.getElementById('phase-overlay');
92
+ const $phaseNumber = document.getElementById('phase-number');
93
+ const $phaseSublabel = document.getElementById('phase-sublabel');
94
+ const $phaseFill = document.getElementById('phase-fill');
95
+ const $phaseStopBtn = document.getElementById('phase-stop-btn');
96
+ const $hfStatus = document.getElementById('hf-status');
97
+ const $hfHint = document.getElementById('hf-hint');
98
+ const $hfLoginForm = document.getElementById('hf-login-form');
99
+ const $hfTokenInput = document.getElementById('hf-token-input');
100
+ const $hfLoginBtn = document.getElementById('hf-login-btn');
101
+ const $hfLogoutSection = document.getElementById('hf-logout-section');
102
+ const $hfLogoutBtn = document.getElementById('hf-logout-btn');
103
  const $datasetRootInput = document.getElementById('dataset-root-input');
104
+ const $datasetRootHint = document.getElementById('dataset-root-hint');
105
+ const $updateRootBtn = document.getElementById('update-root-btn');
106
+ const $leadFramesHead = document.getElementById('lead-frames-head');
107
  const $leadFramesAntennas = document.getElementById('lead-frames-antennas');
 
108
  const $downloadCommunityBtn = document.getElementById('download-community-btn');
109
+ const $communityStatus = document.getElementById('community-status');
110
+ const $communityList = document.getElementById('community-list');
111
+ const $audioFileInput = document.getElementById('audio-file-input');
112
+ const $audioUploadZone = document.getElementById('audio-upload-zone');
113
  const $audioUploadSection = document.getElementById('audio-upload-section');
114
  const $audioUploadStatus = document.getElementById('audio-upload-status');
115
+ const $robotAudioSection = document.getElementById('robot-audio-section');
116
+ const $robotAudioSelect = document.getElementById('robot-audio-select');
117
+ const $audioSelectionStatus = document.getElementById('audio-selection-status');
118
 
119
 
120
  // ═══════════════════════════════════════════════════════════════════════
121
+ // SECTION 2: STATE POLLING & CLOCK SYNCHRONISATION
 
 
 
 
 
 
 
 
122
  //
123
+ // CLOCK SYNC (NTP-style):
124
+ // On each poll we measure the round-trip time (RTT) and estimate
125
+ // the offset between the server clock and the local clock.
126
+ // offset ≈ server_time − midpoint_of_local_times
127
+ // We weight towards the lowest-RTT sample for accuracy (~5–20 ms on
128
+ // LAN/WiFi).
129
  //
130
+ // ADAPTIVE POLLING:
131
+ // During countdown / recording / playing we poll every 200 ms so the
132
+ // phase overlay stays responsive. In idle mode we poll every 1500 ms.
133
  // ═══════════════════════════════════════════════════════════════════════
134
 
135
+ const POLL_IDLE_MS = 1500;
136
+ const POLL_ACTIVE_MS = 200;
137
 
138
  async function fetchState() {
139
  const id = ++stateSeq;
140
+ const t1 = Date.now() / 1000;
141
  try {
142
  const r = await fetch('/api/state', { cache: 'no-store' });
143
  if (!r.ok) { scheduleNextPoll('idle'); return; }
144
  const data = await r.json();
145
+ const t2 = Date.now() / 1000;
146
+ if (id < latestSeq) return; // stale response
147
  latestSeq = id;
148
 
149
+ // NTP-style clock offset
150
  if (data.server_time) {
151
  const rtt = t2 - t1;
152
  const localMidpoint = (t1 + t2) / 2;
153
  const newOffset = data.server_time - localMidpoint;
 
154
  if (rtt < 0.5) {
155
  if (clockOffsetSamples === 0 || rtt < bestRtt) {
156
  clockOffset = newOffset;
 
164
 
165
  updateUI(data);
166
  lastState = data;
 
 
167
  scheduleNextPoll(data.mode);
168
  } catch (e) {
169
  console.error('State poll failed:', e);
 
173
 
174
  function scheduleNextPoll(mode) {
175
  clearTimeout(pollingHandle);
176
+ const isActive = ['preparing', 'countdown', 'recording', 'playing'].includes(mode);
177
  pollingHandle = setTimeout(fetchState, isActive ? POLL_ACTIVE_MS : POLL_IDLE_MS);
178
  }
179
 
180
+ function startPolling() { fetchState(); }
 
 
181
 
182
 
183
  // ═══════════════════════════════════════════════════════════════════════
184
+ // SECTION 3: UI UPDATE
185
+ //
186
+ // updateUI() is called after every poll AND after every user action.
187
+ // It intentionally does ONLY lightweight work on each poll:
188
+ // - mode badge text
189
+ // - busy flag
190
+ // - record-button appearance
191
+ // - phase overlay timing variables
192
+ // - config fields (only when not focused by user)
193
+ //
194
+ // The expensive work — rebuilding the moves list and dataset dropdown —
195
+ // is gated behind dirty flags that are set by user actions only.
196
+ // This eliminates the flickering caused by DOM rebuilds every 1.5 s.
197
  // ═══════════════════════════════════════════════════════════════════════
198
 
199
  function updateUI(s) {
200
+ // ── Mode badge (one text write, no DOM rebuild) ──
201
+ if ($modeBadge.textContent !== s.mode.toUpperCase()) {
202
+ $modeBadge.textContent = s.mode.toUpperCase();
203
+ $modeBadge.dataset.mode = s.mode;
204
+ }
205
 
206
+ // ── Busy flag ──
207
  busy = !['idle', 'queued'].includes(s.mode);
208
 
209
+ // ── Record buttons state ──
210
  const activeEntry = s.datasets?.entries?.find(e => e.id === s.datasets?.active_id);
211
  const isDownloaded = activeEntry?.origin === 'downloaded';
212
  $recordBtn.disabled = busy || isDownloaded;
213
+ if ($recordAudioBtn) $recordAudioBtn.disabled = busy;
 
214
  $downloadedWarning.classList.toggle('visible', isDownloaded && !busy);
 
 
215
  $recordBtn.classList.toggle('is-recording', s.mode === 'recording');
216
  $recordBtn.classList.toggle('is-countdown', s.mode === 'countdown');
217
 
218
+ // ── Phase timing (variables only — rAF loop reads them) ──
219
+ updatePhase(s);
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
220
 
221
+ // ── Config: lightweight field updates ──
222
+ if (s.config) updateConfigUI(s.config);
 
 
 
 
223
 
224
+ // ── Datasets: only rebuild when dirty ──
225
+ if (datasetsDirty && s.datasets) {
226
+ updateDatasetUI(s.datasets);
227
+ datasetsDirty = false;
228
+ }
 
 
 
 
 
 
 
 
 
 
229
 
230
+ // ── Moves list: render into My Moves or Library depending on active dataset ──
231
+ if (movesListDirty) {
232
+ const moves = s.moves || [];
233
+ reconcileMoves(moves);
234
+ if (isDownloaded && $libraryMovesList) {
235
+ renderMoves(moves, s.mode, s.active_move, $libraryMovesList, true);
236
+ $libraryCount.textContent = moves.length;
237
+ } else {
238
+ renderMoves(moves, s.mode, s.active_move, $movesList, false);
239
+ $myMovesCount.textContent = moves.length;
240
  }
241
+ updateUploadBar();
242
+ movesListDirty = false;
243
+ } else {
244
+ // Lightweight: update play/stop buttons and is-playing state
245
+ // without rebuilding the entire list
246
+ updateMoveCardStates(s.mode, s.active_move);
247
+ }
248
+ }
249
 
250
+ /* Update config fields in the UI. Only writes to DOM elements that are
251
+ not currently focused by the user (to avoid overwriting mid-edit). */
252
+ function updateConfigUI(config) {
253
+ const dur = config.preferred_duration || config.default_duration || 5;
254
+ $recDuration.placeholder = dur.toFixed(1);
255
+ if (!$recDuration.value) $recDuration.value = dur.toFixed(1);
256
+
257
+ // HF login status
258
+ if (config.hf_username) {
259
+ hfUsername = config.hf_username;
260
+ $hfStatus.textContent = `Logged in as ${config.hf_username}`;
261
+ $hfStatus.className = 'hf-status logged-in';
262
+ $hfHint.textContent = '';
263
+ $hfLoginForm.style.display = 'none';
264
+ $hfLogoutSection.style.display = '';
265
+ } else {
266
+ hfUsername = null;
267
+ $hfStatus.textContent = 'Not logged in';
268
+ $hfStatus.className = 'hf-status logged-out';
269
+ $hfHint.textContent = '';
270
+ $hfLoginForm.style.display = '';
271
+ $hfLogoutSection.style.display = 'none';
272
  }
273
 
274
+ // Dataset root
275
+ if (config.dataset_root_path) {
276
+ if (!$datasetRootInput.value) $datasetRootInput.value = config.dataset_root_path;
277
+ if ($datasetRootHint) $datasetRootHint.textContent =
278
+ `Datasets folder: ${config.dataset_root_path}`;
279
+ }
280
 
281
+ // Welcome messages radio sync
282
+ if (config.welcome_messages != null) {
283
+ const radio = document.querySelector(
284
+ `input[name="welcome-msgs"][value="${config.welcome_messages}"]`
285
+ );
286
+ if (radio && !radio.checked) radio.checked = true;
287
+ }
288
 
289
+ // Lead compensation
290
+ updateLeadCompUI(config);
 
 
 
 
291
  }
292
 
293
 
294
  // ═══════════════════════════════════════════════════════════════════════
295
+ // SECTION 4: PHASE OVERLAY — Countdown → recording animation
296
+ //
297
+ // The phase overlay is a full-screen overlay that shows countdown
298
+ // numbers (3… 2… 1…) then a recording progress bar. It uses
299
+ // requestAnimationFrame (rAF) for smooth 60 fps animation.
300
+ //
301
+ // How it works:
302
+ // 1. updatePhase(s) — called from updateUI(), stores server timestamps
303
+ // into JS variables (phaseStartAt, phaseEndAt, etc.)
304
+ // 2. animationLoop() — runs every frame via rAF, calls
305
+ // updatePhaseDisplay() which reads those timestamps and the
306
+ // NTP-corrected local time to render the overlay.
307
  // ═══════════════════════════════════════════════════════════════════════
308
 
309
  function updatePhase(s) {
310
+ if (s.mode === 'preparing') {
311
+ phaseMode = 'preparing';
312
+ } else if (s.mode === 'countdown' && s.phase_start_at != null && s.phase_end_at != null) {
313
  phaseMode = 'countdown';
314
  phaseStartAt = s.phase_start_at;
315
  phaseEndAt = s.phase_end_at;
 
331
  }
332
 
333
  function updatePhaseDisplay() {
334
+ // "Preparing…" overlay while audio is being preloaded/resampled
335
+ if (phaseMode === 'preparing') {
336
+ $phaseOverlay.className = 'phase-overlay active phase-countdown';
337
+ $phaseFill.style.width = '100%';
338
+ $phaseNumber.textContent = 'Preparing\u2026';
339
+ $phaseSublabel.textContent = '';
340
+ $phaseStopBtn.textContent = 'Cancel';
341
+ return;
342
+ }
343
+
344
  if (phaseStartAt == null || phaseEndAt == null) {
345
  $phaseOverlay.className = 'phase-overlay';
346
  return;
 
348
 
349
  const now = Date.now() / 1000 + clockOffset;
350
 
351
+ // When phaseMode is 'countdown' but the timer expired, predict
352
+ // the recording phase locally (poll may not have caught up yet).
 
353
  let displayPhase = phaseMode;
354
  if (phaseMode === 'countdown' && now >= phaseEndAt) {
355
  displayPhase = 'recording';
 
358
  if (displayPhase === 'countdown') {
359
  const remaining = phaseEndAt - now;
360
  const secs = Math.ceil(remaining);
 
361
  $phaseOverlay.className = 'phase-overlay active phase-countdown';
362
  $phaseFill.style.width = '100%';
363
  $phaseNumber.textContent = Math.max(1, secs);
 
373
  recStart = countdownEndAt || phaseEndAt;
374
  recEnd = recStart + (requestedDuration || 5);
375
  }
 
376
  const recTotal = recEnd - recStart;
377
  const recRemaining = Math.max(0, recEnd - now);
378
  const ratio = recTotal > 0 ? Math.min(1, Math.max(0, recRemaining / recTotal)) : 0;
379
 
380
  $phaseOverlay.className = 'phase-overlay active phase-recording';
381
  $phaseFill.style.width = (ratio * 100) + '%';
 
382
  $phaseNumber.innerHTML = '<span class="rec-dot"></span>Recording';
383
  $phaseSublabel.textContent = recRemaining.toFixed(1) + 's remaining';
384
  $phaseStopBtn.textContent = 'Stop Recording';
385
  }
386
  }
387
 
 
388
  function animationLoop() {
389
  updatePhaseDisplay();
390
  requestAnimationFrame(animationLoop);
 
397
  // ═══════════════════════════════════════════════════════════════════════
398
 
399
  function updateDatasetUI(ds) {
400
+ const entries = ds.entries || [];
401
+ const localEntries = entries.filter(e => e.origin !== 'downloaded');
402
+ const downloadedEntries = entries.filter(e => e.origin === 'downloaded');
403
+
404
+ // My Moves dropdown — local datasets only
405
  $datasetSelect.innerHTML = '';
406
+ localEntries.forEach(e => {
407
  const opt = document.createElement('option');
408
  opt.value = e.id;
409
+ opt.textContent = e.label || e.id;
410
  $datasetSelect.appendChild(opt);
411
  });
412
+
413
+ // Library dropdown — downloaded datasets only
414
+ if ($libraryDatasetSelect) {
415
+ $libraryDatasetSelect.innerHTML = '';
416
+ if (!downloadedEntries.length) {
417
+ const opt = document.createElement('option');
418
+ opt.value = '';
419
+ opt.textContent = 'No downloaded datasets';
420
+ opt.disabled = true;
421
+ $libraryDatasetSelect.appendChild(opt);
422
+ }
423
+ downloadedEntries.forEach(e => {
424
+ const opt = document.createElement('option');
425
+ opt.value = e.id;
426
+ opt.textContent = e.label || e.id;
427
+ $libraryDatasetSelect.appendChild(opt);
428
+ });
429
+ }
430
+
431
+ // Track last local dataset for tab switching
432
+ const activeEntry = entries.find(e => e.id === ds.active_id);
433
+ if (activeEntry && activeEntry.origin !== 'downloaded') {
434
+ lastLocalDatasetId = ds.active_id;
435
+ }
436
+
437
+ if (!datasetSwitching) {
438
+ if (activeEntry?.origin === 'downloaded' && $libraryDatasetSelect) {
439
+ $libraryDatasetSelect.value = ds.active_id || '';
440
+ } else {
441
+ $datasetSelect.value = ds.active_id || '';
442
+ }
443
+ }
444
  }
445
 
446
 
 
455
  }
456
  }
457
 
458
+ function renderMoves(moves, mode, activeMove, container, readOnly) {
459
+ container = container || $movesList;
460
  if (!moves.length) {
461
+ const icon = readOnly ? '&#11088;' : '&#9898;';
462
+ const msg = readOnly
463
+ ? 'Download datasets from the Community tab to see them here'
464
+ : 'Record your first move to see it here';
465
+ container.innerHTML = `
466
  <div class="moves-empty">
467
+ <div class="moves-empty-icon">${icon}</div>
468
+ ${msg}
469
  </div>`;
470
  return;
471
  }
472
 
473
+ container.innerHTML = '';
474
  moves.forEach(m => {
475
  const card = document.createElement('div');
476
  card.className = 'move-card' +
477
  (activeMove === m.id && mode === 'playing' ? ' is-playing' : '');
478
+ card.dataset.moveId = m.id; // used by updateMoveCardStates()
479
+
480
+ let cb = null;
481
+ if (!readOnly) {
482
+ cb = document.createElement('input');
483
+ cb.type = 'checkbox';
484
+ cb.className = 'move-check';
485
+ cb.checked = selectedMoves.has(m.id);
486
+ cb.onchange = () => {
487
+ cb.checked ? selectedMoves.add(m.id) : selectedMoves.delete(m.id);
488
+ updateUploadBar();
489
+ };
490
+ }
491
 
492
  const info = document.createElement('div');
493
  info.className = 'move-info';
 
503
  dot.className = 'audio-dot ' + (m.has_audio ? 'has' : 'none');
504
  meta.appendChild(dot);
505
  const dur = m.duration != null ? m.duration.toFixed(1) + 's' : '--';
506
+ const date = new Date(m.created_at * 1000).toLocaleString(undefined, {
507
+ month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit'
508
+ });
509
  meta.appendChild(document.createTextNode(dur + ' \u00B7 ' + date));
510
  info.appendChild(meta);
511
 
 
530
  playBtn.textContent = (mode === 'playing' && activeMove === m.id) ? 'Stop' : 'Play';
531
  playBtn.disabled = busy && !(mode === 'playing' && activeMove === m.id);
532
  playBtn.onclick = () => {
533
+ if (playBtn.closest('.move-card')?.classList.contains('is-playing')) stopPlayback();
534
  else queuePlayback(m.id);
535
  };
 
 
 
 
 
 
 
536
  actions.appendChild(playBtn);
 
537
 
538
+ if (!readOnly) {
539
+ const delBtn = document.createElement('button');
540
+ delBtn.className = 'move-action delete-action';
541
+ delBtn.textContent = 'Delete';
542
+ delBtn.disabled = busy;
543
+ delBtn.onclick = () => deleteMove(m.id);
544
+ actions.appendChild(delBtn);
545
+ }
546
+
547
+ if (cb) card.appendChild(cb);
548
  card.appendChild(info);
549
  card.appendChild(actions);
550
+ container.appendChild(card);
551
+ });
552
+ }
553
+
554
+ /* Lightweight update of play/stop buttons and is-playing class on
555
+ existing move cards. Called by updateUI() on every poll when the
556
+ moves list is NOT dirty. This avoids a full DOM rebuild while still
557
+ keeping play/stop buttons responsive. */
558
+ function updateMoveCardStates(mode, activeMove) {
559
+ document.querySelectorAll('.move-card').forEach(card => {
560
+ const id = card.dataset.moveId;
561
+ const isPlaying = (mode === 'playing' && activeMove === id);
562
+ card.classList.toggle('is-playing', isPlaying);
563
+ const playBtn = card.querySelector('.play-action');
564
+ if (playBtn) {
565
+ playBtn.textContent = isPlaying ? 'Stop' : 'Play';
566
+ playBtn.disabled = busy && !isPlaying;
567
+ }
568
+ const delBtn = card.querySelector('.delete-action');
569
+ if (delBtn) delBtn.disabled = busy;
570
  });
571
  }
572
 
 
584
 
585
  // ═══════════════════════════════════════════════════════════════════════
586
  // SECTION 7: USER ACTIONS — API calls triggered by user interactions
587
+ //
588
+ // Each action follows the same pattern:
589
+ // 1. Guard (busy? valid input?)
590
+ // 2. Call the backend API
591
+ // 3. Set dirty flags for affected UI sections
592
+ // 4. Call fetchState() to get the latest state
593
  // ═══════════════════════════════════════════════════════════════════════
594
 
595
+ /* Record Move — motion capture with optional uploaded audio playback.
596
+ Audio source can be: silent, upload (from laptop), or robot file.
597
+ Mic recording during motion is NOT supported (motor noise). */
598
  async function queueRecording() {
599
  if (busy) return;
600
  const dur = parseFloat($recDuration.value) || parseFloat($recDuration.placeholder) || 5;
601
  const label = $recName.value.trim() || null;
602
+ const audioSrc = document.querySelector('input[name="audio-src"]:checked')?.value || 'silent';
 
 
 
 
603
 
604
+ // For upload/robot sources, audio must be prepared first
605
+ if ((audioSrc === 'upload' || audioSrc === 'robot') && !uploadedAudioId) {
606
+ alert('Please select or upload an audio file first, or choose Silent.');
607
+ return;
608
+ }
609
 
610
+ const audioId = uploadedAudioId; // null for 'silent'
611
  requestedDuration = dur;
612
  $recordBtn.disabled = true;
613
  try {
614
+ const payload = {
615
+ duration: dur,
616
+ record_audio: false, // mic-during-motion removed
617
+ record_motion: true,
618
+ label,
619
+ };
620
  if (audioId) payload.uploaded_audio_id = audioId;
621
  const r = await fetch('/api/record', {
622
  method: 'POST',
 
624
  body: JSON.stringify(payload),
625
  });
626
  if (!r.ok) throw new Error(await r.text());
627
+ movesListDirty = true;
628
  await fetchState();
 
629
  const savedLabel = window.localStorage.getItem(STORAGE_KEYS.moveLabel);
630
  $recName.value = savedLabel || '';
 
631
  } catch (e) {
632
  console.error('Record error:', e);
633
  } finally {
 
635
  }
636
  }
637
 
638
+ /* Record Audio — mic-only recording (no motion capture).
639
+ Simple flow: click → countdown → mic records for duration → save. */
640
+ async function queueAudioRecording() {
641
+ if (busy) return;
642
+ const dur = parseFloat($recDuration.value) || parseFloat($recDuration.placeholder) || 5;
643
+ const label = $recName.value.trim() || null;
644
+ requestedDuration = dur;
645
+ if ($recordAudioBtn) $recordAudioBtn.disabled = true;
646
+ try {
647
+ const r = await fetch('/api/record', {
648
+ method: 'POST',
649
+ headers: { 'Content-Type': 'application/json' },
650
+ body: JSON.stringify({
651
+ duration: dur,
652
+ record_audio: true,
653
+ record_motion: false,
654
+ label,
655
+ }),
656
+ });
657
+ if (!r.ok) throw new Error(await r.text());
658
+ movesListDirty = true;
659
+ await fetchState();
660
+ // Auto-switch to "Robot file" source and select the new recording
661
+ const robotRadio = document.querySelector('input[name="audio-src"][value="robot"]');
662
+ if (robotRadio) {
663
+ robotRadio.checked = true;
664
+ robotRadio.dispatchEvent(new Event('change', { bubbles: true }));
665
+ // Wait for file list to load, then auto-select newest (first) entry
666
+ await new Promise(r => setTimeout(r, 1500));
667
+ if ($robotAudioSelect && $robotAudioSelect.options.length > 0) {
668
+ $robotAudioSelect.selectedIndex = 0;
669
+ await selectRobotAudio();
670
+ // Match duration to the audio recording
671
+ if ($recDuration) $recDuration.value = dur;
672
+ }
673
+ }
674
+ } catch (e) {
675
+ console.error('Record audio error:', e);
676
+ } finally {
677
+ if ($recordAudioBtn) $recordAudioBtn.disabled = false;
678
+ }
679
+ }
680
+
681
  async function queuePlayback(id) {
682
  if (busy) return;
683
  try {
 
696
  stopping = true;
697
  try {
698
  const ctrl = new AbortController();
699
+ const tid = setTimeout(() => ctrl.abort(), 3000);
700
  await fetch('/api/play/stop', { method: 'POST', signal: ctrl.signal });
701
  clearTimeout(tid);
702
  await fetchState();
 
710
  $phaseStopBtn.disabled = true;
711
  try {
712
  const ctrl = new AbortController();
713
+ const tid = setTimeout(() => ctrl.abort(), 3000);
714
  await fetch('/api/record/stop', { method: 'POST', signal: ctrl.signal });
715
  clearTimeout(tid);
716
+ movesListDirty = true;
717
  await fetchState();
718
  } catch (e) { console.error(e); }
719
  finally { stopping = false; $phaseStopBtn.disabled = false; }
 
726
  const r = await fetch('/api/moves/' + encodeURIComponent(id), { method: 'DELETE' });
727
  if (!r.ok) throw new Error(await r.text());
728
  selectedMoves.delete(id);
729
+ movesListDirty = true;
730
  await fetchState();
731
  } catch (e) { console.error('Delete error:', e); }
732
  }
 
741
  });
742
  if (!r.ok) throw new Error(await r.text());
743
  selectedMoves.clear();
744
+ movesListDirty = true;
745
+ datasetsDirty = true;
746
  await fetchState();
747
  } catch (e) { console.error(e); }
748
  finally { datasetSwitching = false; }
 
760
  const d = await r.json().catch(() => ({}));
761
  throw new Error(d.detail || r.statusText);
762
  }
763
+ datasetsDirty = true;
764
+ movesListDirty = true;
765
  await fetchState();
766
  $newDatasetInline.classList.remove('active');
767
  $newDatasetBtn.style.display = '';
 
781
  });
782
  if (!r.ok) throw new Error(await r.text());
783
  selectedMoves.clear();
784
+ movesListDirty = true;
785
  await fetchState();
786
  } catch (e) {
787
  console.error(e);
 
803
  body: JSON.stringify({ path }),
804
  });
805
  if (!r.ok) throw new Error(await r.text());
806
+ datasetsDirty = true;
807
+ movesListDirty = true;
808
  await fetchState();
809
  } catch (e) { console.error(e); }
810
  }
 
820
  const data = await r.json();
821
  uploadedAudioId = data.upload_id;
822
  uploadedAudioFilename = data.filename;
823
+ $audioUploadStatus.textContent = `Ready: ${data.filename}` +
824
+ (data.duration ? ` (${data.duration.toFixed(1)}s)` : '');
825
+ if (data.duration && $recDuration) {
826
+ $recDuration.value = (Math.round(data.duration * 10) / 10).toFixed(1);
827
+ }
828
  } catch (e) {
829
  $audioUploadStatus.textContent = 'Failed: ' + e.message;
830
  uploadedAudioId = null;
831
  }
832
+ updateAudioSelectionStatus();
833
+ }
834
+
835
+ /* Update the "Audio: ..." indicator below the move name field. */
836
+ function updateAudioSelectionStatus() {
837
+ if (!$audioSelectionStatus) return;
838
+ const src = document.querySelector('input[name="audio-src"]:checked')?.value;
839
+ if (src === 'silent' || !src) {
840
+ $audioSelectionStatus.textContent = 'Audio: Silent';
841
+ } else if (uploadedAudioFilename) {
842
+ $audioSelectionStatus.textContent = `Audio: ${uploadedAudioFilename}`;
843
+ } else {
844
+ $audioSelectionStatus.textContent = 'Audio: None selected';
845
+ }
846
+ }
847
+
848
+ /* Load the list of audio files available on the robot. */
849
+ async function loadRobotAudioFiles() {
850
+ if (!$robotAudioSelect) return;
851
+ $robotAudioSelect.innerHTML = '<option value="">Loading...</option>';
852
+ try {
853
+ const r = await fetch('/api/robot-audio', { cache: 'no-store' });
854
+ const data = await r.json();
855
+ $robotAudioSelect.innerHTML = '';
856
+ const files = data.files || [];
857
+ if (!files.length) {
858
+ $robotAudioSelect.innerHTML = '<option value="">No audio files found</option>';
859
+ return;
860
+ }
861
+ files.forEach(f => {
862
+ const opt = document.createElement('option');
863
+ opt.value = f.path;
864
+ opt.textContent = f.name + (f.duration_seconds ? ` (${f.duration_seconds.toFixed(1)}s)` : '');
865
+ $robotAudioSelect.appendChild(opt);
866
+ });
867
+ // Auto-select the first file so uploadedAudioId is set immediately.
868
+ // Without this, the user would need to manually click "Use" before recording.
869
+ if (files.length > 0) selectRobotAudio();
870
+ } catch (e) {
871
+ $robotAudioSelect.innerHTML = '<option value="">Error loading files</option>';
872
+ console.error('Robot audio error:', e);
873
+ }
874
+ }
875
+
876
+ /* Select an audio file already on the robot for use in recording. */
877
+ async function selectRobotAudio() {
878
+ const path = $robotAudioSelect?.value;
879
+ if (!path) { updateAudioSelectionStatus(); return; }
880
+ try {
881
+ const r = await fetch('/api/robot-audio/select', {
882
+ method: 'POST',
883
+ headers: { 'Content-Type': 'application/json' },
884
+ body: JSON.stringify({ path }),
885
+ });
886
+ if (!r.ok) throw new Error((await r.json().catch(() => ({}))).detail || r.statusText);
887
+ const data = await r.json();
888
+ uploadedAudioId = data.upload_id;
889
+ uploadedAudioFilename = data.filename;
890
+ if (data.duration && $recDuration) {
891
+ $recDuration.value = (Math.round(data.duration * 10) / 10).toFixed(1);
892
+ }
893
+ } catch (e) {
894
+ console.error('Robot audio select error:', e);
895
+ }
896
+ updateAudioSelectionStatus();
897
  }
898
 
899
 
 
968
  }
969
  communitySelection.clear();
970
  $communityStatus.textContent = 'Download complete.';
971
+ movesListDirty = true;
972
+ datasetsDirty = true;
973
  await fetchState();
974
  } catch (e) {
975
  $communityStatus.textContent = 'Error: ' + e.message;
 
988
  const modelsPayload = config.motion_models;
989
  if (!modelsPayload) return;
990
  const params = modelsPayload.params?.lead_compensation || {};
991
+ if ($leadFramesHead && document.activeElement !== $leadFramesHead &&
992
+ Number.isFinite(params.lead_frames_head)) {
993
  $leadFramesHead.value = String(params.lead_frames_head);
994
  }
995
+ if ($leadFramesAntennas && document.activeElement !== $leadFramesAntennas &&
996
+ Number.isFinite(params.lead_frames_antennas)) {
 
 
 
997
  $leadFramesAntennas.value = String(params.lead_frames_antennas);
998
  }
999
  }
 
1003
  // SECTION 10: EVENT LISTENERS
1004
  // ═══════════════════════════════════════════════════════════════════════
1005
 
1006
+ /* Record Move button — start or stop */
1007
  $recordBtn.addEventListener('click', () => {
1008
+ if (busy && ['preparing', 'countdown', 'recording'].includes(phaseMode)) {
1009
  stopRecording();
1010
  } else {
1011
  queueRecording();
1012
  }
1013
  });
1014
 
1015
+ /* Record Audio button — mic-only recording */
1016
+ if ($recordAudioBtn) {
1017
+ $recordAudioBtn.addEventListener('click', () => {
1018
+ if (busy && ['preparing', 'countdown', 'recording'].includes(phaseMode)) {
1019
+ stopRecording();
1020
+ } else {
1021
+ queueAudioRecording();
1022
+ }
1023
+ });
1024
+ }
1025
+
1026
+ /* Phase overlay stop button — dispatch based on current mode */
1027
+ $phaseStopBtn.addEventListener('click', () => {
1028
+ if (phaseMode === 'playing') stopPlayback();
1029
+ else stopRecording();
1030
+ });
1031
 
1032
+ /* Dataset selector — My Moves tab */
1033
  $datasetSelect.addEventListener('change', e => {
1034
  if (e.target.value) selectDataset(e.target.value);
1035
  });
1036
+ /* Dataset selector — Library tab */
1037
+ if ($libraryDatasetSelect) {
1038
+ $libraryDatasetSelect.addEventListener('change', e => {
1039
+ if (e.target.value) selectDataset(e.target.value);
1040
+ });
1041
+ }
1042
 
1043
  /* New dataset creation (inline form) */
1044
  $newDatasetBtn.addEventListener('click', () => {
 
1056
  if (e.key === 'Enter') { e.preventDefault(); createDataset($newDatasetInput.value.trim()); }
1057
  if (e.key === 'Escape') { $newDatasetInline.classList.remove('active'); $newDatasetBtn.style.display = ''; }
1058
  });
 
1059
  $newDatasetInput.addEventListener('input', e => {
1060
  e.target.value = e.target.value.toLowerCase().replace(/\s+/g, '_');
1061
  });
 
1080
  /* Dataset root update */
1081
  $updateRootBtn.addEventListener('click', updateDatasetRoot);
1082
 
1083
+ /* Audio source radio buttons — show/hide the relevant sub-panel */
1084
  document.querySelectorAll('input[name="audio-src"]').forEach(r => {
1085
  r.addEventListener('change', () => {
1086
+ if ($audioUploadSection) $audioUploadSection.style.display =
1087
+ (r.value === 'upload' && r.checked) ? '' : 'none';
1088
+ if ($robotAudioSection) $robotAudioSection.style.display =
1089
+ (r.value === 'robot' && r.checked) ? '' : 'none';
1090
+ // Clear audio selection when switching between ANY source type.
1091
+ // This prevents stale uploadedAudioId from a previous source.
1092
+ uploadedAudioId = null;
1093
+ uploadedAudioFilename = null;
1094
+ if ($audioUploadStatus) $audioUploadStatus.textContent = '';
1095
+ // Auto-load robot audio files when selecting "Robot file"
1096
+ if (r.value === 'robot' && r.checked) loadRobotAudioFiles();
1097
+ updateAudioSelectionStatus();
1098
  });
1099
  });
1100
 
1101
  /* Audio upload zone — click and drag-and-drop */
1102
+ if ($audioUploadZone) {
1103
+ $audioUploadZone.addEventListener('click', () => $audioFileInput?.click());
1104
+ }
1105
+ if ($audioFileInput) {
1106
+ $audioFileInput.addEventListener('change', e => {
1107
+ if (e.target.files?.[0]) uploadAudioFile(e.target.files[0]);
1108
+ });
1109
+ }
1110
+ if ($audioUploadZone) {
1111
+ $audioUploadZone.addEventListener('dragover', e => {
1112
+ e.preventDefault();
1113
+ $audioUploadZone.classList.add('drag-over');
1114
+ });
1115
+ $audioUploadZone.addEventListener('dragleave', e => {
1116
+ e.preventDefault();
1117
+ $audioUploadZone.classList.remove('drag-over');
1118
+ });
1119
+ $audioUploadZone.addEventListener('drop', e => {
1120
+ e.preventDefault();
1121
+ $audioUploadZone.classList.remove('drag-over');
1122
+ const f = e.dataTransfer?.files?.[0];
1123
+ if (f && /\.(wav|mp3)$/i.test(f.name)) uploadAudioFile(f);
1124
+ });
1125
+ }
1126
+
1127
+ /* Robot audio select — auto-select on dropdown change + manual "Use" button */
1128
+ if ($robotAudioSelect) {
1129
+ $robotAudioSelect.addEventListener('change', selectRobotAudio);
1130
+ }
1131
 
1132
  /* HF login/logout */
1133
  $hfLoginBtn.addEventListener('click', async () => {
 
1209
  }, 500);
1210
  });
1211
 
1212
+ /* Tab switching — auto-switch datasets and auto-fetch community */
1213
  document.querySelectorAll('.section-tab').forEach(tab => {
1214
  tab.addEventListener('click', () => {
1215
  document.querySelectorAll('.section-tab').forEach(t => t.classList.remove('active'));
1216
  document.querySelectorAll('.tab-panel').forEach(p => p.classList.remove('active'));
1217
  tab.classList.add('active');
1218
  document.getElementById('tab-' + tab.dataset.tab)?.classList.add('active');
1219
+
1220
+ // My Moves tab: restore last local dataset if currently on a downloaded one
1221
+ if (tab.dataset.tab === 'my-moves' && lastLocalDatasetId) {
1222
+ const ae = lastState?.datasets?.entries?.find(
1223
+ e => e.id === lastState?.datasets?.active_id
1224
+ );
1225
+ if (ae?.origin === 'downloaded') {
1226
+ selectDataset(lastLocalDatasetId);
1227
+ }
1228
+ }
1229
+
1230
+ // Library tab: auto-select first downloaded dataset
1231
+ if (tab.dataset.tab === 'library') {
1232
+ const downloaded = (lastState?.datasets?.entries || [])
1233
+ .filter(e => e.origin === 'downloaded');
1234
+ if (downloaded.length) {
1235
+ const cur = lastState?.datasets?.active_id;
1236
+ if (!downloaded.some(d => d.id === cur)) {
1237
+ selectDataset(downloaded[0].id);
1238
+ }
1239
+ }
1240
+ }
1241
+
1242
+ if (tab.dataset.tab === 'community' && $communityList.childElementCount === 0) {
1243
+ fetchCommunity();
1244
+ }
1245
  });
1246
  });
1247
 
1248
  /* Community */
 
1249
  $downloadCommunityBtn.addEventListener('click', downloadCommunity);
1250
 
1251
  /* Name field — persist to localStorage */
 
1267
  // SECTION 11: INIT
1268
  // ═══════════════════════════════════════════════════════════════════════
1269
 
 
1270
  startPolling();
marionette/static/style.css CHANGED
@@ -338,6 +338,14 @@ input[type=number]::-webkit-inner-spin-button,
338
  input[type=number]::-webkit-outer-spin-button { -webkit-appearance: none; }
339
  input[type=number] { -moz-appearance: textfield; }
340
 
 
 
 
 
 
 
 
 
341
  /* Small hint text below the fields */
342
  .record-hint {
343
  font-size: .75rem;
@@ -1008,4 +1016,136 @@ input[type=number] { -moz-appearance: textfield; }
1008
  .fade-in-1 { animation-delay: .05s; }
1009
  .fade-in-2 { animation-delay: .1s; }
1010
  .fade-in-3 { animation-delay: .15s; }
1011
- </style>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
338
  input[type=number]::-webkit-outer-spin-button { -webkit-appearance: none; }
339
  input[type=number] { -moz-appearance: textfield; }
340
 
341
+ /* Audio selection indicator below record fields */
342
+ .audio-selection-status {
343
+ font-size: .8rem;
344
+ color: var(--text-muted);
345
+ text-align: center;
346
+ margin: .25rem 0 0;
347
+ }
348
+
349
  /* Small hint text below the fields */
350
  .record-hint {
351
  font-size: .75rem;
 
1016
  .fade-in-1 { animation-delay: .05s; }
1017
  .fade-in-2 { animation-delay: .1s; }
1018
  .fade-in-3 { animation-delay: .15s; }
1019
+
1020
+
1021
+ /* ═══════════════════════════════════════════════════════════════════
1022
+ RECORD ACTIONS — Two-button layout: "Record Move" + "Record Audio"
1023
+
1024
+ The two buttons sit side-by-side on wide screens and stack on
1025
+ mobile. The audio source panel appears inline below Record Move.
1026
+ ═══════════════════════════════════════════════════════════════════ */
1027
+ .record-actions {
1028
+ display: flex;
1029
+ gap: 2rem;
1030
+ align-items: flex-start;
1031
+ justify-content: center;
1032
+ flex-wrap: wrap;
1033
+ width: 100%;
1034
+ }
1035
+
1036
+ .record-action-block {
1037
+ display: flex;
1038
+ flex-direction: column;
1039
+ align-items: center;
1040
+ gap: .75rem;
1041
+ }
1042
+
1043
+ /* Audio button: green instead of red to visually differentiate */
1044
+ .record-btn-audio {
1045
+ background: linear-gradient(145deg, #16a34a, #15803d);
1046
+ box-shadow: 0 8px 40px var(--green-glow), inset 0 1px 0 rgba(255,255,255,.15);
1047
+ }
1048
+ .record-btn-audio:hover {
1049
+ box-shadow: 0 12px 60px var(--green-glow), inset 0 1px 0 rgba(255,255,255,.2);
1050
+ }
1051
+ .record-ring-audio::before {
1052
+ border-color: var(--border);
1053
+ }
1054
+ .record-ring-audio:hover::before {
1055
+ border-color: var(--green);
1056
+ box-shadow: 0 0 40px var(--green-glow);
1057
+ }
1058
+ .rec-icon-audio {
1059
+ font-size: 2rem;
1060
+ line-height: 1;
1061
+ }
1062
+ .record-hint-small {
1063
+ font-size: .7rem;
1064
+ color: var(--text-muted);
1065
+ opacity: .6;
1066
+ text-align: center;
1067
+ }
1068
+ .record-audio-hint {
1069
+ font-size: .65rem;
1070
+ color: var(--text-muted);
1071
+ opacity: .5;
1072
+ text-align: center;
1073
+ max-width: 14rem;
1074
+ margin: .25rem auto 0;
1075
+ line-height: 1.3;
1076
+ }
1077
+
1078
+ /* ═══════════════════════════════════════════════════════════════════
1079
+ AUDIO SOURCE PANEL — Inline below the Record Move button.
1080
+
1081
+ Shows pill-shaped radio tabs: Silent | Upload | Robot file.
1082
+ Only one sub-panel is visible at a time (controlled by JS).
1083
+ ═══════════════════════════════════════════════════════════════════ */
1084
+ .audio-source-panel {
1085
+ display: flex;
1086
+ flex-direction: column;
1087
+ gap: .65rem;
1088
+ width: 100%;
1089
+ max-width: 360px;
1090
+ text-align: center;
1091
+ }
1092
+ .audio-source-label {
1093
+ font-size: .7rem;
1094
+ text-transform: uppercase;
1095
+ letter-spacing: .1em;
1096
+ color: var(--text-muted);
1097
+ font-weight: 600;
1098
+ }
1099
+ .audio-source-hint {
1100
+ font-size: .65rem;
1101
+ color: var(--text-muted);
1102
+ opacity: .55;
1103
+ margin: -.25rem 0 0;
1104
+ line-height: 1.35;
1105
+ }
1106
+ .audio-source-tabs {
1107
+ display: flex;
1108
+ gap: .4rem;
1109
+ flex-wrap: wrap;
1110
+ justify-content: center;
1111
+ }
1112
+ .audio-tab {
1113
+ padding: .35rem .75rem;
1114
+ border-radius: 999px;
1115
+ background: var(--surface);
1116
+ border: 1px solid var(--border);
1117
+ font-size: .75rem;
1118
+ cursor: pointer;
1119
+ transition: all .15s;
1120
+ display: flex;
1121
+ align-items: center;
1122
+ gap: .3rem;
1123
+ }
1124
+ .audio-tab:has(input:checked) {
1125
+ border-color: var(--accent);
1126
+ background: rgba(225,29,72,.06);
1127
+ color: var(--text);
1128
+ }
1129
+ .audio-tab input {
1130
+ accent-color: var(--accent);
1131
+ width: 12px;
1132
+ height: 12px;
1133
+ }
1134
+
1135
+ /* Inline row layout for audio source sections (robot file selector, etc.) */
1136
+ .audio-source-row {
1137
+ display: flex;
1138
+ gap: .5rem;
1139
+ align-items: center;
1140
+ }
1141
+ .audio-source-row .settings-input,
1142
+ .audio-source-row .dataset-select {
1143
+ flex: 1;
1144
+ padding: .5rem .75rem;
1145
+ font-size: .8rem;
1146
+ }
1147
+
1148
+ /* Mobile: stack the two record-action blocks */
1149
+ @media (max-width: 600px) {
1150
+ .record-actions { gap: 1.5rem; }
1151
+ }
marionette/useless_but_emotional/README.md ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ # Useless but emotional
2
+
3
+ Code that's currently unused but might become useful later.
4
+ Each file explains why it was archived and when it might return.
marionette/useless_but_emotional/__init__.py ADDED
File without changes
marionette/useless_but_emotional/mic_agc.py ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Microphone AGC (Automatic Gain Control) management for ReSpeaker USB.
2
+
3
+ Disables the XMOS XVF3800 AGC on the ReSpeaker mic array to prevent
4
+ motor noise from causing the mic to "mute" during recording. The AGC
5
+ aggressively scales sensitivity to avoid saturation, which backfires
6
+ when motors are loud — the mic goes nearly silent.
7
+
8
+ WHY ARCHIVED:
9
+ We no longer record microphone audio during motion capture (the motor
10
+ noise was too disruptive anyway). Audio is now recorded separately
11
+ via the "Record Audio" button, or provided as an uploaded file.
12
+ Without simultaneous mic+motion, the AGC conflict doesn't occur.
13
+
14
+ WHEN TO RESTORE:
15
+ If we re-enable simultaneous microphone + motion recording (e.g. with
16
+ quieter motors or better noise cancellation), bring these functions
17
+ back into app.py and call them from run().
18
+
19
+ USAGE (when active):
20
+ Called in Marionette.run():
21
+ self._disable_mic_agc() # at start
22
+ self._restore_mic_agc() # in finally block
23
+ """
24
+
25
+ import logging
26
+
27
+ logger = logging.getLogger(__name__)
28
+
29
+
30
+ def disable_mic_agc():
31
+ """Disable the mic's automatic gain control for cleaner recordings."""
32
+ try:
33
+ from reachy_mini.media.audio_control_utils import init_respeaker_usb
34
+ except ImportError:
35
+ logger.debug("audio_control_utils not available, skipping AGC config")
36
+ return None
37
+
38
+ try:
39
+ respeaker = init_respeaker_usb()
40
+ if respeaker is None:
41
+ logger.debug("No ReSpeaker USB device found, skipping AGC config")
42
+ return None
43
+ original_agc = respeaker.read("PP_AGCONOFF")
44
+ respeaker.write("PP_AGCONOFF", [0])
45
+ respeaker.close()
46
+ logger.info("Mic AGC disabled (was %s)", original_agc)
47
+ return original_agc
48
+ except Exception as exc:
49
+ logger.warning("Failed to disable mic AGC: %s", exc)
50
+ return None
51
+
52
+
53
+ def restore_mic_agc(original_agc):
54
+ """Restore the mic AGC to its original value."""
55
+ if original_agc is None:
56
+ return
57
+ try:
58
+ from reachy_mini.media.audio_control_utils import init_respeaker_usb
59
+
60
+ respeaker = init_respeaker_usb()
61
+ if respeaker is None:
62
+ return
63
+ respeaker.write("PP_AGCONOFF", original_agc)
64
+ respeaker.close()
65
+ logger.info("Mic AGC restored to %s", original_agc)
66
+ except Exception as exc:
67
+ logger.warning("Failed to restore mic AGC: %s", exc)
pyproject.toml CHANGED
@@ -14,6 +14,7 @@ dependencies = [
14
  "soundfile",
15
  "huggingface-hub==0.34.4",
16
  "python-multipart",
 
17
  ]
18
  keywords = ["reachy-mini-app"]
19
 
 
14
  "soundfile",
15
  "huggingface-hub==0.34.4",
16
  "python-multipart",
17
+ "yt-dlp",
18
  ]
19
  keywords = ["reachy-mini-app"]
20
 
tests/SYNC_TEST_REPORT.md ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Audio-Motion Synchronization Test Report
2
+
3
+ ## Summary
4
+
5
+ Comprehensive testing of audio-motion sync across the Reachy Mini Marionette stack.
6
+ Tests measure the time delay between audio (beeps) and physical motion (antenna collisions)
7
+ using a laptop microphone to capture both sounds from the same clock source.
8
+
9
+ **Final result**: After identifying and compensating a 320ms `push_audio_sample` pipeline
10
+ latency, Marionette playback sync is now **-4ms mean error, 5ms std** — essentially perfect.
11
+
12
+ ## Test Progression
13
+
14
+ ### 1. Audio Roundtrip (`test_audio_roundtrip.py`)
15
+ **Purpose**: Verify beeps can be played on the robot and detected by the laptop mic.
16
+
17
+ - 5 beeps at non-periodic times [0.5, 1.8, 3.5, 5.8, 8.9]s
18
+ - 2kHz frequency, 200ms duration, 0.9 amplitude
19
+ - Robot plays at 16kHz via `push_audio_sample`
20
+ - **Result**: 5/5 beeps detected, sub-millisecond accuracy
21
+
22
+ ### 2. Antenna Collision Position Test (`test_antenna_collision.py`)
23
+ **Purpose**: Verify commanded collisions can be detected from motor position feedback.
24
+
25
+ - Right antenna fixed at -0.68 rad, left swings to 0.70 rad
26
+ - Physical contact at ~0.60 rad, with overshoot to ~0.77 rad
27
+ - PID + comms latency: ~120-160ms from command to peak position
28
+ - **Result**: 5/5 collisions detected from position data
29
+
30
+ ### 3. Collision + Mic Test (`test_collision_mic.py`)
31
+ **Purpose**: Verify collisions can be detected from laptop microphone audio.
32
+
33
+ - Same collision sequence, recorded via laptop mic
34
+ - Mean delay command -> audible collision: +39ms (std 13ms)
35
+ - **Caveat**: Cross-clock measurement (robot clock vs laptop clock). Only relative
36
+ consistency is trustworthy, not absolute delay.
37
+ - **Result**: 5/5 collisions detected via mic
38
+
39
+ ### 4. Combined Beep + Collision Sync (`test_beep_collision_sync.py`)
40
+ **Purpose**: THE KEY TEST. Measure audio pipeline latency using same-clock measurement.
41
+
42
+ Both beeps and collisions are commanded in the same 50Hz loop on the robot.
43
+ Both are detected from the same laptop microphone = no cross-clock bias.
44
+ Beeps at [1.0, 2.3, 4.0, 6.3, 9.4], collisions exactly 1.0s after each.
45
+
46
+ **Critical Finding**: `push_audio_sample` adds ~245ms pipeline latency.
47
+ - Measured interval: 755ms (expected 1000ms)
48
+ - Error: -245ms mean, 6ms std (extremely consistent)
49
+ - This latency is inherent to the chunk-based audio pipeline, not a warm-up issue
50
+ (all 5 pairs show the same delay, including the first)
51
+
52
+ ### 5. SDK Move Sync (`test_move_sync.py`)
53
+ **Purpose**: Test audio-motion sync using SDK's `play_move()` + `play_sound()`.
54
+
55
+ Creates a Marionette-format move (JSON + WAV) and plays it via the SDK's built-in
56
+ `play_move()` method, which internally calls `play_sound()` (file-based audio).
57
+
58
+ | Trial | Mean error | Std | Notes |
59
+ |-------|-----------|------|----------------------|
60
+ | 1 | -5ms* | 19ms | *excl. 1 outlier |
61
+ | 2 | -22ms | 5ms | Excellent, no outliers|
62
+
63
+ **Critical Finding**: `play_sound()` (file-based) has essentially zero latency (~22ms),
64
+ vs `push_audio_sample()` (chunk-based) at ~245ms.
65
+
66
+ ### 6. Marionette E2E Playback Test (`test_marionette_sync.py --test playback`)
67
+ **Purpose**: Test through Marionette's full playback pipeline.
68
+
69
+ Injects a synthetic move (beeps + antenna collisions) into Marionette's active dataset,
70
+ triggers playback via `POST /api/play`, records both sounds via laptop mic.
71
+
72
+ #### Before compensation (AUDIO_LEAD_MS = 0)
73
+
74
+ | Trial | Mean error | Std | Notes |
75
+ |-------|-----------|------|-------|
76
+ | 1 | -320ms | 7ms | 5/5 pairs |
77
+ | 2 | -321ms | 11ms | 5/5 pairs |
78
+
79
+ The extra 75ms beyond raw pipeline latency (245ms) comes from Marionette's
80
+ `start_signal` thread coordination: audio thread wakes up after the first motor
81
+ command, then pushes through the 245ms pipeline.
82
+
83
+ #### After compensation (AUDIO_LEAD_MS = 320)
84
+
85
+ | Trial | Mean error | Std | Notes |
86
+ |-------|-----------|------|-------|
87
+ | 1 | **-4ms** | 5ms | 5/5 pairs, essentially perfect |
88
+
89
+ Audio thread now starts 320ms before the first motion command. By the time chunks
90
+ traverse the GStreamer pipeline, the first audible output coincides with the first
91
+ physical movement.
92
+
93
+ ### 7. Marionette Recording Countdown Test (`test_marionette_sync.py --test recording`)
94
+ **Purpose**: Verify the 3-2-1 countdown timing and go-to-audio delay.
95
+
96
+ Uploads a test WAV (2kHz marker beep at t=0.2s), triggers recording via
97
+ `POST /api/record`, records countdown beeps via laptop mic.
98
+
99
+ **Results** (after fixing beep sample rate bug):
100
+ - 3 countdown beeps detected at 440Hz: gaps of 954ms, 960ms (expected 1000ms)
101
+ - Go beep (880Hz) detected correctly after countdown
102
+ - Marker beep (2kHz) detected at 297ms after go beep
103
+ - 200ms = marker position in audio file
104
+ - 97ms = remaining pipeline overhead for uploaded audio start
105
+
106
+ **Note**: Detection is sensitive to ambient noise. 440Hz/880Hz are in a range with
107
+ significant motor noise and room noise. Tests should be run in a quiet environment.
108
+
109
+ ## Bugs Found and Fixed
110
+
111
+ ### 1. `generate_beep` sample rate mismatch
112
+ **File**: `marionette/recording.py`
113
+
114
+ `generate_beep()` defaults to `sr=48000` but `push_audio_sample` expects 16kHz data.
115
+ Countdown beeps were playing at 1/3 the intended frequency (147Hz instead of 440Hz)
116
+ and 3x the intended duration (300ms instead of 100ms).
117
+
118
+ **Fix**: Pass `sr=sr_out` (the robot's output sample rate) to `generate_beep()`.
119
+
120
+ ### 2. Dataset re-download blocked
121
+ **File**: `marionette/datasets.py`
122
+
123
+ Downloading a dataset that already existed locally raised HTTP 409, preventing updates.
124
+
125
+ **Fix**: Remove existing entry and folder before re-downloading.
126
+
127
+ ## Key Findings
128
+
129
+ ### Audio Pipeline Latency Comparison
130
+
131
+ | Method | API | Mean error | Std | Used by Marionette? |
132
+ |----------------------------|-------------------|-----------|------|---------------------|
133
+ | Direct chunk pushing | push_audio_sample | **-245ms** | 6ms | **YES** (playback + recording) |
134
+ | SDK file playback | play_sound | **-22ms** | 5ms | NO (not interruptible) |
135
+ | Marionette E2E (no comp.) | push_audio_sample | **-320ms** | 9ms | Before fix |
136
+ | Marionette E2E (with comp.)| push_audio_sample | **-4ms** | 5ms | **After fix** |
137
+
138
+ ### Why Marionette Uses push_audio_sample
139
+
140
+ Marionette deliberately uses `push_audio_sample` instead of `play_sound` because:
141
+
142
+ 1. **Cancellability**: `play_sound()` creates a GStreamer pipeline that cannot be interrupted.
143
+ `push_audio_sample` loops can be stopped at any time via `stop_event.set()`.
144
+ (See `marionette/audio.py:61-65` docstring)
145
+
146
+ 2. **play_sound stoppability fix not merged**: A fix exists on branch
147
+ `892-stop_playing-has-no-effect-on-audio-started-by-play_sound-gstreamer-backend`
148
+ (authored by RemiFabre, 2026-02-24) but is not yet merged.
149
+
150
+ 3. **cancel_move not merged**: SDK's `cancel_move()` exists on branch
151
+ `make-dance-cancellable` but was never merged to main.
152
+
153
+ ### Compensation Mechanism (AUDIO_LEAD_MS)
154
+
155
+ **Constant**: `AUDIO_LEAD_MS = 320` in `marionette/models.py`
156
+
157
+ The playback flow now starts audio 320ms before the first motion command:
158
+
159
+ 1. Load JSON trajectory, apply lead compensation
160
+ 2. Preload WAV + resample to 16kHz
161
+ 3. Goto start pose
162
+ 4. Warm GStreamer: `start_playing()` + push 160 silence samples
163
+ 5. Spawn audio thread: `play_preloaded_wav()` waiting on `start_signal`
164
+ 6. **Fire `start_signal` immediately** → audio thread starts pushing chunks
165
+ 7. **Sleep 320ms** → audio fills the GStreamer pipeline
166
+ 8. `_stream_playback()`: first motion command coincides with first audible output
167
+ 9. On cancel: `audio_stop.set()` → audio thread exits
168
+
169
+ The 320ms = 245ms GStreamer pipeline latency + ~75ms thread-scheduling overhead.
170
+
171
+ ## Files Created/Modified
172
+
173
+ ### Production Code Changes
174
+ - `marionette/recording.py` — Fixed `generate_beep` sample rate, added AUDIO_LEAD_MS compensation
175
+ - `marionette/datasets.py` — Allow re-downloading existing datasets
176
+ - `marionette/models.py` — Added `AUDIO_LEAD_MS = 320` constant
177
+
178
+ ### Test Scripts
179
+ - `tests/test_audio_roundtrip.py` - Standalone beep roundtrip test
180
+ - `tests/test_antenna_collision.py` - Position-based collision test
181
+ - `tests/test_collision_mic.py` - Collision + mic delay test
182
+ - `tests/test_beep_collision_sync.py` - Combined sync test (KEY measurement)
183
+ - `tests/test_move_sync.py` - SDK play_move sync test
184
+ - `tests/test_marionette_sync.py` - Marionette E2E sync test (playback + recording)
185
+ - `tests/audio_analysis.py` - Shared detection utilities
186
+ - `tests/antenna_gui.py` - Developer GUI for collision values
187
+
188
+ ### Output Files (gitignored, on disk)
189
+ - `tests/collision_trajectory.png` - Antenna position plot
190
+ - `tests/collision_mic_plot.png` - Collision mic timing plot
191
+ - `tests/beep_collision_sync_plot.png` - Key sync measurement plot
192
+ - `tests/move_sync_plot.png` - SDK play_move sync plot
193
+ - `tests/marionette_sync_plot.png` - Marionette E2E sync plot
194
+ - Various `.wav` and `.json` result files
195
+
196
+ ## Future Work
197
+
198
+ 1. **Re-run recording countdown test** in quiet environment for cleaner detection
199
+ 2. **Tune AUDIO_LEAD_MS** if hardware changes (different robot speaker, GStreamer version)
200
+ 3. **Switch to `play_move`/`play_sound`** once SDK cancel support is merged (eliminating
201
+ the need for compensation entirely — `play_sound` has only ~22ms latency)
tests/antenna_gui.py ADDED
@@ -0,0 +1,135 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """GUI to find antenna collision values on Reachy Mini.
3
+
4
+ Based on reachy_mini/examples/mini_head_position_gui.py.
5
+ Adds an antenna slider so the user can manually find the value
6
+ where the two antennas collide (producing a distinctive click sound).
7
+
8
+ Usage:
9
+ python tests/antenna_gui.py
10
+
11
+ Move the "Antennas" slider until you hear/see the antennas collide.
12
+ The current value is displayed and printed to stdout when you quit.
13
+ """
14
+ from __future__ import annotations
15
+
16
+ import tkinter as tk
17
+
18
+ import numpy as np
19
+ from scipy.spatial.transform import Rotation as R
20
+
21
+ from reachy_mini import ReachyMini
22
+ from reachy_mini.utils import create_head_pose
23
+
24
+
25
+ def main() -> None:
26
+ """Run GUI with head + antenna control."""
27
+ with ReachyMini(media_backend="no_media") as mini:
28
+ root = tk.Tk()
29
+ root.title("Antenna Collision Finder")
30
+
31
+ row = 0
32
+
33
+ # --- Head orientation ---
34
+ roll_var = tk.DoubleVar(value=0.0)
35
+ pitch_var = tk.DoubleVar(value=0.0)
36
+ yaw_var = tk.DoubleVar(value=0.0)
37
+
38
+ tk.Label(root, text="Roll (deg):").grid(row=row, column=0, sticky="e")
39
+ tk.Scale(root, variable=roll_var, from_=-45, to=45, orient=tk.HORIZONTAL, length=300).grid(row=row, column=1)
40
+ row += 1
41
+ tk.Label(root, text="Pitch (deg):").grid(row=row, column=0, sticky="e")
42
+ tk.Scale(root, variable=pitch_var, from_=-45, to=45, orient=tk.HORIZONTAL, length=300).grid(row=row, column=1)
43
+ row += 1
44
+ tk.Label(root, text="Yaw (deg):").grid(row=row, column=0, sticky="e")
45
+ tk.Scale(root, variable=yaw_var, from_=-175, to=175, orient=tk.HORIZONTAL, length=300).grid(row=row, column=1)
46
+ row += 1
47
+
48
+ # --- Body yaw ---
49
+ body_yaw_var = tk.DoubleVar(value=0.0)
50
+ tk.Label(root, text="Body Yaw (deg):").grid(row=row, column=0, sticky="e")
51
+ tk.Scale(root, variable=body_yaw_var, from_=-180, to=180, resolution=1.0, orient=tk.HORIZONTAL, length=300).grid(row=row, column=1)
52
+ row += 1
53
+
54
+ # --- Separator ---
55
+ tk.Frame(root, height=2, bd=1, relief=tk.SUNKEN).grid(row=row, column=0, columnspan=2, sticky="ew", pady=8)
56
+ row += 1
57
+
58
+ # --- Antenna sliders (independent left/right) ---
59
+ left_var = tk.DoubleVar(value=0.0)
60
+ right_var = tk.DoubleVar(value=0.0)
61
+
62
+ tk.Label(root, text="Left antenna (rad):", font=("Helvetica", 12, "bold")).grid(row=row, column=0, sticky="e")
63
+ tk.Scale(
64
+ root,
65
+ variable=left_var,
66
+ from_=-1.5,
67
+ to=1.5,
68
+ resolution=0.01,
69
+ orient=tk.HORIZONTAL,
70
+ length=300,
71
+ ).grid(row=row, column=1)
72
+ row += 1
73
+
74
+ tk.Label(root, text="Right antenna (rad):", font=("Helvetica", 12, "bold")).grid(row=row, column=0, sticky="e")
75
+ tk.Scale(
76
+ root,
77
+ variable=right_var,
78
+ from_=-1.5,
79
+ to=1.5,
80
+ resolution=0.01,
81
+ orient=tk.HORIZONTAL,
82
+ length=300,
83
+ ).grid(row=row, column=1)
84
+ row += 1
85
+
86
+ # Display current value prominently
87
+ value_label = tk.Label(root, text="antennas = [0.00, 0.00]", font=("Courier", 14))
88
+ value_label.grid(row=row, column=0, columnspan=2, pady=5)
89
+ row += 1
90
+
91
+ # Hint
92
+ tk.Label(
93
+ root,
94
+ text="Slide each antenna independently to find collision values.",
95
+ fg="gray",
96
+ ).grid(row=row, column=0, columnspan=2)
97
+ row += 1
98
+
99
+ mini.goto_target(create_head_pose(), antennas=[0.0, 0.0], duration=1.0)
100
+
101
+ def update_robot() -> None:
102
+ left = left_var.get()
103
+ right = right_var.get()
104
+ value_label.config(text=f"antennas = [{left:+.2f}, {right:+.2f}]")
105
+
106
+ head = np.eye(4)
107
+ roll = np.deg2rad(roll_var.get())
108
+ pitch = np.deg2rad(pitch_var.get())
109
+ yaw = np.deg2rad(yaw_var.get())
110
+ head[:3, :3] = R.from_euler("xyz", [roll, pitch, yaw], degrees=False).as_matrix()
111
+
112
+ mini.set_target(
113
+ head=head,
114
+ body_yaw=np.deg2rad(body_yaw_var.get()),
115
+ antennas=np.array([left, right]),
116
+ )
117
+
118
+ root.after(20, update_robot)
119
+
120
+ root.after(20, update_robot)
121
+
122
+ try:
123
+ root.mainloop()
124
+ except KeyboardInterrupt:
125
+ pass
126
+ finally:
127
+ left = left_var.get()
128
+ right = right_var.get()
129
+ print(f"\nFinal antenna values:")
130
+ print(f" antennas = [{left:+.2f}, {right:+.2f}]")
131
+ root.destroy()
132
+
133
+
134
+ if __name__ == "__main__":
135
+ main()
tests/audio_analysis.py ADDED
@@ -0,0 +1,290 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Audio analysis utilities for sync testing.
2
+
3
+ Generates test audio with beep markers, creates antenna collision
4
+ trajectories, and detects both beep onsets and transient events
5
+ in recorded audio for measuring audio-motion sync accuracy.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import numpy as np
10
+
11
+
12
+ # ──────── Test Signal Generation ──────────────────────────────────
13
+
14
+ # Non-periodic beep times so large offsets can't accidentally align.
15
+ DEFAULT_BEEP_TIMES = [1.0, 2.5, 4.0, 5.0, 7.0]
16
+
17
+
18
+ def generate_sync_test_audio(
19
+ sr: int = 48000,
20
+ duration: float = 8.0,
21
+ beep_times: list[float] | None = None,
22
+ beep_freq: float = 1000.0,
23
+ beep_duration: float = 0.1,
24
+ ) -> tuple[np.ndarray, list[float]]:
25
+ """Generate audio with tonal beeps at known timestamps.
26
+
27
+ Returns (audio_data, beep_timestamps).
28
+ Beeps are sine waves with fade-in/out to avoid clicks.
29
+ """
30
+ if beep_times is None:
31
+ beep_times = DEFAULT_BEEP_TIMES
32
+
33
+ n_total = int(sr * duration)
34
+ audio = np.zeros(n_total, dtype=np.float32)
35
+
36
+ for t in beep_times:
37
+ start_sample = int(t * sr)
38
+ n_beep = int(beep_duration * sr)
39
+ if start_sample + n_beep > n_total:
40
+ continue
41
+ t_arr = np.arange(n_beep, dtype=np.float32) / sr
42
+ beep = 0.5 * np.sin(2 * np.pi * beep_freq * t_arr).astype(np.float32)
43
+ # Fade in/out (5ms each)
44
+ fade = int(0.005 * sr)
45
+ if fade > 0 and 2 * fade < n_beep:
46
+ beep[:fade] *= np.linspace(0, 1, fade, dtype=np.float32)
47
+ beep[-fade:] *= np.linspace(1, 0, fade, dtype=np.float32)
48
+ audio[start_sample : start_sample + n_beep] += beep
49
+
50
+ return audio, list(beep_times)
51
+
52
+
53
+ def generate_collision_trajectory(
54
+ beep_times: list[float],
55
+ duration: float,
56
+ motion_sr: int = 100,
57
+ ) -> tuple[list[float], list[dict]]:
58
+ """Generate frames where antennas collide at each beep time.
59
+
60
+ Antennas start apart (±0.3 rad ≈ ±17°) and slam together
61
+ (0.0 rad) at each beep timestamp, then return apart.
62
+ Each collision takes ~200ms (100ms approach + 100ms return).
63
+
64
+ Returns (timestamps, frames).
65
+ """
66
+ from scipy.spatial.transform import Rotation as R
67
+
68
+ n = int(duration * motion_sr)
69
+ dt = 1.0 / motion_sr
70
+ rest_pos = 0.3 # rad, antennas apart
71
+
72
+ timestamps = []
73
+ frames = []
74
+ identity_pose = np.eye(4)
75
+
76
+ for i in range(n):
77
+ t = i * dt
78
+ timestamps.append(t)
79
+
80
+ # Compute antenna position: slam together at each beep time
81
+ antenna_val = rest_pos
82
+ for bt in beep_times:
83
+ approach_start = bt - 0.1 # 100ms before collision
84
+ return_end = bt + 0.1 # 100ms after collision
85
+
86
+ if approach_start <= t <= bt:
87
+ # Approaching: ease from rest to 0
88
+ frac = (t - approach_start) / 0.1
89
+ antenna_val = rest_pos * (1.0 - frac)
90
+ break
91
+ elif bt < t <= return_end:
92
+ # Returning: ease from 0 to rest
93
+ frac = (t - bt) / 0.1
94
+ antenna_val = rest_pos * frac
95
+ break
96
+
97
+ frames.append({
98
+ "head": identity_pose.tolist(),
99
+ "antennas": [-antenna_val, antenna_val],
100
+ "body_yaw": 0.0,
101
+ "check_collision": False,
102
+ })
103
+
104
+ return timestamps, frames
105
+
106
+
107
+ # ──────── Audio Analysis ──────────────────────────────────────────
108
+
109
+
110
+ def detect_beep_onsets(
111
+ audio: np.ndarray,
112
+ sr: int,
113
+ freq: float = 1000.0,
114
+ bandwidth: float = 200.0,
115
+ threshold_db: float = -20.0,
116
+ min_separation: float = 0.3,
117
+ ) -> list[float]:
118
+ """Detect onset times of tonal beeps using bandpass + envelope.
119
+
120
+ 1. Bandpass filter around target frequency
121
+ 2. Compute amplitude envelope via rectification + lowpass
122
+ 3. Rising-edge threshold crossing for onset detection
123
+ """
124
+ from scipy.signal import butter, sosfilt
125
+
126
+ # Bandpass around beep frequency
127
+ low = max(20, freq - bandwidth / 2) / (sr / 2)
128
+ high = min(0.99, (freq + bandwidth / 2) / (sr / 2))
129
+ sos = butter(4, [low, high], btype="bandpass", output="sos")
130
+ filtered = sosfilt(sos, audio.astype(np.float64))
131
+
132
+ # Amplitude envelope: rectify + lowpass at 50Hz
133
+ envelope = np.abs(filtered)
134
+ lp_freq = min(50.0 / (sr / 2), 0.99)
135
+ sos_lp = butter(2, lp_freq, btype="lowpass", output="sos")
136
+ envelope = sosfilt(sos_lp, envelope)
137
+
138
+ # Normalize and threshold
139
+ peak_val = np.max(envelope)
140
+ if peak_val < 1e-10:
141
+ return []
142
+ envelope /= peak_val
143
+ threshold = 10 ** (threshold_db / 20)
144
+
145
+ # Rising-edge threshold crossings (onset = first sample above threshold)
146
+ above = envelope > threshold
147
+ edges = np.diff(above.astype(np.int8))
148
+ onset_indices = np.where(edges > 0)[0] + 1
149
+
150
+ # Filter by minimum separation
151
+ min_distance = int(min_separation * sr)
152
+ if len(onset_indices) > 1:
153
+ filtered_indices = [onset_indices[0]]
154
+ for idx in onset_indices[1:]:
155
+ if idx - filtered_indices[-1] >= min_distance:
156
+ filtered_indices.append(idx)
157
+ onset_indices = filtered_indices
158
+
159
+ return [idx / sr for idx in onset_indices]
160
+
161
+
162
+ def detect_transient_onsets(
163
+ audio: np.ndarray,
164
+ sr: int,
165
+ highpass_freq: float = 2000.0,
166
+ threshold_db: float = -20.0,
167
+ ) -> list[float]:
168
+ """Detect impulsive sounds (antenna collisions) via spectral flux.
169
+
170
+ 1. High-pass filter to separate from tonal beeps
171
+ 2. Compute onset strength via spectral flux
172
+ 3. Peak detection for sharp transients
173
+ """
174
+ from scipy.signal import butter, sosfilt, find_peaks
175
+
176
+ # High-pass to isolate transients from tonal beeps
177
+ hp_freq = min(highpass_freq / (sr / 2), 0.99)
178
+ sos = butter(4, hp_freq, btype="highpass", output="sos")
179
+ filtered = sosfilt(sos, audio.astype(np.float64))
180
+
181
+ # Onset strength: short-term energy in 5ms windows
182
+ win_samples = max(1, int(0.005 * sr))
183
+ energy = np.array([
184
+ np.sum(filtered[i : i + win_samples] ** 2)
185
+ for i in range(0, len(filtered) - win_samples, win_samples)
186
+ ])
187
+
188
+ if len(energy) < 2:
189
+ return []
190
+
191
+ # Spectral flux: positive differences in energy
192
+ flux = np.diff(energy)
193
+ flux = np.maximum(flux, 0)
194
+
195
+ peak_val = np.max(flux)
196
+ if peak_val < 1e-10:
197
+ return []
198
+ flux /= peak_val
199
+ threshold = 10 ** (threshold_db / 20)
200
+
201
+ # Find peaks with minimum 200ms separation
202
+ min_distance = max(1, int(0.2 * sr / win_samples))
203
+ peaks, _ = find_peaks(flux, height=threshold, distance=min_distance)
204
+
205
+ # Convert window indices to seconds
206
+ return [(p * win_samples) / sr for p in peaks]
207
+
208
+
209
+ def measure_sync_offsets(
210
+ beep_onsets: list[float],
211
+ collision_onsets: list[float],
212
+ max_match_distance: float = 0.5,
213
+ ) -> dict:
214
+ """Match each beep to its nearest collision and compute offsets.
215
+
216
+ Returns dict with pairs, mean/max/std offset in milliseconds.
217
+ Positive offset means collision came AFTER beep.
218
+ """
219
+ pairs = []
220
+ remaining_collisions = list(collision_onsets)
221
+
222
+ for bt in beep_onsets:
223
+ if not remaining_collisions:
224
+ break
225
+ distances = [abs(ct - bt) for ct in remaining_collisions]
226
+ best_idx = int(np.argmin(distances))
227
+ if distances[best_idx] <= max_match_distance:
228
+ ct = remaining_collisions.pop(best_idx)
229
+ offset_ms = (ct - bt) * 1000.0
230
+ pairs.append((bt, ct, offset_ms))
231
+
232
+ offsets = [p[2] for p in pairs]
233
+ return {
234
+ "pairs": pairs,
235
+ "n_matched": len(pairs),
236
+ "n_beeps": len(beep_onsets),
237
+ "n_collisions": len(collision_onsets),
238
+ "mean_offset_ms": float(np.mean(offsets)) if offsets else float("nan"),
239
+ "max_offset_ms": float(np.max(np.abs(offsets))) if offsets else float("nan"),
240
+ "std_offset_ms": float(np.std(offsets)) if offsets else float("nan"),
241
+ }
242
+
243
+
244
+ # ──────── Mic Recording Helper ────────────────────────────────────
245
+
246
+
247
+ class MicRecorder:
248
+ """Record from laptop microphone using sounddevice.
249
+
250
+ Usage:
251
+ recorder = MicRecorder(sr=48000)
252
+ recorder.start()
253
+ # ... do stuff ...
254
+ audio = recorder.stop() # returns np.ndarray
255
+ """
256
+
257
+ def __init__(self, sr: int = 48000, channels: int = 1):
258
+ self.sr = sr
259
+ self.channels = channels
260
+ self._frames: list[np.ndarray] = []
261
+ self._stream = None
262
+
263
+ def start(self) -> None:
264
+ import sounddevice as sd
265
+
266
+ self._frames = []
267
+
268
+ def callback(indata, frames, time_info, status):
269
+ self._frames.append(indata.copy())
270
+
271
+ self._stream = sd.InputStream(
272
+ samplerate=self.sr,
273
+ channels=self.channels,
274
+ dtype="float32",
275
+ callback=callback,
276
+ )
277
+ self._stream.start()
278
+
279
+ def stop(self) -> np.ndarray:
280
+ if self._stream is not None:
281
+ self._stream.stop()
282
+ self._stream.close()
283
+ self._stream = None
284
+ if not self._frames:
285
+ return np.zeros(0, dtype=np.float32)
286
+ audio = np.concatenate(self._frames, axis=0)
287
+ # Return mono (first channel if multi-channel)
288
+ if audio.ndim > 1:
289
+ audio = audio[:, 0]
290
+ return audio
tests/e2e/test_ui.py CHANGED
@@ -118,14 +118,11 @@ class TestFormValidation:
118
  assert is_valid, f"Duration {val} should be valid but HTML5 validation rejected it"
119
 
120
  def test_audio_source_radios_exist(self, page: Page, base_url: str):
 
121
  page.goto(base_url)
122
- # Open settings drawer to access audio source radios
123
- page.locator("#settings-btn").click()
124
- page.wait_for_timeout(300)
125
- expect(page.locator('input[name="audio-src"][value="mic"]')).to_be_visible()
126
- expect(page.locator('input[name="audio-src"][value="none"]')).to_be_visible()
127
- # Close drawer
128
- page.locator("#settings-close").click()
129
 
130
  def test_label_field_exists(self, page: Page, base_url: str):
131
  page.goto(base_url)
@@ -140,13 +137,7 @@ class TestRecordingSubmission:
140
  page.goto(base_url)
141
  page.wait_for_timeout(2000)
142
 
143
- # Open settings drawer and set audio source to "none"
144
- page.locator("#settings-btn").click()
145
- page.wait_for_timeout(300)
146
- page.locator('input[name="audio-src"][value="none"]').check()
147
- page.locator("#settings-close").click()
148
- page.wait_for_timeout(300)
149
-
150
  page.locator("#rec-duration").fill("2")
151
  page.locator("#rec-name").fill("e2e-test")
152
 
@@ -189,12 +180,6 @@ class TestRecordingLifecycle:
189
  page.goto(base_url)
190
  page.wait_for_timeout(2000)
191
 
192
- page.locator("#settings-btn").click()
193
- page.wait_for_timeout(300)
194
- page.locator('input[name="audio-src"][value="none"]').check()
195
- page.locator("#settings-close").click()
196
- page.wait_for_timeout(300)
197
-
198
  page.locator("#rec-duration").fill("2")
199
  page.locator("#rec-name").fill("lifecycle-test")
200
  page.locator("#record-btn").click()
@@ -211,12 +196,6 @@ class TestRecordingLifecycle:
211
  page.goto(base_url)
212
  page.wait_for_timeout(2000)
213
 
214
- page.locator("#settings-btn").click()
215
- page.wait_for_timeout(300)
216
- page.locator('input[name="audio-src"][value="none"]').check()
217
- page.locator("#settings-close").click()
218
- page.wait_for_timeout(300)
219
-
220
  page.locator("#rec-duration").fill("2")
221
  page.locator("#record-btn").click()
222
  page.wait_for_timeout(2000)
@@ -232,12 +211,6 @@ class TestRecordingLifecycle:
232
  page.goto(base_url)
233
  page.wait_for_timeout(2000)
234
 
235
- page.locator("#settings-btn").click()
236
- page.wait_for_timeout(300)
237
- page.locator('input[name="audio-src"][value="none"]').check()
238
- page.locator("#settings-close").click()
239
- page.wait_for_timeout(300)
240
-
241
  page.locator("#rec-duration").fill("2")
242
  page.locator("#record-btn").click()
243
  page.wait_for_timeout(2000)
@@ -514,41 +487,52 @@ class TestSwitchDataset:
514
 
515
 
516
  class TestAudioSourceRadios:
517
- def test_default_audio_source(self, page: Page, base_url: str):
 
 
518
  page.goto(base_url)
519
- # Open settings drawer
520
- page.locator("#settings-btn").click()
521
- page.wait_for_timeout(300)
522
- # Default may be mic (if audio available) or none
523
- mic = page.locator('input[name="audio-src"][value="mic"]')
524
- none = page.locator('input[name="audio-src"][value="none"]')
525
- # At least one should be checked
526
- mic_checked = mic.is_checked()
527
- none_checked = none.is_checked()
528
- assert mic_checked or none_checked
529
- page.locator("#settings-close").click()
530
 
531
  def test_upload_radio_shows_upload_area(self, page: Page, base_url: str):
532
  page.goto(base_url)
533
- page.locator("#settings-btn").click()
534
- page.wait_for_timeout(300)
535
  page.locator('input[name="audio-src"][value="upload"]').check()
536
  page.wait_for_timeout(500)
537
  expect(page.locator("#audio-upload-section")).to_be_visible()
538
- page.locator("#settings-close").click()
539
 
540
- def test_none_radio_hides_upload_area(self, page: Page, base_url: str):
 
 
 
 
 
 
541
  page.goto(base_url)
542
- page.locator("#settings-btn").click()
543
- page.wait_for_timeout(300)
544
  page.locator('input[name="audio-src"][value="upload"]').check()
545
  page.wait_for_timeout(500)
546
  expect(page.locator("#audio-upload-section")).to_be_visible()
547
 
548
- page.locator('input[name="audio-src"][value="none"]').check()
549
  page.wait_for_timeout(500)
550
  expect(page.locator("#audio-upload-section")).to_be_hidden()
551
- page.locator("#settings-close").click()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
552
 
553
 
554
  # ──────── Settings panel tests ───────────────────────────────────────
@@ -564,18 +548,16 @@ class TestSettingsPanel:
564
  expect(page.locator("#dataset-root-input")).to_be_visible()
565
  page.locator("#settings-close").click()
566
 
567
- def test_experimental_toggle(self, page: Page, base_url: str):
568
- import httpx
569
- # Enable motion_models via API first
570
- httpx.post(f"{base_url}/api/experiments", json={"motion_models": True}, timeout=5)
571
-
572
  page.goto(base_url)
573
- # Wait for poll to update the UI
574
- page.wait_for_timeout(3000)
575
-
576
- # After poll, the experimental checkbox should be attached
577
- checkbox = page.locator("#feature-motion-models")
578
- expect(checkbox).to_be_attached()
 
579
 
580
  def test_dataset_root_displayed(self, page: Page, base_url: str):
581
  page.goto(base_url)
@@ -597,12 +579,6 @@ class TestFormEdgeCases:
597
  page.goto(base_url)
598
  page.wait_for_timeout(2000)
599
 
600
- page.locator("#settings-btn").click()
601
- page.wait_for_timeout(300)
602
- page.locator('input[name="audio-src"][value="none"]').check()
603
- page.locator("#settings-close").click()
604
- page.wait_for_timeout(300)
605
-
606
  page.locator("#rec-name").fill("")
607
  page.locator("#rec-duration").fill("2")
608
  page.locator("#record-btn").click()
@@ -618,12 +594,6 @@ class TestFormEdgeCases:
618
  page.goto(base_url)
619
  page.wait_for_timeout(2000)
620
 
621
- page.locator("#settings-btn").click()
622
- page.wait_for_timeout(300)
623
- page.locator('input[name="audio-src"][value="none"]').check()
624
- page.locator("#settings-close").click()
625
- page.wait_for_timeout(300)
626
-
627
  long_label = "a" * 80
628
  page.locator("#rec-name").fill(long_label)
629
  page.locator("#rec-duration").fill("2")
@@ -640,12 +610,6 @@ class TestFormEdgeCases:
640
  page.goto(base_url)
641
  page.wait_for_timeout(2000)
642
 
643
- page.locator("#settings-btn").click()
644
- page.wait_for_timeout(300)
645
- page.locator('input[name="audio-src"][value="none"]').check()
646
- page.locator("#settings-close").click()
647
- page.wait_for_timeout(300)
648
-
649
  page.locator("#rec-name").fill("my move @#$!")
650
  page.locator("#rec-duration").fill("2")
651
  page.locator("#record-btn").click()
@@ -709,17 +673,18 @@ class TestCommunitySection:
709
  tab = page.locator('.section-tab[data-tab="community"]')
710
  tab.click()
711
  page.wait_for_timeout(500)
712
- fetch_btn = page.locator("#fetch-community-btn")
713
  expect(fetch_btn).to_be_visible()
714
 
715
- def test_fetch_community_button_clickable(self, page: Page, base_url: str):
 
716
  page.goto(base_url)
717
  page.wait_for_timeout(2000)
718
  tab = page.locator('.section-tab[data-tab="community"]')
719
  tab.click()
720
  page.wait_for_timeout(500)
721
- fetch_btn = page.locator("#fetch-community-btn")
722
- expect(fetch_btn).to_be_enabled()
723
 
724
 
725
  # ──────── HF upload section tests ─────────────────────────────────
@@ -760,13 +725,7 @@ class TestResponsiveness:
760
  page.goto(base_url)
761
  page.wait_for_timeout(2000)
762
 
763
- # Set audio to silent to avoid audio dependency
764
- page.locator("#settings-btn").click()
765
- page.wait_for_timeout(300)
766
- page.locator('input[name="audio-src"][value="none"]').check()
767
- page.locator("#settings-close").click()
768
- page.wait_for_timeout(300)
769
-
770
  page.locator("#rec-duration").fill("10")
771
  page.locator("#record-btn").click()
772
  page.wait_for_timeout(2000)
@@ -826,13 +785,7 @@ class TestResponsiveness:
826
  page.goto(base_url)
827
  page.wait_for_timeout(2000)
828
 
829
- # Set audio to silent
830
- page.locator("#settings-btn").click()
831
- page.wait_for_timeout(300)
832
- page.locator('input[name="audio-src"][value="none"]').check()
833
- page.locator("#settings-close").click()
834
- page.wait_for_timeout(300)
835
-
836
  page.locator("#rec-duration").fill("10")
837
  page.locator("#record-btn").click()
838
  page.wait_for_timeout(2000)
@@ -854,27 +807,44 @@ class TestResponsiveness:
854
  # ──────── Mic-only radio tests ───────────────────────────────────────
855
 
856
 
857
- class TestMicOnlyOption:
858
- """Tests for the mic-only (no motion) recording option."""
859
 
860
- def test_mic_only_radio_visible(self, page: Page, base_url: str):
861
- """The mic-only radio option should be visible in settings."""
862
  page.goto(base_url)
863
- page.locator("#settings-btn").click()
864
- page.wait_for_timeout(300)
865
- mic_only = page.locator('input[name="audio-src"][value="mic-only"]')
866
- expect(mic_only).to_be_attached()
867
- page.locator("#settings-close").click()
868
 
869
- def test_mic_only_radio_selectable(self, page: Page, base_url: str):
870
- """The mic-only radio can be selected."""
871
  page.goto(base_url)
872
- page.locator("#settings-btn").click()
873
- page.wait_for_timeout(300)
874
- mic_only = page.locator('input[name="audio-src"][value="mic-only"]')
875
- mic_only.check()
876
- assert mic_only.is_checked()
877
- page.locator("#settings-close").click()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
878
 
879
  def test_audio_only_badge_on_move(self, page: Page, base_url: str, test_marionette):
880
  """Audio-only moves should show an 'audio only' badge."""
@@ -952,3 +922,208 @@ class TestWelcomeMessagesUI:
952
  checked = page.locator('input[name="welcome-msgs"]:checked')
953
  assert checked.get_attribute("value") == "0"
954
  page.locator("#settings-close").click()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
118
  assert is_valid, f"Duration {val} should be valid but HTML5 validation rejected it"
119
 
120
  def test_audio_source_radios_exist(self, page: Page, base_url: str):
121
+ """Audio source radios are inline in the record hero section."""
122
  page.goto(base_url)
123
+ expect(page.locator('input[name="audio-src"][value="silent"]')).to_be_visible()
124
+ expect(page.locator('input[name="audio-src"][value="upload"]')).to_be_visible()
125
+ expect(page.locator('input[name="audio-src"][value="robot"]')).to_be_visible()
 
 
 
 
126
 
127
  def test_label_field_exists(self, page: Page, base_url: str):
128
  page.goto(base_url)
 
137
  page.goto(base_url)
138
  page.wait_for_timeout(2000)
139
 
140
+ # Silent audio is the default no need to change source
 
 
 
 
 
 
141
  page.locator("#rec-duration").fill("2")
142
  page.locator("#rec-name").fill("e2e-test")
143
 
 
180
  page.goto(base_url)
181
  page.wait_for_timeout(2000)
182
 
 
 
 
 
 
 
183
  page.locator("#rec-duration").fill("2")
184
  page.locator("#rec-name").fill("lifecycle-test")
185
  page.locator("#record-btn").click()
 
196
  page.goto(base_url)
197
  page.wait_for_timeout(2000)
198
 
 
 
 
 
 
 
199
  page.locator("#rec-duration").fill("2")
200
  page.locator("#record-btn").click()
201
  page.wait_for_timeout(2000)
 
211
  page.goto(base_url)
212
  page.wait_for_timeout(2000)
213
 
 
 
 
 
 
 
214
  page.locator("#rec-duration").fill("2")
215
  page.locator("#record-btn").click()
216
  page.wait_for_timeout(2000)
 
487
 
488
 
489
  class TestAudioSourceRadios:
490
+ """Audio source radios are now inline in the record hero section."""
491
+
492
+ def test_default_audio_source_is_silent(self, page: Page, base_url: str):
493
  page.goto(base_url)
494
+ silent = page.locator('input[name="audio-src"][value="silent"]')
495
+ expect(silent).to_be_checked()
 
 
 
 
 
 
 
 
 
496
 
497
  def test_upload_radio_shows_upload_area(self, page: Page, base_url: str):
498
  page.goto(base_url)
 
 
499
  page.locator('input[name="audio-src"][value="upload"]').check()
500
  page.wait_for_timeout(500)
501
  expect(page.locator("#audio-upload-section")).to_be_visible()
 
502
 
503
+ def test_robot_radio_shows_robot_area(self, page: Page, base_url: str):
504
+ page.goto(base_url)
505
+ page.locator('input[name="audio-src"][value="robot"]').check()
506
+ page.wait_for_timeout(500)
507
+ expect(page.locator("#robot-audio-section")).to_be_visible()
508
+
509
+ def test_silent_radio_hides_upload_area(self, page: Page, base_url: str):
510
  page.goto(base_url)
 
 
511
  page.locator('input[name="audio-src"][value="upload"]').check()
512
  page.wait_for_timeout(500)
513
  expect(page.locator("#audio-upload-section")).to_be_visible()
514
 
515
+ page.locator('input[name="audio-src"][value="silent"]').check()
516
  page.wait_for_timeout(500)
517
  expect(page.locator("#audio-upload-section")).to_be_hidden()
518
+
519
+ def test_switching_source_hides_previous_panel(self, page: Page, base_url: str):
520
+ page.goto(base_url)
521
+ # Show upload panel
522
+ page.locator('input[name="audio-src"][value="upload"]').check()
523
+ page.wait_for_timeout(500)
524
+ expect(page.locator("#audio-upload-section")).to_be_visible()
525
+
526
+ # Switch to robot — upload should hide, robot should show
527
+ page.locator('input[name="audio-src"][value="robot"]').check()
528
+ page.wait_for_timeout(500)
529
+ expect(page.locator("#audio-upload-section")).to_be_hidden()
530
+ expect(page.locator("#robot-audio-section")).to_be_visible()
531
+
532
+ # Switch to silent — robot should hide
533
+ page.locator('input[name="audio-src"][value="silent"]').check()
534
+ page.wait_for_timeout(500)
535
+ expect(page.locator("#robot-audio-section")).to_be_hidden()
536
 
537
 
538
  # ──────── Settings panel tests ───────────────────────────────────────
 
548
  expect(page.locator("#dataset-root-input")).to_be_visible()
549
  page.locator("#settings-close").click()
550
 
551
+ def test_lead_compensation_section_visible(self, page: Page, base_url: str):
552
+ """Lead compensation settings should be visible in the settings drawer."""
 
 
 
553
  page.goto(base_url)
554
+ page.wait_for_timeout(2000)
555
+ page.locator("#settings-btn").click()
556
+ page.wait_for_timeout(300)
557
+ expect(page.locator("#lead-comp-section")).to_be_visible()
558
+ expect(page.locator("#lead-frames-head")).to_be_visible()
559
+ expect(page.locator("#lead-frames-antennas")).to_be_visible()
560
+ page.locator("#settings-close").click()
561
 
562
  def test_dataset_root_displayed(self, page: Page, base_url: str):
563
  page.goto(base_url)
 
579
  page.goto(base_url)
580
  page.wait_for_timeout(2000)
581
 
 
 
 
 
 
 
582
  page.locator("#rec-name").fill("")
583
  page.locator("#rec-duration").fill("2")
584
  page.locator("#record-btn").click()
 
594
  page.goto(base_url)
595
  page.wait_for_timeout(2000)
596
 
 
 
 
 
 
 
597
  long_label = "a" * 80
598
  page.locator("#rec-name").fill(long_label)
599
  page.locator("#rec-duration").fill("2")
 
610
  page.goto(base_url)
611
  page.wait_for_timeout(2000)
612
 
 
 
 
 
 
 
613
  page.locator("#rec-name").fill("my move @#$!")
614
  page.locator("#rec-duration").fill("2")
615
  page.locator("#record-btn").click()
 
673
  tab = page.locator('.section-tab[data-tab="community"]')
674
  tab.click()
675
  page.wait_for_timeout(500)
676
+ fetch_btn = page.locator("#download-community-btn")
677
  expect(fetch_btn).to_be_visible()
678
 
679
+ def test_download_community_button_exists(self, page: Page, base_url: str):
680
+ """Download button should exist (disabled until items selected)."""
681
  page.goto(base_url)
682
  page.wait_for_timeout(2000)
683
  tab = page.locator('.section-tab[data-tab="community"]')
684
  tab.click()
685
  page.wait_for_timeout(500)
686
+ btn = page.locator("#download-community-btn")
687
+ expect(btn).to_be_attached()
688
 
689
 
690
  # ──────── HF upload section tests ─────────────────────────────────
 
725
  page.goto(base_url)
726
  page.wait_for_timeout(2000)
727
 
728
+ # Silent audio is the default no settings drawer needed
 
 
 
 
 
 
729
  page.locator("#rec-duration").fill("10")
730
  page.locator("#record-btn").click()
731
  page.wait_for_timeout(2000)
 
785
  page.goto(base_url)
786
  page.wait_for_timeout(2000)
787
 
788
+ # Silent audio is the default
 
 
 
 
 
 
789
  page.locator("#rec-duration").fill("10")
790
  page.locator("#record-btn").click()
791
  page.wait_for_timeout(2000)
 
807
  # ──────── Mic-only radio tests ───────────────────────────────────────
808
 
809
 
810
+ class TestRecordAudioButton:
811
+ """Tests for the standalone Record Audio button (replaces mic-only radio)."""
812
 
813
+ def test_record_audio_button_visible(self, page: Page, base_url: str):
814
+ """The Record Audio button should be visible in the record hero."""
815
  page.goto(base_url)
816
+ btn = page.locator("#record-audio-btn")
817
+ expect(btn).to_be_visible()
 
 
 
818
 
819
+ def test_record_audio_button_has_label(self, page: Page, base_url: str):
820
+ """The Record Audio button should display its label."""
821
  page.goto(base_url)
822
+ btn = page.locator("#record-audio-btn")
823
+ expect(btn).to_contain_text("Record Audio")
824
+
825
+ def test_record_audio_hint_visible(self, page: Page, base_url: str):
826
+ """A hint below Record Audio explains where recordings go."""
827
+ page.goto(base_url)
828
+ hint = page.locator(".record-audio-hint")
829
+ expect(hint).to_be_visible()
830
+ hint_text = hint.text_content() or ""
831
+ assert "robot file" in hint_text.lower() or "moves list" in hint_text.lower()
832
+
833
+ def test_record_audio_triggers_recording(self, page: Page, base_url: str, test_marionette):
834
+ """Clicking Record Audio should start an audio-only recording."""
835
+ page.goto(base_url)
836
+ page.wait_for_timeout(2000)
837
+
838
+ page.locator("#rec-duration").fill("2")
839
+ page.locator("#record-audio-btn").click()
840
+ page.wait_for_timeout(2000)
841
+
842
+ badge_text = page.locator("#mode-badge").text_content() or ""
843
+ assert "idle" not in badge_text.lower()
844
+
845
+ # Cleanup
846
+ test_marionette._set_idle_state()
847
+ test_marionette._pending_recording = None
848
 
849
  def test_audio_only_badge_on_move(self, page: Page, base_url: str, test_marionette):
850
  """Audio-only moves should show an 'audio only' badge."""
 
922
  checked = page.locator('input[name="welcome-msgs"]:checked')
923
  assert checked.get_attribute("value") == "0"
924
  page.locator("#settings-close").click()
925
+
926
+
927
+ # ──────── Move card display tests ──────────────────────────────────────
928
+
929
+
930
+ class TestMoveCardDisplay:
931
+ """Tests for move card content — timestamps, badges, metadata."""
932
+
933
+ def _inject_move(self, test_marionette, move_id="e2e-card-test", audio_only=False):
934
+ import json, time
935
+ data_dir = test_marionette._dataset_dir
936
+ move_data = {
937
+ "description": "card display test",
938
+ "created_at": time.time(),
939
+ "time": [0.0, 0.01, 0.02],
940
+ "set_target_data": [
941
+ {"head": [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]], "antennas": [0,0], "body_yaw": 0.0}
942
+ for _ in range(3)
943
+ ],
944
+ }
945
+ if audio_only:
946
+ move_data["audio_only"] = True
947
+ move_data["set_target_data"] = []
948
+ (data_dir / f"{move_id}.json").write_text(json.dumps(move_data))
949
+ test_marionette._refresh_recordings()
950
+ return data_dir / f"{move_id}.json"
951
+
952
+ def test_move_card_shows_time(self, page: Page, base_url: str, test_marionette):
953
+ """Move cards should show time (hour:minute), not just the date."""
954
+ path = self._inject_move(test_marionette)
955
+ page.goto(base_url)
956
+ page.wait_for_timeout(2000)
957
+
958
+ meta = page.locator(".move-meta-row").first
959
+ meta_text = meta.text_content() or ""
960
+ # Should contain a colon-separated time (e.g. "14:32" or "2:32 PM")
961
+ assert ":" in meta_text
962
+
963
+ path.unlink(missing_ok=True)
964
+ test_marionette._refresh_recordings()
965
+
966
+ def test_audio_only_move_shows_badge(self, page: Page, base_url: str, test_marionette):
967
+ """Audio-only moves should display an 'audio only' badge."""
968
+ path = self._inject_move(test_marionette, "e2e-ao-card", audio_only=True)
969
+ page.goto(base_url)
970
+ page.wait_for_timeout(2000)
971
+
972
+ badge = page.locator(".move-badge.audio-only")
973
+ assert badge.count() > 0
974
+
975
+ path.unlink(missing_ok=True)
976
+ test_marionette._refresh_recordings()
977
+
978
+ def test_moves_list_stable_during_polling(self, page: Page, base_url: str, test_marionette):
979
+ """Move cards should not flicker during poll cycles (dirty-flag pattern)."""
980
+ path = self._inject_move(test_marionette, "e2e-stable-test")
981
+ page.goto(base_url)
982
+ page.wait_for_timeout(2000)
983
+
984
+ # Grab the first move card element reference
985
+ card = page.locator(".move-card").first
986
+ card_id = card.evaluate("el => el.dataset.moveId || el.textContent.slice(0,20)")
987
+
988
+ # Wait through two poll cycles (~3s) and verify the same element persists
989
+ page.wait_for_timeout(3500)
990
+ card_after = page.locator(".move-card").first
991
+ card_id_after = card_after.evaluate("el => el.dataset.moveId || el.textContent.slice(0,20)")
992
+ assert card_id == card_id_after
993
+
994
+ path.unlink(missing_ok=True)
995
+ test_marionette._refresh_recordings()
996
+
997
+
998
+ # ──────── Robot file selection tests ────────────────────────────────────
999
+
1000
+
1001
+ class TestRobotFileSelection:
1002
+ """Tests for the 'Robot file' audio source tab and dropdown."""
1003
+
1004
+ def test_robot_tab_shows_dropdown(self, page: Page, base_url: str):
1005
+ """Clicking the 'Robot file' radio should show the robot audio dropdown."""
1006
+ page.goto(base_url)
1007
+ page.locator('input[name="audio-src"][value="robot"]').check()
1008
+ page.wait_for_timeout(500)
1009
+ expect(page.locator("#robot-audio-section")).to_be_visible()
1010
+ expect(page.locator("#robot-audio-select")).to_be_visible()
1011
+
1012
+ def test_robot_dropdown_has_options(self, page: Page, base_url: str, test_marionette):
1013
+ """After injecting an audio-only move, the robot dropdown should list it."""
1014
+ import json, time
1015
+ data_dir = test_marionette._dataset_dir
1016
+ # Create an audio-only move with a WAV file
1017
+ move_data = {
1018
+ "description": "robot file source",
1019
+ "audio_only": True,
1020
+ "created_at": time.time(),
1021
+ "time": [0.0, 0.01],
1022
+ "set_target_data": [],
1023
+ }
1024
+ (data_dir / "e2e-robot-src.json").write_text(json.dumps(move_data))
1025
+ # Create a matching WAV file (robot audio list scans for WAV files)
1026
+ import struct, wave
1027
+ wav_path = data_dir / "e2e-robot-src.wav"
1028
+ with wave.open(str(wav_path), "w") as wf:
1029
+ wf.setnchannels(1)
1030
+ wf.setsampwidth(2)
1031
+ wf.setframerate(16000)
1032
+ wf.writeframes(struct.pack("<" + "h" * 160, *([0] * 160)))
1033
+ test_marionette._refresh_recordings()
1034
+
1035
+ page.goto(base_url)
1036
+ page.locator('input[name="audio-src"][value="robot"]').check()
1037
+ page.wait_for_timeout(1500) # Allow time for file list to load
1038
+
1039
+ options = page.locator("#robot-audio-select option")
1040
+ # Should have at least one file entry
1041
+ assert options.count() >= 1
1042
+
1043
+ # Cleanup
1044
+ (data_dir / "e2e-robot-src.json").unlink(missing_ok=True)
1045
+ wav_path.unlink(missing_ok=True)
1046
+ test_marionette._refresh_recordings()
1047
+
1048
+
1049
+ # ──────── Playback UI tests ────────────────────────────────────────────
1050
+
1051
+
1052
+ class TestPlaybackUI:
1053
+ """Tests for playback UI controls — play button, phase overlay."""
1054
+
1055
+ def _inject_move(self, test_marionette, move_id="e2e-playui"):
1056
+ import json, time
1057
+ data_dir = test_marionette._dataset_dir
1058
+ move_data = {
1059
+ "description": "playback ui test",
1060
+ "created_at": time.time(),
1061
+ "time": [i * 0.01 for i in range(200)],
1062
+ "set_target_data": [
1063
+ {"head": [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]], "antennas": [0,0], "body_yaw": 0.0}
1064
+ for _ in range(200)
1065
+ ],
1066
+ }
1067
+ (data_dir / f"{move_id}.json").write_text(json.dumps(move_data))
1068
+ test_marionette._refresh_recordings()
1069
+ return data_dir / f"{move_id}.json"
1070
+
1071
+ def test_play_button_queues_playback(self, page: Page, base_url: str, test_marionette):
1072
+ """Clicking play on a move should change mode from idle."""
1073
+ path = self._inject_move(test_marionette)
1074
+ page.goto(base_url)
1075
+ page.wait_for_timeout(2000)
1076
+
1077
+ page.locator(".play-action").first.click()
1078
+ page.wait_for_timeout(2000)
1079
+
1080
+ badge_text = page.locator("#mode-badge").text_content() or ""
1081
+ assert "queued" in badge_text.lower() or "playing" in badge_text.lower()
1082
+
1083
+ test_marionette._set_idle_state()
1084
+ test_marionette._pending_playback = None
1085
+ path.unlink(missing_ok=True)
1086
+ test_marionette._refresh_recordings()
1087
+
1088
+ def test_stop_button_exists_in_phase_overlay(self, page: Page, base_url: str):
1089
+ """The phase overlay should contain a stop button."""
1090
+ page.goto(base_url)
1091
+ stop_btn = page.locator("#phase-stop-btn")
1092
+ expect(stop_btn).to_be_attached()
1093
+
1094
+ def test_phase_overlay_hidden_when_idle(self, page: Page, base_url: str):
1095
+ """Phase overlay should not have 'active' class in idle state."""
1096
+ page.goto(base_url)
1097
+ page.wait_for_timeout(2000)
1098
+ overlay = page.locator("#phase-overlay")
1099
+ expect(overlay).not_to_have_class(re.compile("active"))
1100
+
1101
+
1102
+ # ──────── Tab structure tests ──────────────────────────────────────────
1103
+
1104
+
1105
+ class TestTabStructure:
1106
+ """Tests for the 3-tab layout: My Moves, Library, Community."""
1107
+
1108
+ def test_three_tabs_exist(self, page: Page, base_url: str):
1109
+ page.goto(base_url)
1110
+ expect(page.locator('.section-tab[data-tab="my-moves"]')).to_be_visible()
1111
+ expect(page.locator('.section-tab[data-tab="library"]')).to_be_visible()
1112
+ expect(page.locator('.section-tab[data-tab="community"]')).to_be_visible()
1113
+
1114
+ def test_my_moves_tab_active_by_default(self, page: Page, base_url: str):
1115
+ page.goto(base_url)
1116
+ expect(page.locator('.section-tab[data-tab="my-moves"]')).to_have_class(re.compile("active"))
1117
+ expect(page.locator('#tab-my-moves')).to_have_class(re.compile("active"))
1118
+
1119
+ def test_library_tab_shows_library_panel(self, page: Page, base_url: str):
1120
+ page.goto(base_url)
1121
+ page.locator('.section-tab[data-tab="library"]').click()
1122
+ page.wait_for_timeout(500)
1123
+ expect(page.locator('#tab-library')).to_have_class(re.compile("active"))
1124
+ expect(page.locator('#library-dataset-select')).to_be_visible()
1125
+
1126
+ def test_audio_source_label_visible(self, page: Page, base_url: str):
1127
+ page.goto(base_url)
1128
+ expect(page.locator(".audio-source-label")).to_be_visible()
1129
+ expect(page.locator(".audio-source-label")).to_contain_text("Audio source")
tests/test_antenna_collision.py ADDED
@@ -0,0 +1,425 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Test antenna collisions on Reachy Mini.
3
+
4
+ Standalone script (no Marionette). Commands the left antenna to collide
5
+ with the right antenna at known timestamps, using non-periodic intervals
6
+ so detection alignment is unambiguous.
7
+
8
+ Collision setup:
9
+ - Right antenna fixed at -0.68 rad
10
+ - Left antenna moves from 0.0 to 0.70 rad (collision at ~0.60 rad)
11
+ - Hold 100ms, then return to rest
12
+ - Low PID + flexible antennas = safe, produces audible click
13
+
14
+ Records present antenna positions at 50Hz on the robot side (no network
15
+ latency), saves to JSON, SCPs back, and analyzes to detect collision times
16
+ from the trajectory.
17
+
18
+ Usage:
19
+ python tests/test_antenna_collision.py [--host reachy-mini.local]
20
+ """
21
+ from __future__ import annotations
22
+
23
+ import argparse
24
+ import json
25
+ import subprocess
26
+ import sys
27
+ import tempfile
28
+ import time
29
+ from pathlib import Path
30
+
31
+ import numpy as np
32
+
33
+ # Collision parameters
34
+ RIGHT_REST = -0.68 # right antenna fixed position (rad)
35
+ LEFT_REST = 0.0 # left antenna rest position (rad)
36
+ LEFT_COLLISION = 0.70 # left antenna collision target (past contact at ~0.60)
37
+ HOLD_DURATION = 0.2 # seconds to hold at collision position
38
+
39
+ # Non-periodic collision times — gaps are all different (1.3, 1.7, 2.3, 3.1s)
40
+ # so there's no ambiguity when matching detected events to expected events.
41
+ COLLISION_TIMES = [1.0, 2.3, 4.0, 6.3, 9.4]
42
+
43
+ ROBOT_USER = "pollen"
44
+ ROBOT_PYTHON = "/venvs/apps_venv/bin/python"
45
+ REMOTE_RESULTS = "/tmp/collision_positions.json"
46
+
47
+ ROBOT_COLLISION_SCRIPT = """\
48
+ import numpy as np
49
+ import os, sys, time, json
50
+
51
+ collision_times = json.loads(sys.argv[1])
52
+ right_rest = float(sys.argv[2])
53
+ left_rest = float(sys.argv[3])
54
+ left_collision = float(sys.argv[4])
55
+ hold_duration = float(sys.argv[5])
56
+ results_path = sys.argv[6]
57
+
58
+ print(f"robot: connecting to ReachyMini", flush=True)
59
+ from reachy_mini import ReachyMini
60
+ from reachy_mini.utils import create_head_pose
61
+ r = ReachyMini(media_backend="no_media")
62
+
63
+ # Go to rest position
64
+ r.goto_target(create_head_pose(), antennas=[left_rest, right_rest], duration=1.0)
65
+ time.sleep(1.5)
66
+
67
+ print(f"robot: rest position — left={left_rest}, right={right_rest}", flush=True)
68
+ print(f"robot: will collide at times: {collision_times}", flush=True)
69
+ print(f"robot: collision target: left={left_collision}, hold={hold_duration}s", flush=True)
70
+
71
+ # State machine
72
+ DT = 0.02 # 50Hz update rate
73
+ total_duration = max(collision_times) + hold_duration + 1.0
74
+ n_steps = int(total_duration / DT)
75
+
76
+ # Build timeline: for each step, determine left antenna target
77
+ left_targets = np.full(n_steps, left_rest, dtype=np.float64)
78
+ for ct in collision_times:
79
+ start_step = int(ct / DT)
80
+ end_step = int((ct + hold_duration) / DT)
81
+ end_step = min(end_step, n_steps)
82
+ left_targets[start_step:end_step] = left_collision
83
+
84
+ # Recording arrays
85
+ timestamps = []
86
+ left_present = []
87
+ right_present = []
88
+ left_target_log = []
89
+
90
+ print("robot: MARK_START", flush=True)
91
+ t0 = time.monotonic()
92
+
93
+ for i in range(n_steps):
94
+ left = float(left_targets[i])
95
+ r.set_target(
96
+ head=np.eye(4),
97
+ body_yaw=0.0,
98
+ antennas=np.array([left, right_rest]),
99
+ )
100
+
101
+ # Read present position
102
+ pos = r.get_present_antenna_joint_positions()
103
+ elapsed = time.monotonic() - t0
104
+ timestamps.append(elapsed)
105
+ left_present.append(pos[0])
106
+ right_present.append(pos[1])
107
+ left_target_log.append(left)
108
+
109
+ # Real-time pacing
110
+ target_time = (i + 1) * DT
111
+ now = time.monotonic() - t0
112
+ if target_time > now:
113
+ time.sleep(target_time - now)
114
+
115
+ elapsed = time.monotonic() - t0
116
+ print(f"robot: finished {n_steps} steps in {elapsed:.3f}s (expected {total_duration:.3f}s)", flush=True)
117
+
118
+ # Save results
119
+ results = {
120
+ "collision_times": collision_times,
121
+ "left_collision_target": left_collision,
122
+ "right_rest": right_rest,
123
+ "hold_duration": hold_duration,
124
+ "timestamps": timestamps,
125
+ "left_present": left_present,
126
+ "right_present": right_present,
127
+ "left_target": left_target_log,
128
+ }
129
+ with open(results_path, "w") as f:
130
+ json.dump(results, f)
131
+ print(f"robot: saved {len(timestamps)} samples to {results_path}", flush=True)
132
+
133
+ # Return to rest
134
+ r.goto_target(create_head_pose(), antennas=[left_rest, right_rest], duration=1.0)
135
+ time.sleep(1.5)
136
+ print("robot: done", flush=True)
137
+ os._exit(0)
138
+ """
139
+
140
+
141
+ def scp_to_robot(local_path: Path, remote_path: str, host: str) -> None:
142
+ """Copy a file to the robot via SCP."""
143
+ target = f"{ROBOT_USER}@{host}:{remote_path}"
144
+ result = subprocess.run(
145
+ ["scp", "-o", "ConnectTimeout=5", str(local_path), target],
146
+ capture_output=True, text=True, timeout=15,
147
+ )
148
+ if result.returncode != 0:
149
+ raise RuntimeError(f"SCP failed: {result.stderr}")
150
+ print(f" Copied to {target}")
151
+
152
+
153
+ def scp_from_robot(remote_path: str, local_path: Path, host: str) -> None:
154
+ """Copy a file from the robot via SCP."""
155
+ source = f"{ROBOT_USER}@{host}:{remote_path}"
156
+ result = subprocess.run(
157
+ ["scp", "-o", "ConnectTimeout=5", source, str(local_path)],
158
+ capture_output=True, text=True, timeout=15,
159
+ )
160
+ if result.returncode != 0:
161
+ raise RuntimeError(f"SCP failed: {result.stderr}")
162
+ print(f" Copied from {source}")
163
+
164
+
165
+ def run_on_robot(host: str) -> subprocess.Popen:
166
+ """SCP script to robot and start collision sequence (non-blocking)."""
167
+ with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
168
+ f.write(ROBOT_COLLISION_SCRIPT)
169
+ local_script = Path(f.name)
170
+
171
+ remote_script = "/tmp/collision_test.py"
172
+ try:
173
+ scp_to_robot(local_script, remote_script, host)
174
+ finally:
175
+ local_script.unlink()
176
+
177
+ args_str = (
178
+ f"{ROBOT_PYTHON} {remote_script} "
179
+ f"'{json.dumps(COLLISION_TIMES)}' "
180
+ f"{RIGHT_REST} {LEFT_REST} {LEFT_COLLISION} {HOLD_DURATION} "
181
+ f"{REMOTE_RESULTS}"
182
+ )
183
+ proc = subprocess.Popen(
184
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{host}", args_str],
185
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
186
+ )
187
+ return proc
188
+
189
+
190
+ def detect_collisions(data: dict) -> list[dict]:
191
+ """Detect collision events from present position trajectory.
192
+
193
+ For each commanded collision, finds the position peak time (when the
194
+ antenna reached its maximum excursion and reversed). This is the moment
195
+ of collision impact or max overshoot.
196
+
197
+ Returns list of dicts with detection details per event.
198
+ """
199
+ ts = np.array(data["timestamps"])
200
+ left_pos = np.array(data["left_present"])
201
+ left_tgt = np.array(data["left_target"])
202
+
203
+ # Find rising edges in target (rest → collision command)
204
+ tgt_diff = np.diff(left_tgt)
205
+ command_indices = np.where(tgt_diff > 0.3)[0]
206
+
207
+ events = []
208
+ for cmd_idx in command_indices:
209
+ cmd_time = ts[cmd_idx]
210
+
211
+ # Look 0-500ms after command for position peak
212
+ window_start = cmd_idx
213
+ window_end = min(cmd_idx + 25, len(ts)) # 500ms at 50Hz
214
+ if window_end <= window_start + 2:
215
+ continue
216
+
217
+ window_pos = left_pos[window_start:window_end]
218
+ window_ts = ts[window_start:window_end]
219
+
220
+ # Position peak = moment of max excursion (collision or overshoot)
221
+ peak_idx = np.argmax(window_pos)
222
+ peak_time = window_ts[peak_idx]
223
+ peak_pos = window_pos[peak_idx]
224
+
225
+ # Check for velocity stall (plateau = physical stop at collision)
226
+ dt = np.diff(window_ts)
227
+ dt[dt < 1e-6] = 1e-6
228
+ vel = np.diff(window_pos) / dt
229
+
230
+ # A stall is consecutive near-zero velocity while position > 0.1
231
+ stall_time = None
232
+ for j in range(len(vel) - 1):
233
+ if (abs(vel[j]) < 1.0 and abs(vel[j + 1]) < 1.0
234
+ and window_pos[j + 1] > 0.1):
235
+ stall_time = window_ts[j + 1]
236
+ break
237
+
238
+ events.append({
239
+ "cmd_time": cmd_time,
240
+ "peak_time": peak_time,
241
+ "peak_pos": peak_pos,
242
+ "latency_ms": (peak_time - cmd_time) * 1000,
243
+ "stall_time": stall_time,
244
+ })
245
+
246
+ return events
247
+
248
+
249
+ def plot_results(data: dict, events: list[dict], output_path: Path) -> None:
250
+ """Generate a plot of antenna trajectories with collision markers."""
251
+ import matplotlib
252
+ matplotlib.use("Agg")
253
+ import matplotlib.pyplot as plt
254
+
255
+ ts = np.array(data["timestamps"])
256
+ left_pos = np.array(data["left_present"])
257
+ right_pos = np.array(data["right_present"])
258
+ left_tgt = np.array(data["left_target"])
259
+ expected_times = data["collision_times"]
260
+
261
+ fig, ax = plt.subplots(figsize=(14, 6))
262
+
263
+ # Plot antenna trajectories
264
+ ax.plot(ts, left_pos, "b-", linewidth=1.5, label="Left antenna (present)", zorder=3)
265
+ ax.plot(ts, right_pos, "r-", linewidth=1.5, label="Right antenna (present)", zorder=3)
266
+ ax.plot(ts, left_tgt, "b--", linewidth=0.8, alpha=0.4, label="Left antenna (target)")
267
+
268
+ # Expected collision times (green dashed)
269
+ for i, ct in enumerate(expected_times):
270
+ label = "Expected collision" if i == 0 else None
271
+ ax.axvline(ct, color="green", linestyle="--", linewidth=1.5, alpha=0.7, label=label)
272
+
273
+ # Detected collision times (red solid)
274
+ for i, ev in enumerate(events):
275
+ label = "Detected (peak)" if i == 0 else None
276
+ ax.axvline(ev["peak_time"], color="red", linestyle="-", linewidth=1.5, alpha=0.7, label=label)
277
+ # Annotate with peak position
278
+ ax.annotate(
279
+ f'{ev["peak_pos"]:.2f}r\n{ev["latency_ms"]:.0f}ms',
280
+ xy=(ev["peak_time"], ev["peak_pos"]),
281
+ xytext=(10, 10), textcoords="offset points",
282
+ fontsize=8, color="red",
283
+ arrowprops=dict(arrowstyle="->", color="red", lw=0.8),
284
+ )
285
+
286
+ ax.set_xlabel("Time (s)")
287
+ ax.set_ylabel("Position (rad)")
288
+ ax.set_title("Antenna Collision Test — Present Positions + Detection")
289
+ ax.legend(loc="upper right", fontsize=9)
290
+ ax.grid(True, alpha=0.3)
291
+ ax.set_xlim(ts[0] - 0.2, ts[-1] + 0.2)
292
+
293
+ fig.tight_layout()
294
+ fig.savefig(str(output_path), dpi=150)
295
+ plt.close(fig)
296
+ print(f" Plot saved to {output_path}")
297
+
298
+
299
+ def analyze_results(data: dict) -> dict:
300
+ """Analyze collision position data and print report."""
301
+ ts = np.array(data["timestamps"])
302
+ left_pos = np.array(data["left_present"])
303
+ left_tgt = np.array(data["left_target"])
304
+ expected_times = data["collision_times"]
305
+
306
+ print(f"\n{'='*60}")
307
+ print("Collision Detection Analysis")
308
+ print(f"{'='*60}")
309
+ print(f" Samples: {len(ts)} at ~{1/np.mean(np.diff(ts)):.0f}Hz")
310
+ print(f" Duration: {ts[-1]:.2f}s")
311
+ print(f" Left antenna range: [{left_pos.min():.3f}, {left_pos.max():.3f}] rad")
312
+ print(f" Expected collisions: {len(expected_times)} at {expected_times}")
313
+
314
+ # Detect collisions
315
+ events = detect_collisions(data)
316
+
317
+ # Per-collision detail
318
+ print(f"\n--- Per-collision detail ---")
319
+ for i, ev in enumerate(events):
320
+ stall_info = f", stall at t={ev['stall_time']:.3f}s" if ev["stall_time"] else ""
321
+ print(f" Collision {i+1}: cmd t={ev['cmd_time']:.1f}s → "
322
+ f"peak {ev['peak_pos']:.3f} rad at t={ev['peak_time']:.3f}s "
323
+ f"(latency {ev['latency_ms']:.0f}ms{stall_info})")
324
+
325
+ # Match detected peaks to expected times
326
+ detected_peak_times = [ev["peak_time"] for ev in events]
327
+ print(f"\n--- Matching ---")
328
+ matched = 0
329
+ for i, et in enumerate(expected_times):
330
+ if not detected_peak_times:
331
+ print(f" Expected {et:.1f}s: NO MATCH")
332
+ continue
333
+ nearest = min(detected_peak_times, key=lambda d: abs(d - et))
334
+ offset_ms = (nearest - et) * 1000
335
+ ok = abs(nearest - et) < 0.3
336
+ if ok:
337
+ matched += 1
338
+ status = "OK" if ok else "MISS"
339
+ print(f" Expected {et:.1f}s → peak at {nearest:.3f}s "
340
+ f"(offset {offset_ms:+.0f}ms) [{status}]")
341
+
342
+ success = matched >= len(expected_times) - 1
343
+ print(f"\n {'PASS' if success else 'FAIL'}: "
344
+ f"{matched}/{len(expected_times)} collisions matched")
345
+
346
+ # Generate plot
347
+ plot_path = Path("tests/collision_trajectory.png")
348
+ plot_results(data, events, plot_path)
349
+
350
+ return {
351
+ "success": success,
352
+ "expected": len(expected_times),
353
+ "detected": len(events),
354
+ "matched": matched,
355
+ "events": events,
356
+ }
357
+
358
+
359
+ def main():
360
+ parser = argparse.ArgumentParser(description="Antenna collision test")
361
+ parser.add_argument("--host", default="reachy-mini.local", help="Robot hostname/IP")
362
+ args = parser.parse_args()
363
+
364
+ print(f"\n{'='*60}")
365
+ print("Antenna Collision Test")
366
+ print(f"{'='*60}")
367
+ print(f" Right antenna fixed at {RIGHT_REST} rad")
368
+ print(f" Left antenna: rest={LEFT_REST}, collision={LEFT_COLLISION} rad")
369
+ print(f" Collision times: {COLLISION_TIMES}")
370
+ gaps = [COLLISION_TIMES[i+1] - COLLISION_TIMES[i] for i in range(len(COLLISION_TIMES)-1)]
371
+ print(f" Gaps between collisions: {[f'{g:.1f}s' for g in gaps]}")
372
+ print(f" Hold duration: {HOLD_DURATION}s\n")
373
+
374
+ # Stop any running app
375
+ print("[1/4] Stopping any running app on robot...")
376
+ subprocess.run(
377
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{args.host}",
378
+ "curl -sf -X POST http://127.0.0.1:8000/api/apps/stop-current-app >/dev/null 2>&1 || true"],
379
+ capture_output=True, timeout=10,
380
+ )
381
+ time.sleep(1)
382
+ print(" Done")
383
+
384
+ # Run collisions (with position recording)
385
+ print("\n[2/4] Running collision sequence + recording positions...")
386
+ proc = run_on_robot(args.host)
387
+
388
+ # Stream robot output in real time
389
+ print("\n--- Robot output ---")
390
+ for line in iter(proc.stdout.readline, ""):
391
+ line = line.rstrip()
392
+ if line:
393
+ print(f" {line}")
394
+ proc.wait()
395
+ print("--- End robot output ---")
396
+
397
+ if proc.returncode != 0:
398
+ print(f"\nFAILED — Return code: {proc.returncode}")
399
+ return 1
400
+
401
+ # SCP results back
402
+ print("\n[3/4] Fetching position data from robot...")
403
+ local_results = Path("tests/collision_positions.json")
404
+ scp_from_robot(REMOTE_RESULTS, local_results, args.host)
405
+ print(f" Saved to {local_results}")
406
+
407
+ # Load and analyze
408
+ print("\n[4/4] Analyzing collision trajectory...")
409
+ with open(local_results) as f:
410
+ data = json.load(f)
411
+
412
+ result = analyze_results(data)
413
+
414
+ print(f"\n{'='*60}")
415
+ if result["success"]:
416
+ print("RESULT: PASS — Collisions detected from present position trajectory")
417
+ else:
418
+ print("RESULT: FAIL — Could not reliably detect collisions")
419
+ print(f"{'='*60}\n")
420
+
421
+ return 0 if result["success"] else 1
422
+
423
+
424
+ if __name__ == "__main__":
425
+ sys.exit(main())
tests/test_api.py CHANGED
@@ -585,27 +585,27 @@ class TestHfAutoLogin:
585
  """When HF login is detected, username appears in state."""
586
  marionette._hf_checked = False
587
  marionette._hf_username = None
588
- import marionette.main as mm
589
- original_whoami = mm.hf_whoami
590
- mm.hf_whoami = lambda: {"name": "testuser"}
591
  try:
592
  config = client.get("/api/state").json()["config"]
593
  assert config["hf_username"] == "testuser"
594
  finally:
595
- mm.hf_whoami = original_whoami
596
 
597
  def test_cached_after_first_check(self, marionette: Marionette):
598
  """HF login check is cached after first call."""
599
- import marionette.main as mm
600
  call_count = 0
601
- original_whoami = mm.hf_whoami
602
 
603
  def counting_whoami():
604
  nonlocal call_count
605
  call_count += 1
606
  return {"name": "cached-user"}
607
 
608
- mm.hf_whoami = counting_whoami
609
  marionette._hf_checked = False
610
  marionette._hf_username = None
611
  try:
@@ -615,17 +615,17 @@ class TestHfAutoLogin:
615
  assert result2 == "cached-user"
616
  assert call_count == 1
617
  finally:
618
- mm.hf_whoami = original_whoami
619
 
620
  def test_sync_without_username_uses_autodetected(
621
  self, client: TestClient, marionette: Marionette, tmp_path: Path
622
  ):
623
  """Sync endpoint uses auto-detected username when none provided."""
624
- import marionette.main as mm
625
- original_whoami = mm.hf_whoami
626
  marionette._hf_checked = False
627
  marionette._hf_username = None
628
- mm.hf_whoami = lambda: {"name": "auto-user"}
629
  try:
630
  # No moves exist, so sync will fail with 400 (no moves found),
631
  # but we verify it gets past the username check
@@ -633,26 +633,26 @@ class TestHfAutoLogin:
633
  # 404 = move not found (got past the username validation)
634
  assert resp.status_code == 404
635
  finally:
636
- mm.hf_whoami = original_whoami
637
 
638
  def test_sync_without_username_and_no_login_returns_400(
639
  self, client: TestClient, marionette: Marionette
640
  ):
641
  """Sync without username and no HF login returns 400."""
642
- import marionette.main as mm
643
- original_whoami = mm.hf_whoami
644
- original_get_token = mm.hf_get_token
645
  marionette._hf_checked = False
646
  marionette._hf_username = None
647
- mm.hf_whoami = None
648
- mm.hf_get_token = lambda: None
649
  try:
650
  resp = client.post("/api/datasets/sync", json={"move_ids": ["some-move"]})
651
  assert resp.status_code == 400
652
  assert "not logged in" in resp.json()["detail"].lower()
653
  finally:
654
- mm.hf_whoami = original_whoami
655
- mm.hf_get_token = original_get_token
656
 
657
 
658
  class TestHfTokenLogin:
@@ -668,11 +668,11 @@ class TestHfTokenLogin:
668
 
669
  def test_save_token_success(self, client: TestClient, marionette: Marionette):
670
  """Valid token saves and returns username."""
671
- import marionette.main as mm
672
- original_login = mm.hf_login
673
- original_whoami = mm.hf_whoami
674
- mm.hf_login = lambda token, add_to_git_credential: None
675
- mm.hf_whoami = lambda: {"name": "token-user"}
676
  try:
677
  resp = client.post("/api/hf-auth/save-token", json={"token": "hf_valid_token_12345"})
678
  assert resp.status_code == 200
@@ -683,14 +683,14 @@ class TestHfTokenLogin:
683
  config = client.get("/api/state").json()["config"]
684
  assert config["hf_username"] == "token-user"
685
  finally:
686
- mm.hf_login = original_login
687
- mm.hf_whoami = original_whoami
688
 
689
  def test_delete_token_success(self, client: TestClient, marionette: Marionette):
690
  """Logout clears the cached username."""
691
- import marionette.main as mm
692
- original_logout = mm.hf_logout
693
- mm.hf_logout = lambda: None
694
  marionette._hf_username = "some-user"
695
  marionette._hf_checked = True
696
  try:
@@ -700,15 +700,15 @@ class TestHfTokenLogin:
700
  assert marionette._hf_username is None
701
  assert marionette._hf_checked is False
702
  finally:
703
- mm.hf_logout = original_logout
704
 
705
  def test_delete_token_clears_state(self, client: TestClient, marionette: Marionette):
706
  """After logout, state reports no username."""
707
- import marionette.main as mm
708
- original_logout = mm.hf_logout
709
- original_whoami = mm.hf_whoami
710
- mm.hf_logout = lambda: None
711
- mm.hf_whoami = lambda: None # whoami returns None after logout
712
  marionette._hf_username = "old-user"
713
  marionette._hf_checked = True
714
  try:
@@ -716,8 +716,8 @@ class TestHfTokenLogin:
716
  config = client.get("/api/state").json()["config"]
717
  assert config["hf_username"] is None
718
  finally:
719
- mm.hf_logout = original_logout
720
- mm.hf_whoami = original_whoami
721
 
722
 
723
  class TestSensorData:
@@ -813,6 +813,9 @@ class TestTimingSync:
813
  def test_uploaded_audio_waits_for_capture_start(
814
  self, marionette: Marionette, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
815
  ):
 
 
 
816
  wav_path = tmp_path / "uploaded.wav"
817
  wav_path.write_bytes(b"fake")
818
  request = RecordingRequest(
@@ -827,15 +830,14 @@ class TestTimingSync:
827
 
828
  events: list[tuple[str, bool | None]] = []
829
 
830
- class _FakeMedia:
831
- def get_output_audio_samplerate(self):
832
- return 16000
833
-
834
- def start_playing(self):
835
- events.append(("pipeline_primed", None))
836
-
837
  class _FakeReachy:
838
- media = _FakeMedia()
 
 
 
 
 
 
839
 
840
  def fake_play_preloaded_wav(
841
  _reachy, _wav_data, _stop_event, chunk_duration=0.02, pipeline_ready=False, start_signal=None
@@ -845,6 +847,14 @@ class TestTimingSync:
845
  start_signal.wait(timeout=1.0)
846
  events.append(("audio_after_wait", start_signal.is_set() if start_signal else None))
847
 
 
 
 
 
 
 
 
 
848
  def fake_capture_motion(
849
  _reachy,
850
  _stop_event,
@@ -857,15 +867,17 @@ class TestTimingSync:
857
  on_capture_start()
858
  return [0.0, 0.01], [{"head": [], "antennas": [], "body_yaw": 0.0, "check_collision": False}], [], None
859
 
860
- monkeypatch.setattr(marionette, "_preload_wav", lambda *_a, **_k: (np.zeros(8, dtype=np.float32), 16000))
861
- monkeypatch.setattr(marionette, "_play_preloaded_wav", fake_play_preloaded_wav)
862
  monkeypatch.setattr(marionette, "_capture_motion", fake_capture_motion)
863
  monkeypatch.setattr(marionette, "_save_recording", lambda *_a, **_k: None)
864
  monkeypatch.setattr(marionette, "_refresh_recordings", lambda *_a, **_k: None)
865
 
866
- marionette._run_capture_and_save(_FakeReachy(), threading.Event(), request)
 
 
 
 
 
867
 
868
- assert ("pipeline_primed", None) in events
869
  assert ("audio_before_wait", False) in events
870
  assert ("audio_after_wait", True) in events
871
 
@@ -888,7 +900,7 @@ class TestTimingSync:
888
  def set_target_antenna_joint_positions(self, _antennas):
889
  return None
890
 
891
- monkeypatch.setattr("marionette.main.time.time", lambda: (_ for _ in ()).throw(RuntimeError("wall clock used")))
892
  marionette._playback_cancel_event.clear()
893
  assert marionette._stream_playback(_FakeReachy(), _FakeMove()) is True
894
 
@@ -949,15 +961,21 @@ class TestCommunityDatasets:
949
  resp = client.post("/api/datasets/download", json={"repo_id": "no-slash"})
950
  assert resp.status_code == 400
951
 
952
- def test_download_rejects_duplicate_folder(self, client: TestClient, marionette: Marionette):
953
  # Create a dataset first
954
  client.post("/api/datasets", json={"name": "existing-ds"})
955
- # Try to download with the same folder name
 
 
 
956
  resp = client.post("/api/datasets/download", json={
957
  "repo_id": "someone/existing-ds",
958
  "name": "existing-ds",
959
  })
960
- assert resp.status_code == 409
 
 
 
961
 
962
 
963
  # ──────── Corrupt data tests ────────────────────────────────────────
@@ -1071,23 +1089,23 @@ class TestSyncDatasetExtended:
1071
  assert resp.status_code == 422
1072
 
1073
  def test_sync_nonexistent_moves(self, client: TestClient, marionette: Marionette):
1074
- import marionette.main as mm
1075
- original_whoami = mm.hf_whoami
1076
  marionette._hf_checked = False
1077
  marionette._hf_username = None
1078
- mm.hf_whoami = lambda: {"name": "testuser"}
1079
  try:
1080
  resp = client.post("/api/datasets/sync", json={
1081
  "move_ids": ["fake-move-id"],
1082
  })
1083
  assert resp.status_code == 404
1084
  finally:
1085
- mm.hf_whoami = original_whoami
1086
 
1087
  def test_sync_no_active_dataset(self, client: TestClient, marionette: Marionette):
1088
- import marionette.main as mm
1089
- original_whoami = mm.hf_whoami
1090
- mm.hf_whoami = lambda: {"name": "testuser"}
1091
  marionette._hf_checked = False
1092
  marionette._hf_username = None
1093
 
@@ -1104,7 +1122,7 @@ class TestSyncDatasetExtended:
1104
  assert resp.status_code == 404
1105
  finally:
1106
  marionette._active_dataset_id = old_id
1107
- mm.hf_whoami = original_whoami
1108
 
1109
  def test_record_on_downloaded_dataset_rejected(
1110
  self, client: TestClient, marionette: Marionette
@@ -1118,81 +1136,6 @@ class TestSyncDatasetExtended:
1118
  assert "downloaded" in resp.json()["detail"].lower()
1119
 
1120
 
1121
- # ──────── Mic AGC config tests ────────────────────────────────────
1122
-
1123
-
1124
- class TestMicAgcConfig:
1125
- """Tests for _disable_mic_agc / _restore_mic_agc."""
1126
-
1127
- AGC_PATCH = "reachy_mini.media.audio_control_utils.init_respeaker_usb"
1128
-
1129
- def _make_mock_respeaker(self, agc_value=None):
1130
- """Create a mock ReSpeaker device."""
1131
- mock = MagicMock()
1132
- mock.read.return_value = agc_value if agc_value is not None else [1]
1133
- return mock
1134
-
1135
- def test_disable_agc_writes_zero(self, marionette: Marionette):
1136
- mock_dev = self._make_mock_respeaker(agc_value=[1])
1137
- with patch(self.AGC_PATCH, return_value=mock_dev):
1138
- marionette._disable_mic_agc()
1139
- mock_dev.read.assert_called_once_with("PP_AGCONOFF")
1140
- mock_dev.write.assert_called_once_with("PP_AGCONOFF", [0])
1141
- mock_dev.close.assert_called_once()
1142
- assert marionette._original_agc == [1]
1143
-
1144
- def test_disable_agc_saves_original_value(self, marionette: Marionette):
1145
- mock_dev = self._make_mock_respeaker(agc_value=[0])
1146
- with patch(self.AGC_PATCH, return_value=mock_dev):
1147
- marionette._disable_mic_agc()
1148
- assert marionette._original_agc == [0]
1149
-
1150
- def test_disable_agc_no_device(self, marionette: Marionette):
1151
- """Gracefully skips when no USB device is found."""
1152
- with patch(self.AGC_PATCH, return_value=None):
1153
- marionette._disable_mic_agc() # Should not raise
1154
- assert not hasattr(marionette, "_original_agc") or marionette._original_agc is None
1155
-
1156
- def test_disable_agc_import_error(self, marionette: Marionette):
1157
- """Gracefully skips when audio_control_utils is not importable."""
1158
- with patch.dict("sys.modules", {"reachy_mini.media.audio_control_utils": None}):
1159
- marionette._disable_mic_agc() # Should not raise
1160
-
1161
- def test_disable_agc_usb_error(self, marionette: Marionette):
1162
- """Gracefully handles USB communication errors."""
1163
- mock_dev = self._make_mock_respeaker()
1164
- mock_dev.read.side_effect = OSError("USB transfer failed")
1165
- with patch(self.AGC_PATCH, return_value=mock_dev):
1166
- marionette._disable_mic_agc() # Should not raise
1167
-
1168
- def test_restore_agc_writes_original(self, marionette: Marionette):
1169
- marionette._original_agc = [1]
1170
- mock_dev = self._make_mock_respeaker()
1171
- with patch(self.AGC_PATCH, return_value=mock_dev):
1172
- marionette._restore_mic_agc()
1173
- mock_dev.write.assert_called_once_with("PP_AGCONOFF", [1])
1174
- mock_dev.close.assert_called_once()
1175
-
1176
- def test_restore_agc_skips_when_no_original(self, marionette: Marionette):
1177
- """Skips restore when AGC was never disabled (no _original_agc)."""
1178
- mock_dev = self._make_mock_respeaker()
1179
- with patch(self.AGC_PATCH, return_value=mock_dev):
1180
- marionette._restore_mic_agc() # Should not raise
1181
- mock_dev.write.assert_not_called()
1182
-
1183
- def test_restore_agc_skips_when_original_is_none(self, marionette: Marionette):
1184
- marionette._original_agc = None
1185
- mock_dev = self._make_mock_respeaker()
1186
- with patch(self.AGC_PATCH, return_value=mock_dev):
1187
- marionette._restore_mic_agc()
1188
- mock_dev.write.assert_not_called()
1189
-
1190
- def test_restore_agc_device_gone(self, marionette: Marionette):
1191
- """Gracefully handles device disconnected at restore time."""
1192
- marionette._original_agc = [1]
1193
- with patch(self.AGC_PATCH, return_value=None):
1194
- marionette._restore_mic_agc() # Should not raise
1195
-
1196
 
1197
  # ──────── API contract tests (refactoring protection) ──────────────
1198
 
@@ -1448,3 +1391,323 @@ class TestVersionEndpoint:
1448
  data = client.get("/api/version").json()
1449
  assert isinstance(data["version"], str)
1450
  assert len(data["version"]) > 0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
585
  """When HF login is detected, username appears in state."""
586
  marionette._hf_checked = False
587
  marionette._hf_username = None
588
+ import marionette.datasets as md
589
+ original_whoami = md.hf_whoami
590
+ md.hf_whoami = lambda: {"name": "testuser"}
591
  try:
592
  config = client.get("/api/state").json()["config"]
593
  assert config["hf_username"] == "testuser"
594
  finally:
595
+ md.hf_whoami = original_whoami
596
 
597
  def test_cached_after_first_check(self, marionette: Marionette):
598
  """HF login check is cached after first call."""
599
+ import marionette.datasets as md
600
  call_count = 0
601
+ original_whoami = md.hf_whoami
602
 
603
  def counting_whoami():
604
  nonlocal call_count
605
  call_count += 1
606
  return {"name": "cached-user"}
607
 
608
+ md.hf_whoami = counting_whoami
609
  marionette._hf_checked = False
610
  marionette._hf_username = None
611
  try:
 
615
  assert result2 == "cached-user"
616
  assert call_count == 1
617
  finally:
618
+ md.hf_whoami = original_whoami
619
 
620
  def test_sync_without_username_uses_autodetected(
621
  self, client: TestClient, marionette: Marionette, tmp_path: Path
622
  ):
623
  """Sync endpoint uses auto-detected username when none provided."""
624
+ import marionette.datasets as md
625
+ original_whoami = md.hf_whoami
626
  marionette._hf_checked = False
627
  marionette._hf_username = None
628
+ md.hf_whoami = lambda: {"name": "auto-user"}
629
  try:
630
  # No moves exist, so sync will fail with 400 (no moves found),
631
  # but we verify it gets past the username check
 
633
  # 404 = move not found (got past the username validation)
634
  assert resp.status_code == 404
635
  finally:
636
+ md.hf_whoami = original_whoami
637
 
638
  def test_sync_without_username_and_no_login_returns_400(
639
  self, client: TestClient, marionette: Marionette
640
  ):
641
  """Sync without username and no HF login returns 400."""
642
+ import marionette.datasets as md
643
+ original_whoami = md.hf_whoami
644
+ original_get_token = md.hf_get_token
645
  marionette._hf_checked = False
646
  marionette._hf_username = None
647
+ md.hf_whoami = None
648
+ md.hf_get_token = lambda: None
649
  try:
650
  resp = client.post("/api/datasets/sync", json={"move_ids": ["some-move"]})
651
  assert resp.status_code == 400
652
  assert "not logged in" in resp.json()["detail"].lower()
653
  finally:
654
+ md.hf_whoami = original_whoami
655
+ md.hf_get_token = original_get_token
656
 
657
 
658
  class TestHfTokenLogin:
 
668
 
669
  def test_save_token_success(self, client: TestClient, marionette: Marionette):
670
  """Valid token saves and returns username."""
671
+ import marionette.datasets as md
672
+ original_login = md.hf_login
673
+ original_whoami = md.hf_whoami
674
+ md.hf_login = lambda token, add_to_git_credential: None
675
+ md.hf_whoami = lambda: {"name": "token-user"}
676
  try:
677
  resp = client.post("/api/hf-auth/save-token", json={"token": "hf_valid_token_12345"})
678
  assert resp.status_code == 200
 
683
  config = client.get("/api/state").json()["config"]
684
  assert config["hf_username"] == "token-user"
685
  finally:
686
+ md.hf_login = original_login
687
+ md.hf_whoami = original_whoami
688
 
689
  def test_delete_token_success(self, client: TestClient, marionette: Marionette):
690
  """Logout clears the cached username."""
691
+ import marionette.datasets as md
692
+ original_logout = md.hf_logout
693
+ md.hf_logout = lambda: None
694
  marionette._hf_username = "some-user"
695
  marionette._hf_checked = True
696
  try:
 
700
  assert marionette._hf_username is None
701
  assert marionette._hf_checked is False
702
  finally:
703
+ md.hf_logout = original_logout
704
 
705
  def test_delete_token_clears_state(self, client: TestClient, marionette: Marionette):
706
  """After logout, state reports no username."""
707
+ import marionette.datasets as md
708
+ original_logout = md.hf_logout
709
+ original_whoami = md.hf_whoami
710
+ md.hf_logout = lambda: None
711
+ md.hf_whoami = lambda: None # whoami returns None after logout
712
  marionette._hf_username = "old-user"
713
  marionette._hf_checked = True
714
  try:
 
716
  config = client.get("/api/state").json()["config"]
717
  assert config["hf_username"] is None
718
  finally:
719
+ md.hf_logout = original_logout
720
+ md.hf_whoami = original_whoami
721
 
722
 
723
  class TestSensorData:
 
813
  def test_uploaded_audio_waits_for_capture_start(
814
  self, marionette: Marionette, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
815
  ):
816
+ """Audio preload is now done in _perform_recording before countdown.
817
+ _run_capture_and_save receives the pre-built audio thread and events.
818
+ Verify the audio thread waits for audio_start (fired by on_capture_start)."""
819
  wav_path = tmp_path / "uploaded.wav"
820
  wav_path.write_bytes(b"fake")
821
  request = RecordingRequest(
 
830
 
831
  events: list[tuple[str, bool | None]] = []
832
 
 
 
 
 
 
 
 
833
  class _FakeReachy:
834
+ class media:
835
+ @staticmethod
836
+ def get_output_audio_samplerate():
837
+ return 16000
838
+
839
+ audio_start = threading.Event()
840
+ audio_stop = threading.Event()
841
 
842
  def fake_play_preloaded_wav(
843
  _reachy, _wav_data, _stop_event, chunk_duration=0.02, pipeline_ready=False, start_signal=None
 
847
  start_signal.wait(timeout=1.0)
848
  events.append(("audio_after_wait", start_signal.is_set() if start_signal else None))
849
 
850
+ audio_thread = threading.Thread(
851
+ target=fake_play_preloaded_wav,
852
+ args=(_FakeReachy(), (np.zeros(8, dtype=np.float32), 16000), audio_stop),
853
+ kwargs={"pipeline_ready": True, "start_signal": audio_start},
854
+ daemon=True,
855
+ )
856
+ audio_thread.start()
857
+
858
  def fake_capture_motion(
859
  _reachy,
860
  _stop_event,
 
867
  on_capture_start()
868
  return [0.0, 0.01], [{"head": [], "antennas": [], "body_yaw": 0.0, "check_collision": False}], [], None
869
 
 
 
870
  monkeypatch.setattr(marionette, "_capture_motion", fake_capture_motion)
871
  monkeypatch.setattr(marionette, "_save_recording", lambda *_a, **_k: None)
872
  monkeypatch.setattr(marionette, "_refresh_recordings", lambda *_a, **_k: None)
873
 
874
+ marionette._run_capture_and_save(
875
+ _FakeReachy(), threading.Event(), request,
876
+ audio_thread=audio_thread,
877
+ audio_start=audio_start,
878
+ audio_stop=audio_stop,
879
+ )
880
 
 
881
  assert ("audio_before_wait", False) in events
882
  assert ("audio_after_wait", True) in events
883
 
 
900
  def set_target_antenna_joint_positions(self, _antennas):
901
  return None
902
 
903
+ monkeypatch.setattr("marionette.recording.time.time", lambda: (_ for _ in ()).throw(RuntimeError("wall clock used")))
904
  marionette._playback_cancel_event.clear()
905
  assert marionette._stream_playback(_FakeReachy(), _FakeMove()) is True
906
 
 
961
  resp = client.post("/api/datasets/download", json={"repo_id": "no-slash"})
962
  assert resp.status_code == 400
963
 
964
+ def test_download_removes_existing_folder_on_redownload(self, client: TestClient, marionette: Marionette):
965
  # Create a dataset first
966
  client.post("/api/datasets", json={"name": "existing-ds"})
967
+ old_ids = {e.dataset_id for e in marionette._datasets.values() if e.folder == "existing-ds"}
968
+ assert old_ids, "dataset should exist"
969
+ # Re-download replaces the old entry (will fail at HF fetch, but the old entry
970
+ # should already be removed before the network call).
971
  resp = client.post("/api/datasets/download", json={
972
  "repo_id": "someone/existing-ds",
973
  "name": "existing-ds",
974
  })
975
+ # Network call fails → 502, but old entry was cleaned up
976
+ assert resp.status_code == 502
977
+ remaining = {e.dataset_id for e in marionette._datasets.values() if e.folder == "existing-ds"}
978
+ assert remaining.isdisjoint(old_ids), "old dataset entry should have been removed"
979
 
980
 
981
  # ──────── Corrupt data tests ────────────────────────────────────────
 
1089
  assert resp.status_code == 422
1090
 
1091
  def test_sync_nonexistent_moves(self, client: TestClient, marionette: Marionette):
1092
+ import marionette.datasets as md
1093
+ original_whoami = md.hf_whoami
1094
  marionette._hf_checked = False
1095
  marionette._hf_username = None
1096
+ md.hf_whoami = lambda: {"name": "testuser"}
1097
  try:
1098
  resp = client.post("/api/datasets/sync", json={
1099
  "move_ids": ["fake-move-id"],
1100
  })
1101
  assert resp.status_code == 404
1102
  finally:
1103
+ md.hf_whoami = original_whoami
1104
 
1105
  def test_sync_no_active_dataset(self, client: TestClient, marionette: Marionette):
1106
+ import marionette.datasets as md
1107
+ original_whoami = md.hf_whoami
1108
+ md.hf_whoami = lambda: {"name": "testuser"}
1109
  marionette._hf_checked = False
1110
  marionette._hf_username = None
1111
 
 
1122
  assert resp.status_code == 404
1123
  finally:
1124
  marionette._active_dataset_id = old_id
1125
+ md.hf_whoami = original_whoami
1126
 
1127
  def test_record_on_downloaded_dataset_rejected(
1128
  self, client: TestClient, marionette: Marionette
 
1136
  assert "downloaded" in resp.json()["detail"].lower()
1137
 
1138
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1139
 
1140
  # ──────── API contract tests (refactoring protection) ──────────────
1141
 
 
1391
  data = client.get("/api/version").json()
1392
  assert isinstance(data["version"], str)
1393
  assert len(data["version"]) > 0
1394
+
1395
+
1396
+ # ──────── Robot audio endpoint tests ─────────────────────────────
1397
+
1398
+
1399
+ class TestRobotAudioEndpoints:
1400
+ """Tests for GET /api/robot-audio and POST /api/robot-audio/select."""
1401
+
1402
+ def test_robot_audio_list_empty_by_default(self, client: TestClient):
1403
+ resp = client.get("/api/robot-audio")
1404
+ assert resp.status_code == 200
1405
+ data = resp.json()
1406
+ assert "files" in data
1407
+ assert isinstance(data["files"], list)
1408
+
1409
+ def test_robot_audio_list_finds_audio_only_wav(
1410
+ self, client: TestClient, marionette: Marionette
1411
+ ):
1412
+ """Audio-only recordings appear in robot-audio list."""
1413
+ from conftest import make_wav_bytes
1414
+ import json
1415
+
1416
+ wav_bytes = make_wav_bytes(1.0)
1417
+ wav_path = marionette._dataset_dir / "robot-test.wav"
1418
+ wav_path.write_bytes(wav_bytes)
1419
+ json_path = marionette._dataset_dir / "robot-test.json"
1420
+ json_path.write_text(json.dumps({
1421
+ "time": [0.0], "set_target_data": [], "audio_only": True,
1422
+ }))
1423
+ marionette._refresh_recordings()
1424
+
1425
+ resp = client.get("/api/robot-audio")
1426
+ assert resp.status_code == 200
1427
+ files = resp.json()["files"]
1428
+ names = [f["name"] for f in files]
1429
+ assert "robot-test" in names
1430
+
1431
+ # Each file should have name, path, duration_seconds
1432
+ for f in files:
1433
+ assert "name" in f
1434
+ assert "path" in f
1435
+ assert "duration_seconds" in f
1436
+
1437
+ # Cleanup
1438
+ wav_path.unlink(missing_ok=True)
1439
+ json_path.unlink(missing_ok=True)
1440
+ marionette._refresh_recordings()
1441
+
1442
+ def test_robot_audio_excludes_non_audio_only(
1443
+ self, client: TestClient, marionette: Marionette
1444
+ ):
1445
+ """Normal motion moves with WAV files should NOT appear in robot-audio list."""
1446
+ from conftest import make_wav_bytes
1447
+ import json
1448
+
1449
+ wav_bytes = make_wav_bytes(1.0)
1450
+ wav_path = marionette._dataset_dir / "motion-move.wav"
1451
+ wav_path.write_bytes(wav_bytes)
1452
+ json_path = marionette._dataset_dir / "motion-move.json"
1453
+ json_path.write_text(json.dumps({
1454
+ "time": [0.0, 0.01],
1455
+ "set_target_data": [
1456
+ {"head": [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]], "antennas": [0,0]},
1457
+ {"head": [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]], "antennas": [0,0]},
1458
+ ],
1459
+ }))
1460
+ marionette._refresh_recordings()
1461
+
1462
+ resp = client.get("/api/robot-audio")
1463
+ assert resp.status_code == 200
1464
+ files = resp.json()["files"]
1465
+ names = [f["name"] for f in files]
1466
+ assert "motion-move" not in names
1467
+
1468
+ # Cleanup
1469
+ wav_path.unlink(missing_ok=True)
1470
+ json_path.unlink(missing_ok=True)
1471
+ marionette._refresh_recordings()
1472
+
1473
+ def test_robot_audio_select_returns_upload_id(
1474
+ self, client: TestClient, marionette: Marionette
1475
+ ):
1476
+ """POST /api/robot-audio/select with valid WAV returns upload_id."""
1477
+ from conftest import make_wav_bytes
1478
+
1479
+ wav_bytes = make_wav_bytes(1.5)
1480
+ wav_path = marionette._dataset_dir / "selectable.wav"
1481
+ wav_path.write_bytes(wav_bytes)
1482
+
1483
+ resp = client.post(
1484
+ "/api/robot-audio/select",
1485
+ json={"path": str(wav_path)},
1486
+ )
1487
+ assert resp.status_code == 200
1488
+ data = resp.json()
1489
+ assert "upload_id" in data
1490
+ assert data["filename"] == "selectable.wav"
1491
+
1492
+ # Cleanup
1493
+ wav_path.unlink(missing_ok=True)
1494
+
1495
+ def test_robot_audio_select_rejects_path_traversal(self, client: TestClient):
1496
+ """Path outside dataset/temp dirs is rejected with 403."""
1497
+ resp = client.post(
1498
+ "/api/robot-audio/select",
1499
+ json={"path": "/etc/passwd"},
1500
+ )
1501
+ assert resp.status_code in (403, 404)
1502
+
1503
+ def test_robot_audio_select_rejects_nonexistent(self, client: TestClient):
1504
+ """Non-existent file returns 404."""
1505
+ resp = client.post(
1506
+ "/api/robot-audio/select",
1507
+ json={"path": "/tmp/does-not-exist-at-all.wav"},
1508
+ )
1509
+ assert resp.status_code in (403, 404)
1510
+
1511
+ def test_robot_audio_select_rejects_non_wav(
1512
+ self, client: TestClient, marionette: Marionette
1513
+ ):
1514
+ """Non-WAV file within allowed dirs is rejected."""
1515
+ txt_path = marionette._dataset_dir / "notes.txt"
1516
+ txt_path.write_text("not audio")
1517
+
1518
+ resp = client.post(
1519
+ "/api/robot-audio/select",
1520
+ json={"path": str(txt_path)},
1521
+ )
1522
+ assert resp.status_code == 400
1523
+
1524
+ # Cleanup
1525
+ txt_path.unlink(missing_ok=True)
1526
+
1527
+
1528
+ # ──────── Move metadata tests ────────────────────────────────────
1529
+
1530
+
1531
+ class TestMoveMetadata:
1532
+ """Tests for move metadata fields in the state endpoint."""
1533
+
1534
+ def test_move_created_at_is_float(
1535
+ self, client: TestClient, marionette: Marionette, sample_move_json: dict
1536
+ ):
1537
+ """created_at should be a numeric Unix timestamp."""
1538
+ data_dir = marionette._dataset_dir
1539
+ (data_dir / "meta-test.json").write_text(json.dumps(sample_move_json))
1540
+ marionette._refresh_recordings()
1541
+
1542
+ state = client.get("/api/state").json()
1543
+ move = next((m for m in state["moves"] if m["id"] == "meta-test"), None)
1544
+ assert move is not None
1545
+ assert isinstance(move["created_at"], float)
1546
+ assert move["created_at"] > 0
1547
+
1548
+ # Cleanup
1549
+ (data_dir / "meta-test.json").unlink(missing_ok=True)
1550
+ marionette._refresh_recordings()
1551
+
1552
+ def test_move_created_at_is_recent(
1553
+ self, client: TestClient, marionette: Marionette, sample_move_json: dict
1554
+ ):
1555
+ """A freshly created move should have a recent timestamp."""
1556
+ import time
1557
+
1558
+ data_dir = marionette._dataset_dir
1559
+ (data_dir / "recent-test.json").write_text(json.dumps(sample_move_json))
1560
+ marionette._refresh_recordings()
1561
+
1562
+ state = client.get("/api/state").json()
1563
+ move = next((m for m in state["moves"] if m["id"] == "recent-test"), None)
1564
+ assert move is not None
1565
+ # Should be within last 60 seconds
1566
+ assert abs(move["created_at"] - time.time()) < 60
1567
+
1568
+ # Cleanup
1569
+ (data_dir / "recent-test.json").unlink(missing_ok=True)
1570
+ marionette._refresh_recordings()
1571
+
1572
+ def test_audio_only_move_in_robot_audio_list(
1573
+ self, client: TestClient, marionette: Marionette
1574
+ ):
1575
+ """Audio-only recordings with WAV files should appear in robot-audio list."""
1576
+ from conftest import make_wav_bytes
1577
+
1578
+ data_dir = marionette._dataset_dir
1579
+ move_data = {
1580
+ "description": "audio only for robot list",
1581
+ "audio_only": True,
1582
+ "time": [0.0, 0.01, 0.02],
1583
+ "set_target_data": [],
1584
+ }
1585
+ (data_dir / "ao-robot.json").write_text(json.dumps(move_data))
1586
+ (data_dir / "ao-robot.wav").write_bytes(make_wav_bytes(1.0))
1587
+ marionette._refresh_recordings()
1588
+
1589
+ resp = client.get("/api/robot-audio")
1590
+ assert resp.status_code == 200
1591
+ names = [f["name"] for f in resp.json()["files"]]
1592
+ assert "ao-robot" in names
1593
+
1594
+ # Cleanup
1595
+ (data_dir / "ao-robot.json").unlink(missing_ok=True)
1596
+ (data_dir / "ao-robot.wav").unlink(missing_ok=True)
1597
+ marionette._refresh_recordings()
1598
+
1599
+ def test_record_with_uploaded_audio_accepted(
1600
+ self, client: TestClient, marionette: Marionette
1601
+ ):
1602
+ """Upload audio, then start a recording using that audio ID."""
1603
+ from conftest import make_wav_bytes
1604
+
1605
+ wav = make_wav_bytes(2.0)
1606
+ upload_resp = client.post(
1607
+ "/api/upload-audio",
1608
+ files={"file": ("track.wav", BytesIO(wav), "audio/wav")},
1609
+ )
1610
+ assert upload_resp.status_code == 200
1611
+ upload_id = upload_resp.json()["upload_id"]
1612
+
1613
+ resp = client.post("/api/record", json={
1614
+ "duration": 2.0,
1615
+ "record_audio": False,
1616
+ "record_motion": True,
1617
+ "uploaded_audio_id": upload_id,
1618
+ "label": "with-audio",
1619
+ })
1620
+ assert resp.status_code == 200
1621
+ data = resp.json()
1622
+ assert data["accepted"] is True
1623
+ assert marionette._pending_recording is not None
1624
+ assert marionette._pending_recording.uploaded_audio_path is not None
1625
+
1626
+ # Cleanup
1627
+ marionette._set_idle_state()
1628
+ marionette._pending_recording = None
1629
+
1630
+
1631
+ # ──────── Audio analysis unit tests ───────────────────────────────
1632
+
1633
+
1634
+ class TestAudioAnalysis:
1635
+ """Unit tests for the audio analysis utilities (no hardware needed)."""
1636
+
1637
+ def test_generate_sync_test_audio(self):
1638
+ from audio_analysis import generate_sync_test_audio
1639
+
1640
+ audio, beep_times = generate_sync_test_audio(sr=48000, duration=8.0)
1641
+ assert audio.shape == (8 * 48000,)
1642
+ assert len(beep_times) == 5
1643
+ # Audio should have non-zero samples at beep positions
1644
+ for t in beep_times:
1645
+ idx = int(t * 48000)
1646
+ assert np.max(np.abs(audio[idx : idx + 4800])) > 0.1
1647
+
1648
+ def test_detect_beep_onsets_synthetic(self):
1649
+ from audio_analysis import detect_beep_onsets, generate_sync_test_audio
1650
+
1651
+ sr = 48000
1652
+ audio, expected_times = generate_sync_test_audio(sr=sr, duration=8.0)
1653
+ detected = detect_beep_onsets(audio, sr, freq=1000.0)
1654
+
1655
+ # Should detect all 5 beeps
1656
+ assert len(detected) >= 4, f"Only detected {len(detected)}/5 beeps"
1657
+
1658
+ # Each detected onset should be within 100ms of an expected time
1659
+ for dt in detected:
1660
+ distances = [abs(dt - et) for et in expected_times]
1661
+ assert min(distances) < 0.1, f"Detected onset {dt:.3f}s doesn't match any expected beep"
1662
+
1663
+ def test_generate_collision_trajectory(self):
1664
+ from audio_analysis import generate_collision_trajectory
1665
+
1666
+ beep_times = [1.0, 2.5, 4.0]
1667
+ timestamps, frames = generate_collision_trajectory(beep_times, duration=5.0)
1668
+
1669
+ assert len(timestamps) == 500 # 5s * 100Hz
1670
+ assert len(frames) == 500
1671
+
1672
+ # At a beep time, antennas should be at 0 (collided)
1673
+ idx_at_beep = int(1.0 * 100)
1674
+ antennas = frames[idx_at_beep]["antennas"]
1675
+ assert abs(antennas[0]) < 0.05, f"Antennas should be near 0 at collision: {antennas}"
1676
+
1677
+ # Between beeps, antennas should be apart
1678
+ idx_between = int(1.5 * 100)
1679
+ antennas = frames[idx_between]["antennas"]
1680
+ assert abs(antennas[0]) > 0.1, f"Antennas should be apart between beeps: {antennas}"
1681
+
1682
+ def test_measure_sync_offsets(self):
1683
+ from audio_analysis import measure_sync_offsets
1684
+
1685
+ beep_onsets = [1.0, 2.5, 4.0, 5.0, 7.0]
1686
+ # Collisions arrive 50ms after each beep
1687
+ collision_onsets = [1.05, 2.55, 4.05, 5.05, 7.05]
1688
+
1689
+ result = measure_sync_offsets(beep_onsets, collision_onsets)
1690
+ assert result["n_matched"] == 5
1691
+ assert abs(result["mean_offset_ms"] - 50.0) < 1.0
1692
+ assert result["max_offset_ms"] < 55.0
1693
+
1694
+ def test_measure_sync_offsets_missing_collisions(self):
1695
+ from audio_analysis import measure_sync_offsets
1696
+
1697
+ beep_onsets = [1.0, 2.5, 4.0, 5.0, 7.0]
1698
+ collision_onsets = [1.02, 4.01] # Only 2 matched
1699
+
1700
+ result = measure_sync_offsets(beep_onsets, collision_onsets)
1701
+ assert result["n_matched"] == 2
1702
+ assert result["n_beeps"] == 5
1703
+
1704
+ def test_datasets_payload_sorted_by_label(self, marionette: Marionette, tmp_path: Path):
1705
+ """Verify _datasets_payload returns entries sorted by label."""
1706
+ ds_root = marionette._dataset_root
1707
+ (ds_root / "zzz_first_label").mkdir(exist_ok=True)
1708
+ (ds_root / "aaa_last_label").mkdir(exist_ok=True)
1709
+ marionette._load_dataset_registry()
1710
+
1711
+ payload = marionette._datasets_payload()
1712
+ labels = [e.get("label", e["id"]) for e in payload["entries"]]
1713
+ assert labels == sorted(labels, key=str.lower), f"Entries not sorted by label: {labels}"
tests/test_audio_roundtrip.py ADDED
@@ -0,0 +1,314 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Standalone audio roundtrip test: generate beeps, play on robot, record on laptop mic.
3
+
4
+ No Marionette involved. Tests raw audio pipeline:
5
+ laptop generates WAV → SCP to robot → robot plays via reachy_mini SDK
6
+ → laptop mic records → analyze for beep detection
7
+
8
+ Usage:
9
+ python tests/test_audio_roundtrip.py [--host reachy-mini.local]
10
+ """
11
+ from __future__ import annotations
12
+
13
+ import argparse
14
+ import subprocess
15
+ import sys
16
+ import tempfile
17
+ import time
18
+ from pathlib import Path
19
+
20
+ import numpy as np
21
+ import sounddevice as sd
22
+ import soundfile as sf
23
+
24
+ # Add tests/ to path for audio_analysis imports
25
+ sys.path.insert(0, str(Path(__file__).parent))
26
+ from audio_analysis import detect_beep_onsets, generate_sync_test_audio
27
+
28
+ ROBOT_USER = "pollen"
29
+ ROBOT_PYTHON = "/venvs/apps_venv/bin/python"
30
+ ROBOT_SR = 16000 # robot output sample rate
31
+ LAPTOP_SR = 48000 # laptop mic sample rate
32
+
33
+
34
+ BEEP_FREQ = 2000.0 # Hz — away from speech/noise band
35
+ BEEP_DURATION = 0.2 # seconds — long enough for strong detection
36
+ BEEP_AMPLITUDE = 0.9 # loud
37
+
38
+
39
+ def generate_test_wav(path: Path, sr: int = ROBOT_SR) -> list[float]:
40
+ """Generate a WAV file with 5 loud beeps at non-periodic timestamps."""
41
+ # Non-periodic gaps (1.3, 1.7, 2.3, 3.1s) — no ambiguity when aligning
42
+ beep_times = [0.5, 1.8, 3.5, 5.8, 8.9]
43
+ duration = 10.0
44
+ n_total = int(sr * duration)
45
+ audio = np.zeros(n_total, dtype=np.float32)
46
+
47
+ for t in beep_times:
48
+ start = int(t * sr)
49
+ n_beep = int(BEEP_DURATION * sr)
50
+ if start + n_beep > n_total:
51
+ continue
52
+ t_arr = np.arange(n_beep, dtype=np.float32) / sr
53
+ beep = BEEP_AMPLITUDE * np.sin(2 * np.pi * BEEP_FREQ * t_arr).astype(np.float32)
54
+ # Fade in/out (5ms)
55
+ fade = int(0.005 * sr)
56
+ if fade > 0 and 2 * fade < n_beep:
57
+ beep[:fade] *= np.linspace(0, 1, fade, dtype=np.float32)
58
+ beep[-fade:] *= np.linspace(1, 0, fade, dtype=np.float32)
59
+ audio[start : start + n_beep] += beep
60
+
61
+ sf.write(str(path), audio, sr)
62
+ print(f" Generated {path.name}: {duration}s, {sr}Hz, {BEEP_FREQ}Hz beeps, {len(beep_times)} at {beep_times}")
63
+ return beep_times
64
+
65
+
66
+ def scp_to_robot(local_path: Path, remote_path: str, host: str) -> None:
67
+ """Copy a file to the robot via SCP."""
68
+ target = f"{ROBOT_USER}@{host}:{remote_path}"
69
+ result = subprocess.run(
70
+ ["scp", "-o", "ConnectTimeout=5", str(local_path), target],
71
+ capture_output=True, text=True, timeout=15,
72
+ )
73
+ if result.returncode != 0:
74
+ raise RuntimeError(f"SCP failed: {result.stderr}")
75
+ print(f" Copied to {target}")
76
+
77
+
78
+ ROBOT_PLAY_SCRIPT = """\
79
+ import numpy as np
80
+ import soundfile as sf
81
+ import os, sys, time
82
+
83
+ wav_path = sys.argv[1]
84
+ print("robot: loading WAV " + wav_path, flush=True)
85
+ data, sr = sf.read(wav_path, dtype="float32")
86
+ if data.ndim > 1:
87
+ data = data[:, 0] # mono
88
+
89
+ print("robot: connecting to ReachyMini", flush=True)
90
+ from reachy_mini import ReachyMini
91
+ r = ReachyMini()
92
+
93
+ robot_sr = int(r.media.get_output_audio_samplerate() or 16000)
94
+ print(f"robot: output sr={robot_sr}, file sr={sr}", flush=True)
95
+
96
+ # Resample if needed
97
+ if sr != robot_sr:
98
+ from scipy.signal import resample
99
+ n_out = int(len(data) * robot_sr / sr)
100
+ data = resample(data, n_out).astype(np.float32)
101
+ print(f"robot: resampled to {robot_sr}Hz ({len(data)} samples)", flush=True)
102
+
103
+ # Play via push_audio_sample at real-time speed
104
+ chunk_duration = 0.02 # 20ms
105
+ chunk_size = int(robot_sr * chunk_duration)
106
+ r.media.start_playing()
107
+
108
+ print("robot: MARK_START", flush=True)
109
+ t0 = time.monotonic()
110
+ for i in range(0, len(data), chunk_size):
111
+ chunk = data[i:i+chunk_size]
112
+ if len(chunk) < chunk_size:
113
+ chunk = np.pad(chunk, (0, chunk_size - len(chunk)))
114
+ r.media.push_audio_sample(chunk)
115
+ # Sleep to maintain real-time pacing
116
+ elapsed = time.monotonic() - t0
117
+ target = (i + chunk_size) / robot_sr
118
+ if target > elapsed:
119
+ time.sleep(target - elapsed)
120
+
121
+ elapsed = time.monotonic() - t0
122
+ print(f"robot: pushed {len(data)} samples in {elapsed:.3f}s (audio={len(data)/robot_sr:.3f}s)", flush=True)
123
+
124
+ # Wait for buffer to drain
125
+ time.sleep(0.5)
126
+ r.media.stop_playing()
127
+ print("robot: done", flush=True)
128
+ os._exit(0)
129
+ """
130
+
131
+ REMOTE_PLAY_SCRIPT = "/tmp/roundtrip_play.py"
132
+
133
+
134
+ def play_on_robot(host: str, wav_path: str) -> subprocess.Popen:
135
+ """Start playback on the robot via SSH (non-blocking).
136
+
137
+ SCPs a playback script to the robot, then executes it.
138
+ Returns the Popen process so we can wait/kill it.
139
+ """
140
+ # Write script to a temp file and SCP it
141
+ with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
142
+ f.write(ROBOT_PLAY_SCRIPT)
143
+ local_script = Path(f.name)
144
+ try:
145
+ scp_to_robot(local_script, REMOTE_PLAY_SCRIPT, host)
146
+ finally:
147
+ local_script.unlink()
148
+
149
+ proc = subprocess.Popen(
150
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{host}",
151
+ f"{ROBOT_PYTHON} {REMOTE_PLAY_SCRIPT} {wav_path}"],
152
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
153
+ )
154
+ return proc
155
+
156
+
157
+ def record_from_mic(duration: float, sr: int = LAPTOP_SR) -> np.ndarray:
158
+ """Record from default laptop mic for `duration` seconds."""
159
+ print(f" Recording from laptop mic for {duration:.1f}s at {sr}Hz...")
160
+ audio = sd.rec(int(duration * sr), samplerate=sr, channels=1, dtype="float32")
161
+ sd.wait()
162
+ return audio.flatten()
163
+
164
+
165
+ def analyze_recording(
166
+ captured: np.ndarray,
167
+ sr: int,
168
+ expected_beep_times: list[float],
169
+ beep_freq: float = BEEP_FREQ,
170
+ ) -> dict:
171
+ """Analyze captured audio for beep detection."""
172
+ detected = detect_beep_onsets(
173
+ captured, sr, freq=beep_freq, bandwidth=150.0, threshold_db=-12.0,
174
+ min_separation=1.0, # beeps are ≥1.3s apart
175
+ )
176
+
177
+ print(f"\n=== Analysis ===")
178
+ print(f" Captured: {len(captured)} samples ({len(captured)/sr:.2f}s)")
179
+ print(f" Expected beeps: {len(expected_beep_times)} at {expected_beep_times}")
180
+ print(f" Detected beeps: {len(detected)} at {[f'{t:.3f}' for t in detected]}")
181
+
182
+ # Compute RMS level
183
+ rms = np.sqrt(np.mean(captured ** 2))
184
+ peak = np.max(np.abs(captured))
185
+ print(f" Audio levels: RMS={rms:.6f}, peak={peak:.6f}")
186
+
187
+ if not detected:
188
+ print(" FAIL: No beeps detected!")
189
+ return {"success": False, "detected": 0, "expected": len(expected_beep_times)}
190
+
191
+ # Try to match detected beeps to expected (with unknown offset)
192
+ # The mic recording starts before playback, so there's an offset
193
+ best_offset = None
194
+ best_matches = 0
195
+ for d in detected:
196
+ for e in expected_beep_times:
197
+ candidate_offset = d - e
198
+ matches = 0
199
+ for et in expected_beep_times:
200
+ shifted = et + candidate_offset
201
+ if any(abs(shifted - dt) < 0.15 for dt in detected):
202
+ matches += 1
203
+ if matches > best_matches:
204
+ best_matches = matches
205
+ best_offset = candidate_offset
206
+
207
+ print(f" Best alignment: offset={best_offset:.3f}s, matched {best_matches}/{len(expected_beep_times)}")
208
+
209
+ if best_offset is not None:
210
+ for et in expected_beep_times:
211
+ shifted = et + best_offset
212
+ match = min(detected, key=lambda d: abs(d - shifted))
213
+ error_ms = (match - shifted) * 1000
214
+ matched = abs(match - shifted) < 0.15
215
+ status = "OK" if matched else "MISS"
216
+ print(f" expected {et:.1f}s → shifted {shifted:.3f}s, nearest {match:.3f}s ({error_ms:+.1f}ms) [{status}]")
217
+
218
+ success = best_matches >= max(1, len(expected_beep_times) - 1)
219
+ print(f"\n {'PASS' if success else 'FAIL'}: {best_matches}/{len(expected_beep_times)} beeps matched")
220
+
221
+ return {
222
+ "success": success,
223
+ "detected": len(detected),
224
+ "expected": len(expected_beep_times),
225
+ "matched": best_matches,
226
+ "offset_s": best_offset,
227
+ }
228
+
229
+
230
+ def main():
231
+ parser = argparse.ArgumentParser(description="Audio roundtrip test")
232
+ parser.add_argument("--host", default="reachy-mini.local", help="Robot hostname/IP")
233
+ args = parser.parse_args()
234
+
235
+ print(f"\n{'='*60}")
236
+ print("Audio Roundtrip Test: Laptop → Robot speakers → Laptop mic")
237
+ print(f"{'='*60}\n")
238
+
239
+ # Step 1: Stop any running app
240
+ print("[1/5] Stopping any running app on robot...")
241
+ subprocess.run(
242
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{args.host}",
243
+ "curl -sf -X POST http://127.0.0.1:8000/api/apps/stop-current-app >/dev/null 2>&1 || true"],
244
+ capture_output=True, timeout=10,
245
+ )
246
+ time.sleep(1)
247
+ print(" Done")
248
+
249
+ # Step 2: Generate test WAV
250
+ print("\n[2/5] Generating test audio...")
251
+ with tempfile.TemporaryDirectory() as tmpdir:
252
+ wav_path = Path(tmpdir) / "beeps.wav"
253
+ expected_times = generate_test_wav(wav_path, sr=ROBOT_SR)
254
+
255
+ # Step 3: SCP to robot
256
+ print("\n[3/5] Copying WAV to robot...")
257
+ remote_wav = "/tmp/roundtrip_test_beeps.wav"
258
+ scp_to_robot(wav_path, remote_wav, args.host)
259
+
260
+ # Also save locally for reference
261
+ local_ref = Path(tmpdir) / "beeps_reference.wav"
262
+
263
+ # Step 4: Start mic recording, then trigger playback
264
+ print("\n[4/5] Starting mic recording + robot playback...")
265
+ record_duration = 25.0 # 10s audio + ~8s SSH startup + buffer
266
+
267
+ # Start mic recording in background
268
+ print(f" Starting {record_duration}s mic recording...")
269
+ mic_data = sd.rec(
270
+ int(record_duration * LAPTOP_SR),
271
+ samplerate=LAPTOP_SR, channels=1, dtype="float32",
272
+ )
273
+
274
+ # Give mic a moment to start, then trigger robot playback
275
+ time.sleep(0.5)
276
+ print(" Triggering robot playback...")
277
+ proc = play_on_robot(args.host, remote_wav)
278
+
279
+ # Wait for both mic recording and robot playback to finish
280
+ sd.wait()
281
+ print(" Mic recording done")
282
+
283
+ # Collect robot output
284
+ try:
285
+ stdout, _ = proc.communicate(timeout=30)
286
+ for line in stdout.strip().split("\n"):
287
+ print(f" {line}")
288
+ except subprocess.TimeoutExpired:
289
+ proc.kill()
290
+ print(" WARNING: Robot playback process timed out")
291
+
292
+ captured = mic_data.flatten()
293
+
294
+ # Save captured audio for manual inspection
295
+ captured_path = Path("tests/captured_roundtrip.wav")
296
+ sf.write(str(captured_path), captured, LAPTOP_SR)
297
+ print(f" Saved captured audio to {captured_path}")
298
+
299
+ # Step 5: Analyze
300
+ print("\n[5/5] Analyzing captured audio...")
301
+ result = analyze_recording(captured, LAPTOP_SR, expected_times)
302
+
303
+ print(f"\n{'='*60}")
304
+ if result["success"]:
305
+ print("RESULT: PASS — Beeps successfully detected through roundtrip")
306
+ else:
307
+ print("RESULT: FAIL — Could not reliably detect beeps")
308
+ print(f"{'='*60}\n")
309
+
310
+ return 0 if result["success"] else 1
311
+
312
+
313
+ if __name__ == "__main__":
314
+ sys.exit(main())
tests/test_beep_collision_sync.py ADDED
@@ -0,0 +1,486 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Combined beep + collision sync test.
3
+
4
+ The robot plays beeps through its speaker AND performs antenna collisions,
5
+ with exactly 1.0s between each beep and its corresponding collision.
6
+ The laptop mic records everything. Since both events are detected from
7
+ the same mic recording, the measured interval is free of cross-clock bias.
8
+
9
+ If audio and motion are perfectly synced, each beep-collision pair should
10
+ be exactly 1.0s apart in the mic recording. Deviations measure the true
11
+ audio-motion sync error.
12
+
13
+ Beep times (non-periodic, gaps 1.3/1.7/2.3/3.1s):
14
+ [1.0, 2.3, 4.0, 6.3, 9.4]
15
+ Collision times (each beep + 1.0s):
16
+ [2.0, 3.3, 5.0, 7.3, 10.4]
17
+
18
+ Usage:
19
+ python tests/test_beep_collision_sync.py [--host reachy-mini.local]
20
+ """
21
+ from __future__ import annotations
22
+
23
+ import argparse
24
+ import json
25
+ import subprocess
26
+ import sys
27
+ import tempfile
28
+ import time
29
+ from pathlib import Path
30
+
31
+ import numpy as np
32
+ import sounddevice as sd
33
+ import soundfile as sf
34
+
35
+ sys.path.insert(0, str(Path(__file__).parent))
36
+ from audio_analysis import detect_beep_onsets, detect_transient_onsets
37
+
38
+ # Timing — non-periodic gaps (1.3, 1.7, 2.3, 3.1s)
39
+ BEEP_TIMES = [1.0, 2.3, 4.0, 6.3, 9.4]
40
+ BEEP_COLLISION_OFFSET = 1.0 # seconds between beep and its collision
41
+ COLLISION_TIMES = [t + BEEP_COLLISION_OFFSET for t in BEEP_TIMES]
42
+
43
+ # Audio parameters
44
+ BEEP_FREQ = 2000.0
45
+ BEEP_DURATION = 0.2
46
+ BEEP_AMPLITUDE = 0.9
47
+ ROBOT_SR = 16000
48
+
49
+ # Collision parameters
50
+ RIGHT_REST = -0.68
51
+ LEFT_REST = 0.0
52
+ LEFT_COLLISION = 0.70
53
+ HOLD_DURATION = 0.2
54
+
55
+ ROBOT_USER = "pollen"
56
+ ROBOT_PYTHON = "/venvs/apps_venv/bin/python"
57
+ LAPTOP_SR = 48000
58
+ MIC_DURATION = 30.0
59
+ REMOTE_RESULTS = "/tmp/beep_collision_results.json"
60
+
61
+ ROBOT_SCRIPT = """\
62
+ import numpy as np
63
+ import os, sys, time, json
64
+
65
+ beep_times = json.loads(sys.argv[1])
66
+ collision_times = json.loads(sys.argv[2])
67
+ right_rest = float(sys.argv[3])
68
+ left_rest = float(sys.argv[4])
69
+ left_collision = float(sys.argv[5])
70
+ hold_duration = float(sys.argv[6])
71
+ wav_path = sys.argv[7]
72
+ results_path = sys.argv[8]
73
+ beep_freq = float(sys.argv[9])
74
+ beep_duration = float(sys.argv[10])
75
+ beep_amplitude = float(sys.argv[11])
76
+
77
+ print("robot: connecting to ReachyMini", flush=True)
78
+ from reachy_mini import ReachyMini
79
+ from reachy_mini.utils import create_head_pose
80
+ r = ReachyMini()
81
+
82
+ robot_sr = int(r.media.get_output_audio_samplerate() or 16000)
83
+ print(f"robot: audio sr={robot_sr}", flush=True)
84
+
85
+ # Generate beep audio
86
+ total_duration = max(collision_times) + hold_duration + 1.0
87
+ n_total = int(robot_sr * total_duration)
88
+ audio = np.zeros(n_total, dtype=np.float32)
89
+ for bt in beep_times:
90
+ start = int(bt * robot_sr)
91
+ n_beep = int(beep_duration * robot_sr)
92
+ if start + n_beep > n_total:
93
+ continue
94
+ t_arr = np.arange(n_beep, dtype=np.float32) / robot_sr
95
+ beep = beep_amplitude * np.sin(2 * np.pi * beep_freq * t_arr).astype(np.float32)
96
+ fade = int(0.005 * robot_sr)
97
+ if fade > 0 and 2 * fade < n_beep:
98
+ beep[:fade] *= np.linspace(0, 1, fade, dtype=np.float32)
99
+ beep[-fade:] *= np.linspace(1, 0, fade, dtype=np.float32)
100
+ audio[start:start + n_beep] += beep
101
+
102
+ print(f"robot: generated {total_duration:.1f}s audio with {len(beep_times)} beeps", flush=True)
103
+
104
+ # Build collision timeline at 50Hz
105
+ DT = 0.02
106
+ n_steps = int(total_duration / DT)
107
+ left_targets = np.full(n_steps, left_rest, dtype=np.float64)
108
+ for ct in collision_times:
109
+ start_step = int(ct / DT)
110
+ end_step = int((ct + hold_duration) / DT)
111
+ end_step = min(end_step, n_steps)
112
+ left_targets[start_step:end_step] = left_collision
113
+
114
+ # Go to rest
115
+ r.goto_target(create_head_pose(), antennas=[left_rest, right_rest], duration=1.0)
116
+ time.sleep(1.5)
117
+ print(f"robot: beeps at {beep_times}", flush=True)
118
+ print(f"robot: collisions at {collision_times}", flush=True)
119
+
120
+ # Recording arrays
121
+ timestamps = []
122
+ left_present = []
123
+ right_present = []
124
+ left_target_log = []
125
+
126
+ # Start audio playback
127
+ r.media.start_playing()
128
+ r.media.push_audio_sample(np.zeros(160, dtype=np.float32))
129
+ time.sleep(0.05)
130
+
131
+ # Audio chunk tracking
132
+ chunk_size = int(robot_sr * DT) # 20ms audio chunks match motion DT
133
+ audio_idx = 0
134
+
135
+ print("robot: MARK_START", flush=True)
136
+ t0 = time.monotonic()
137
+
138
+ for i in range(n_steps):
139
+ # Push audio chunk
140
+ chunk_start = i * chunk_size
141
+ chunk_end = chunk_start + chunk_size
142
+ if chunk_end <= len(audio):
143
+ r.media.push_audio_sample(audio[chunk_start:chunk_end])
144
+
145
+ # Set antenna target
146
+ left = float(left_targets[i])
147
+ r.set_target(
148
+ head=np.eye(4),
149
+ body_yaw=0.0,
150
+ antennas=np.array([left, right_rest]),
151
+ )
152
+
153
+ # Read present position
154
+ pos = r.get_present_antenna_joint_positions()
155
+ elapsed = time.monotonic() - t0
156
+ timestamps.append(elapsed)
157
+ left_present.append(pos[0])
158
+ right_present.append(pos[1])
159
+ left_target_log.append(left)
160
+
161
+ # Real-time pacing
162
+ target_time = (i + 1) * DT
163
+ now = time.monotonic() - t0
164
+ if target_time > now:
165
+ time.sleep(target_time - now)
166
+
167
+ elapsed = time.monotonic() - t0
168
+ print(f"robot: finished {n_steps} steps in {elapsed:.3f}s", flush=True)
169
+
170
+ # Drain audio buffer and stop
171
+ time.sleep(0.5)
172
+ r.media.stop_playing()
173
+
174
+ # Save results
175
+ results = {
176
+ "beep_times": beep_times,
177
+ "collision_times": collision_times,
178
+ "beep_collision_offset": collision_times[0] - beep_times[0],
179
+ "left_collision_target": left_collision,
180
+ "right_rest": right_rest,
181
+ "hold_duration": hold_duration,
182
+ "timestamps": timestamps,
183
+ "left_present": left_present,
184
+ "right_present": right_present,
185
+ "left_target": left_target_log,
186
+ }
187
+ with open(results_path, "w") as f:
188
+ json.dump(results, f)
189
+ print(f"robot: saved {len(timestamps)} samples to {results_path}", flush=True)
190
+
191
+ r.goto_target(create_head_pose(), antennas=[left_rest, right_rest], duration=1.0)
192
+ time.sleep(1.5)
193
+ print("robot: done", flush=True)
194
+ os._exit(0)
195
+ """
196
+
197
+
198
+ def scp_to_robot(local_path: Path, remote_path: str, host: str) -> None:
199
+ target = f"{ROBOT_USER}@{host}:{remote_path}"
200
+ result = subprocess.run(
201
+ ["scp", "-o", "ConnectTimeout=5", str(local_path), target],
202
+ capture_output=True, text=True, timeout=15,
203
+ )
204
+ if result.returncode != 0:
205
+ raise RuntimeError(f"SCP failed: {result.stderr}")
206
+ print(f" Copied to {target}")
207
+
208
+
209
+ def scp_from_robot(remote_path: str, local_path: Path, host: str) -> None:
210
+ source = f"{ROBOT_USER}@{host}:{remote_path}"
211
+ result = subprocess.run(
212
+ ["scp", "-o", "ConnectTimeout=5", source, str(local_path)],
213
+ capture_output=True, text=True, timeout=15,
214
+ )
215
+ if result.returncode != 0:
216
+ raise RuntimeError(f"SCP failed: {result.stderr}")
217
+ print(f" Copied from {source}")
218
+
219
+
220
+ def start_robot(host: str) -> subprocess.Popen:
221
+ with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
222
+ f.write(ROBOT_SCRIPT)
223
+ local_script = Path(f.name)
224
+
225
+ remote_script = "/tmp/beep_collision_sync.py"
226
+ try:
227
+ scp_to_robot(local_script, remote_script, host)
228
+ finally:
229
+ local_script.unlink()
230
+
231
+ args_str = (
232
+ f"{ROBOT_PYTHON} {remote_script} "
233
+ f"'{json.dumps(BEEP_TIMES)}' "
234
+ f"'{json.dumps(COLLISION_TIMES)}' "
235
+ f"{RIGHT_REST} {LEFT_REST} {LEFT_COLLISION} {HOLD_DURATION} "
236
+ f"/dev/null " # wav_path unused, generated inline
237
+ f"{REMOTE_RESULTS} "
238
+ f"{BEEP_FREQ} {BEEP_DURATION} {BEEP_AMPLITUDE}"
239
+ )
240
+ proc = subprocess.Popen(
241
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{host}", args_str],
242
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
243
+ )
244
+ return proc
245
+
246
+
247
+ def plot_combined(
248
+ mic_audio: np.ndarray,
249
+ mic_sr: int,
250
+ mic_start: float,
251
+ mark_start: float,
252
+ robot_data: dict,
253
+ detected_beeps: list[float],
254
+ detected_collisions: list[float],
255
+ pairs: list[dict],
256
+ output_path: Path,
257
+ ) -> None:
258
+ import matplotlib
259
+ matplotlib.use("Agg")
260
+ import matplotlib.pyplot as plt
261
+
262
+ mic_t = np.arange(len(mic_audio)) / mic_sr
263
+ robot_offset = mark_start - mic_start
264
+
265
+ robot_ts = np.array(robot_data["timestamps"])
266
+ left_pos = np.array(robot_data["left_present"])
267
+ right_pos = np.array(robot_data["right_present"])
268
+ left_tgt = np.array(robot_data["left_target"])
269
+
270
+ fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(18, 10), sharex=True)
271
+
272
+ # --- Top: Mic waveform ---
273
+ ax1.plot(mic_t, mic_audio, "k-", linewidth=0.3, alpha=0.5)
274
+ ax1.set_ylabel("Mic amplitude")
275
+ ax1.set_title("Beep + Collision Sync Test — Laptop Mic Recording")
276
+ ax1.grid(True, alpha=0.3)
277
+
278
+ # Detected beeps (blue)
279
+ for i, bt in enumerate(detected_beeps):
280
+ label = "Detected beep" if i == 0 else None
281
+ ax1.axvline(bt, color="blue", linestyle="-", linewidth=1.2, alpha=0.7, label=label)
282
+
283
+ # Detected collisions (red)
284
+ for i, ct in enumerate(detected_collisions):
285
+ label = "Detected collision" if i == 0 else None
286
+ ax1.axvline(ct, color="red", linestyle="-", linewidth=1.2, alpha=0.7, label=label)
287
+
288
+ # Annotate pairs
289
+ for p in pairs:
290
+ mid = (p["beep_mic_t"] + p["collision_mic_t"]) / 2
291
+ ax1.annotate(
292
+ f'{p["interval_ms"]:.0f}ms',
293
+ xy=(mid, ax1.get_ylim()[1] * 0.8),
294
+ ha="center", fontsize=9, color="purple", fontweight="bold",
295
+ bbox=dict(boxstyle="round,pad=0.2", facecolor="lightyellow", alpha=0.8),
296
+ )
297
+
298
+ ax1.legend(loc="upper right", fontsize=9)
299
+
300
+ # --- Bottom: Robot trajectory ---
301
+ ax2.plot(robot_ts + robot_offset, left_pos, "b-", linewidth=1.5, label="Left antenna (present)")
302
+ ax2.plot(robot_ts + robot_offset, right_pos, "r-", linewidth=1.5, label="Right antenna (present)")
303
+ ax2.plot(robot_ts + robot_offset, left_tgt, "b--", linewidth=0.8, alpha=0.4, label="Left antenna (target)")
304
+
305
+ # Expected command times (robot clock → mic clock)
306
+ for i, bt in enumerate(BEEP_TIMES):
307
+ mic_bt = robot_offset + bt
308
+ label = "Beep cmd" if i == 0 else None
309
+ ax2.axvline(mic_bt, color="blue", linestyle="--", linewidth=1.0, alpha=0.5, label=label)
310
+ for i, ct in enumerate(COLLISION_TIMES):
311
+ mic_ct = robot_offset + ct
312
+ label = "Collision cmd" if i == 0 else None
313
+ ax2.axvline(mic_ct, color="red", linestyle="--", linewidth=1.0, alpha=0.5, label=label)
314
+
315
+ # Detected events on trajectory panel too
316
+ for bt in detected_beeps:
317
+ ax2.axvline(bt, color="blue", linestyle="-", linewidth=0.8, alpha=0.4)
318
+ for ct in detected_collisions:
319
+ ax2.axvline(ct, color="red", linestyle="-", linewidth=0.8, alpha=0.4)
320
+
321
+ ax2.set_xlabel("Time since mic start (s)")
322
+ ax2.set_ylabel("Position (rad)")
323
+ ax2.set_title("Robot Antenna Trajectory (aligned to mic clock)")
324
+ ax2.legend(loc="upper right", fontsize=9)
325
+ ax2.grid(True, alpha=0.3)
326
+
327
+ # Zoom to active region
328
+ active_start = robot_offset - 0.5
329
+ active_end = robot_offset + max(COLLISION_TIMES) + 2.0
330
+ ax1.set_xlim(active_start, active_end)
331
+
332
+ fig.tight_layout()
333
+ fig.savefig(str(output_path), dpi=150)
334
+ plt.close(fig)
335
+ print(f" Plot saved to {output_path}")
336
+
337
+
338
+ def main():
339
+ parser = argparse.ArgumentParser(description="Beep + collision sync test")
340
+ parser.add_argument("--host", default="reachy-mini.local")
341
+ args = parser.parse_args()
342
+
343
+ print(f"\n{'='*60}")
344
+ print("Beep + Collision Sync Test")
345
+ print(f"{'='*60}")
346
+ print(f" Beep times: {BEEP_TIMES}")
347
+ print(f" Collision times: {COLLISION_TIMES}")
348
+ print(f" Expected interval: {BEEP_COLLISION_OFFSET:.1f}s (beep → collision)")
349
+ gaps = [BEEP_TIMES[i+1] - BEEP_TIMES[i] for i in range(len(BEEP_TIMES)-1)]
350
+ print(f" Gaps between pairs: {[f'{g:.1f}s' for g in gaps]}\n")
351
+
352
+ # Step 1: Stop running apps
353
+ print("[1/5] Stopping any running app...")
354
+ subprocess.run(
355
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{args.host}",
356
+ "curl -sf -X POST http://127.0.0.1:8000/api/apps/stop-current-app >/dev/null 2>&1 || true"],
357
+ capture_output=True, timeout=10,
358
+ )
359
+ time.sleep(1)
360
+
361
+ # Step 2: Start mic recording
362
+ print(f"[2/5] Starting mic recording ({MIC_DURATION}s)...")
363
+ mic_start = time.monotonic()
364
+ mic_data = sd.rec(
365
+ int(MIC_DURATION * LAPTOP_SR),
366
+ samplerate=LAPTOP_SR, channels=1, dtype="float32",
367
+ )
368
+
369
+ # Step 3: Start robot
370
+ time.sleep(0.3)
371
+ print("[3/5] Starting robot (beeps + collisions)...")
372
+ proc = start_robot(args.host)
373
+
374
+ # Read stdout, capture MARK_START
375
+ mark_start = None
376
+ print("\n--- Robot output ---")
377
+ for line in iter(proc.stdout.readline, ""):
378
+ line = line.rstrip()
379
+ if not line:
380
+ continue
381
+ laptop_time = time.monotonic()
382
+ print(f" {line}")
383
+ if "MARK_START" in line:
384
+ mark_start = laptop_time
385
+ proc.wait()
386
+ print("--- End robot output ---")
387
+
388
+ sd.wait()
389
+ captured = mic_data.flatten()
390
+ print(f"\n Mic recording done")
391
+
392
+ if mark_start is None:
393
+ print("\nFAILED: Never received MARK_START")
394
+ return 1
395
+
396
+ robot_offset = mark_start - mic_start
397
+ print(f" MARK_START at mic_t={robot_offset:.3f}s")
398
+
399
+ # Save mic audio
400
+ mic_path = Path("tests/beep_collision_mic.wav")
401
+ sf.write(str(mic_path), captured, LAPTOP_SR)
402
+ print(f" Saved mic to {mic_path}")
403
+
404
+ # Step 4: Fetch robot data
405
+ print("\n[4/5] Fetching robot data...")
406
+ local_results = Path("tests/beep_collision_positions.json")
407
+ scp_from_robot(REMOTE_RESULTS, local_results, args.host)
408
+ with open(local_results) as f:
409
+ robot_data = json.load(f)
410
+
411
+ # Step 5: Analyze
412
+ print("\n[5/5] Analyzing...")
413
+
414
+ # Detect beeps (tonal, bandpass around 2kHz)
415
+ detected_beeps = detect_beep_onsets(
416
+ captured, LAPTOP_SR, freq=BEEP_FREQ, bandwidth=150.0, threshold_db=-12.0,
417
+ min_separation=1.0, # beeps are ≥1.3s apart
418
+ )
419
+ print(f" Detected {len(detected_beeps)} beeps at: "
420
+ f"{[f'{t:.3f}' for t in detected_beeps]}")
421
+
422
+ # Detect collisions (impulsive, highpass >2kHz)
423
+ detected_collisions = detect_transient_onsets(
424
+ captured, LAPTOP_SR, highpass_freq=3000.0,
425
+ )
426
+ print(f" Detected {len(detected_collisions)} collisions at: "
427
+ f"{[f'{t:.3f}' for t in detected_collisions]}")
428
+
429
+ # Match beep-collision pairs
430
+ # For each detected beep, find the nearest collision ~1s later
431
+ print(f"\n{'='*60}")
432
+ print("Beep → Collision Interval Analysis")
433
+ print(f" (Expected interval: {BEEP_COLLISION_OFFSET*1000:.0f}ms)")
434
+ print(f"{'='*60}")
435
+
436
+ pairs = []
437
+ for i, bt in enumerate(detected_beeps):
438
+ # Look for a collision between 0.5s and 2.0s after the beep
439
+ candidates = [ct for ct in detected_collisions if 0.5 < (ct - bt) < 2.0]
440
+ if not candidates:
441
+ print(f" Beep {i+1} at {bt:.3f}s: NO COLLISION FOUND in [+0.5, +2.0]s window")
442
+ continue
443
+
444
+ nearest = min(candidates, key=lambda ct: abs((ct - bt) - BEEP_COLLISION_OFFSET))
445
+ interval_ms = (nearest - bt) * 1000
446
+ error_ms = interval_ms - BEEP_COLLISION_OFFSET * 1000
447
+ pairs.append({
448
+ "beep_mic_t": bt,
449
+ "collision_mic_t": nearest,
450
+ "interval_ms": interval_ms,
451
+ "error_ms": error_ms,
452
+ })
453
+ print(f" Pair {len(pairs)}: beep {bt:.3f}s → collision {nearest:.3f}s = "
454
+ f"{interval_ms:.0f}ms (error {error_ms:+.0f}ms)")
455
+
456
+ if pairs:
457
+ errors = [p["error_ms"] for p in pairs]
458
+ intervals = [p["interval_ms"] for p in pairs]
459
+ print(f"\n Pairs matched: {len(pairs)}/{len(BEEP_TIMES)}")
460
+ print(f" Mean interval: {np.mean(intervals):.0f}ms (expected {BEEP_COLLISION_OFFSET*1000:.0f}ms)")
461
+ print(f" Mean error: {np.mean(errors):+.0f}ms")
462
+ print(f" Std error: {np.std(errors):.0f}ms")
463
+ print(f" Min/Max error: {min(errors):+.0f}ms / {max(errors):+.0f}ms")
464
+ else:
465
+ print(f"\n No pairs matched!")
466
+
467
+ # Generate plot
468
+ plot_path = Path("tests/beep_collision_sync_plot.png")
469
+ plot_combined(
470
+ captured, LAPTOP_SR, mic_start, mark_start,
471
+ robot_data, detected_beeps, detected_collisions, pairs, plot_path,
472
+ )
473
+
474
+ success = len(pairs) >= len(BEEP_TIMES) - 1
475
+ print(f"\n{'='*60}")
476
+ if success:
477
+ print("RESULT: PASS — Beep-collision pairs detected and measured")
478
+ else:
479
+ print("RESULT: FAIL — Could not reliably detect pairs")
480
+ print(f"{'='*60}\n")
481
+
482
+ return 0 if success else 1
483
+
484
+
485
+ if __name__ == "__main__":
486
+ sys.exit(main())
tests/test_collision_mic.py ADDED
@@ -0,0 +1,399 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Measure laptop-to-collision delay using antenna collisions + laptop mic.
3
+
4
+ Combines the collision test (robot-side position recording) with laptop mic
5
+ recording to measure the full end-to-end delay from the laptop issuing a
6
+ command to the collision sound being captured.
7
+
8
+ Delay chain measured:
9
+ laptop SSH command → robot receives → daemon → motor → physical collision
10
+ → sound through air → laptop mic → detected
11
+
12
+ Clock sync: Robot prints "MARK_START" to stdout. The laptop notes the time
13
+ it receives this line. The robot's t=0 maps to that laptop timestamp (with
14
+ ~few ms SSH stdout latency).
15
+
16
+ Usage:
17
+ python tests/test_collision_mic.py [--host reachy-mini.local]
18
+ """
19
+ from __future__ import annotations
20
+
21
+ import argparse
22
+ import json
23
+ import subprocess
24
+ import sys
25
+ import tempfile
26
+ import time
27
+ from pathlib import Path
28
+
29
+ import numpy as np
30
+ import sounddevice as sd
31
+ import soundfile as sf
32
+
33
+ sys.path.insert(0, str(Path(__file__).parent))
34
+ from audio_analysis import detect_transient_onsets
35
+
36
+ # Collision parameters (same as test_antenna_collision.py)
37
+ RIGHT_REST = -0.68
38
+ LEFT_REST = 0.0
39
+ LEFT_COLLISION = 0.70
40
+ HOLD_DURATION = 0.2
41
+ COLLISION_TIMES = [1.0, 2.3, 4.0, 6.3, 9.4]
42
+
43
+ ROBOT_USER = "pollen"
44
+ ROBOT_PYTHON = "/venvs/apps_venv/bin/python"
45
+ REMOTE_RESULTS = "/tmp/collision_positions.json"
46
+ LAPTOP_SR = 48000
47
+ MIC_DURATION = 30.0 # seconds — covers ~8s SSH startup + 10.5s sequence + buffer
48
+
49
+ # Robot script: same as test_antenna_collision.py
50
+ ROBOT_COLLISION_SCRIPT = """\
51
+ import numpy as np
52
+ import os, sys, time, json
53
+
54
+ collision_times = json.loads(sys.argv[1])
55
+ right_rest = float(sys.argv[2])
56
+ left_rest = float(sys.argv[3])
57
+ left_collision = float(sys.argv[4])
58
+ hold_duration = float(sys.argv[5])
59
+ results_path = sys.argv[6]
60
+
61
+ print(f"robot: connecting to ReachyMini", flush=True)
62
+ from reachy_mini import ReachyMini
63
+ from reachy_mini.utils import create_head_pose
64
+ r = ReachyMini(media_backend="no_media")
65
+
66
+ r.goto_target(create_head_pose(), antennas=[left_rest, right_rest], duration=1.0)
67
+ time.sleep(1.5)
68
+
69
+ print(f"robot: rest position — left={left_rest}, right={right_rest}", flush=True)
70
+ print(f"robot: will collide at times: {collision_times}", flush=True)
71
+
72
+ DT = 0.02
73
+ total_duration = max(collision_times) + hold_duration + 1.0
74
+ n_steps = int(total_duration / DT)
75
+
76
+ left_targets = np.full(n_steps, left_rest, dtype=np.float64)
77
+ for ct in collision_times:
78
+ start_step = int(ct / DT)
79
+ end_step = int((ct + hold_duration) / DT)
80
+ end_step = min(end_step, n_steps)
81
+ left_targets[start_step:end_step] = left_collision
82
+
83
+ timestamps = []
84
+ left_present = []
85
+ right_present = []
86
+ left_target_log = []
87
+
88
+ print("robot: MARK_START", flush=True)
89
+ t0 = time.monotonic()
90
+
91
+ for i in range(n_steps):
92
+ left = float(left_targets[i])
93
+ r.set_target(
94
+ head=np.eye(4),
95
+ body_yaw=0.0,
96
+ antennas=np.array([left, right_rest]),
97
+ )
98
+
99
+ pos = r.get_present_antenna_joint_positions()
100
+ elapsed = time.monotonic() - t0
101
+ timestamps.append(elapsed)
102
+ left_present.append(pos[0])
103
+ right_present.append(pos[1])
104
+ left_target_log.append(left)
105
+
106
+ target_time = (i + 1) * DT
107
+ now = time.monotonic() - t0
108
+ if target_time > now:
109
+ time.sleep(target_time - now)
110
+
111
+ elapsed = time.monotonic() - t0
112
+ print(f"robot: finished {n_steps} steps in {elapsed:.3f}s", flush=True)
113
+
114
+ results = {
115
+ "collision_times": collision_times,
116
+ "left_collision_target": left_collision,
117
+ "right_rest": right_rest,
118
+ "hold_duration": hold_duration,
119
+ "timestamps": timestamps,
120
+ "left_present": left_present,
121
+ "right_present": right_present,
122
+ "left_target": left_target_log,
123
+ }
124
+ with open(results_path, "w") as f:
125
+ json.dump(results, f)
126
+ print(f"robot: saved {len(timestamps)} samples to {results_path}", flush=True)
127
+
128
+ r.goto_target(create_head_pose(), antennas=[left_rest, right_rest], duration=1.0)
129
+ time.sleep(1.5)
130
+ print("robot: done", flush=True)
131
+ os._exit(0)
132
+ """
133
+
134
+
135
+ def scp_to_robot(local_path: Path, remote_path: str, host: str) -> None:
136
+ target = f"{ROBOT_USER}@{host}:{remote_path}"
137
+ result = subprocess.run(
138
+ ["scp", "-o", "ConnectTimeout=5", str(local_path), target],
139
+ capture_output=True, text=True, timeout=15,
140
+ )
141
+ if result.returncode != 0:
142
+ raise RuntimeError(f"SCP failed: {result.stderr}")
143
+ print(f" Copied to {target}")
144
+
145
+
146
+ def scp_from_robot(remote_path: str, local_path: Path, host: str) -> None:
147
+ source = f"{ROBOT_USER}@{host}:{remote_path}"
148
+ result = subprocess.run(
149
+ ["scp", "-o", "ConnectTimeout=5", source, str(local_path)],
150
+ capture_output=True, text=True, timeout=15,
151
+ )
152
+ if result.returncode != 0:
153
+ raise RuntimeError(f"SCP failed: {result.stderr}")
154
+ print(f" Copied from {source}")
155
+
156
+
157
+ def start_robot(host: str) -> subprocess.Popen:
158
+ with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
159
+ f.write(ROBOT_COLLISION_SCRIPT)
160
+ local_script = Path(f.name)
161
+
162
+ remote_script = "/tmp/collision_mic_test.py"
163
+ try:
164
+ scp_to_robot(local_script, remote_script, host)
165
+ finally:
166
+ local_script.unlink()
167
+
168
+ args_str = (
169
+ f"{ROBOT_PYTHON} {remote_script} "
170
+ f"'{json.dumps(COLLISION_TIMES)}' "
171
+ f"{RIGHT_REST} {LEFT_REST} {LEFT_COLLISION} {HOLD_DURATION} "
172
+ f"{REMOTE_RESULTS}"
173
+ )
174
+ proc = subprocess.Popen(
175
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{host}", args_str],
176
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
177
+ )
178
+ return proc
179
+
180
+
181
+ def plot_combined(
182
+ mic_audio: np.ndarray,
183
+ mic_sr: int,
184
+ mic_start: float,
185
+ mark_start: float,
186
+ robot_data: dict,
187
+ detected_mic_times: list[float],
188
+ output_path: Path,
189
+ ) -> None:
190
+ """Plot mic waveform + robot trajectory on aligned time axes."""
191
+ import matplotlib
192
+ matplotlib.use("Agg")
193
+ import matplotlib.pyplot as plt
194
+
195
+ # Time axis for mic (in "mic seconds since mic_start")
196
+ mic_t = np.arange(len(mic_audio)) / mic_sr
197
+
198
+ # Offset to align robot clock with mic clock:
199
+ # robot t=0 happened at laptop time mark_start
200
+ # mic t=0 happened at laptop time mic_start
201
+ # So robot t=X is at mic_t = (mark_start - mic_start) + X
202
+ robot_offset = mark_start - mic_start
203
+
204
+ robot_ts = np.array(robot_data["timestamps"])
205
+ left_pos = np.array(robot_data["left_present"])
206
+ right_pos = np.array(robot_data["right_present"])
207
+ left_tgt = np.array(robot_data["left_target"])
208
+
209
+ fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(16, 10), sharex=True)
210
+
211
+ # --- Top: Mic waveform ---
212
+ ax1.plot(mic_t, mic_audio, "k-", linewidth=0.3, alpha=0.6)
213
+ ax1.set_ylabel("Mic amplitude")
214
+ ax1.set_title("Laptop Mic Recording + Detected Collision Transients")
215
+ ax1.grid(True, alpha=0.3)
216
+
217
+ # Expected collision times in mic coordinates
218
+ for i, ct in enumerate(COLLISION_TIMES):
219
+ mic_expected = robot_offset + ct
220
+ label = "Expected (cmd time)" if i == 0 else None
221
+ ax1.axvline(mic_expected, color="green", linestyle="--", linewidth=1.2, alpha=0.7, label=label)
222
+
223
+ # Detected transients
224
+ for i, dt in enumerate(detected_mic_times):
225
+ label = "Detected transient" if i == 0 else None
226
+ ax1.axvline(dt, color="red", linestyle="-", linewidth=1.2, alpha=0.7, label=label)
227
+
228
+ ax1.legend(loc="upper right", fontsize=9)
229
+
230
+ # --- Bottom: Robot trajectory ---
231
+ # Plot in mic time coordinates
232
+ ax2.plot(robot_ts + robot_offset, left_pos, "b-", linewidth=1.5, label="Left antenna (present)")
233
+ ax2.plot(robot_ts + robot_offset, right_pos, "r-", linewidth=1.5, label="Right antenna (present)")
234
+ ax2.plot(robot_ts + robot_offset, left_tgt, "b--", linewidth=0.8, alpha=0.4, label="Left antenna (target)")
235
+
236
+ for i, ct in enumerate(COLLISION_TIMES):
237
+ mic_expected = robot_offset + ct
238
+ label = "Expected (cmd time)" if i == 0 else None
239
+ ax2.axvline(mic_expected, color="green", linestyle="--", linewidth=1.2, alpha=0.7, label=label)
240
+
241
+ for i, dt in enumerate(detected_mic_times):
242
+ label = "Detected transient" if i == 0 else None
243
+ ax2.axvline(dt, color="red", linestyle="-", linewidth=1.2, alpha=0.7, label=label)
244
+
245
+ ax2.set_xlabel("Time since mic start (s)")
246
+ ax2.set_ylabel("Position (rad)")
247
+ ax2.set_title("Robot Antenna Trajectory (aligned to mic clock)")
248
+ ax2.legend(loc="upper right", fontsize=9)
249
+ ax2.grid(True, alpha=0.3)
250
+
251
+ # Zoom to active region
252
+ active_start = robot_offset - 0.5
253
+ active_end = robot_offset + max(COLLISION_TIMES) + 2.0
254
+ ax1.set_xlim(active_start, active_end)
255
+
256
+ fig.tight_layout()
257
+ fig.savefig(str(output_path), dpi=150)
258
+ plt.close(fig)
259
+ print(f" Plot saved to {output_path}")
260
+
261
+
262
+ def main():
263
+ parser = argparse.ArgumentParser(description="Collision + mic delay measurement")
264
+ parser.add_argument("--host", default="reachy-mini.local")
265
+ args = parser.parse_args()
266
+
267
+ print(f"\n{'='*60}")
268
+ print("Collision + Mic Delay Measurement")
269
+ print(f"{'='*60}")
270
+ print(f" Collision times: {COLLISION_TIMES}")
271
+ print(f" Mic duration: {MIC_DURATION}s at {LAPTOP_SR}Hz\n")
272
+
273
+ # Step 1: Stop running apps
274
+ print("[1/5] Stopping any running app...")
275
+ subprocess.run(
276
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{args.host}",
277
+ "curl -sf -X POST http://127.0.0.1:8000/api/apps/stop-current-app >/dev/null 2>&1 || true"],
278
+ capture_output=True, timeout=10,
279
+ )
280
+ time.sleep(1)
281
+
282
+ # Step 2: Start mic recording
283
+ print(f"[2/5] Starting mic recording ({MIC_DURATION}s)...")
284
+ mic_start = time.monotonic()
285
+ mic_data = sd.rec(
286
+ int(MIC_DURATION * LAPTOP_SR),
287
+ samplerate=LAPTOP_SR, channels=1, dtype="float32",
288
+ )
289
+
290
+ # Step 3: Start robot collision sequence
291
+ time.sleep(0.3) # let mic stabilize
292
+ print("[3/5] Starting collision sequence on robot...")
293
+ proc = start_robot(args.host)
294
+
295
+ # Read stdout, capture MARK_START timestamp
296
+ mark_start = None
297
+ print("\n--- Robot output ---")
298
+ for line in iter(proc.stdout.readline, ""):
299
+ line = line.rstrip()
300
+ if not line:
301
+ continue
302
+ laptop_time = time.monotonic()
303
+ print(f" {line}")
304
+ if "MARK_START" in line:
305
+ mark_start = laptop_time
306
+ proc.wait()
307
+ print("--- End robot output ---")
308
+
309
+ # Wait for mic recording to finish
310
+ sd.wait()
311
+ captured = mic_data.flatten()
312
+ mic_end = time.monotonic()
313
+ print(f"\n Mic recording done ({mic_end - mic_start:.1f}s)")
314
+
315
+ if mark_start is None:
316
+ print("\nFAILED: Never received MARK_START from robot")
317
+ return 1
318
+
319
+ robot_offset = mark_start - mic_start
320
+ print(f" MARK_START received at mic_t={robot_offset:.3f}s")
321
+
322
+ # Save mic audio
323
+ mic_path = Path("tests/collision_mic.wav")
324
+ sf.write(str(mic_path), captured, LAPTOP_SR)
325
+ print(f" Saved mic audio to {mic_path}")
326
+
327
+ # Step 4: Fetch robot position data
328
+ print("\n[4/5] Fetching position data from robot...")
329
+ local_results = Path("tests/collision_positions.json")
330
+ scp_from_robot(REMOTE_RESULTS, local_results, args.host)
331
+
332
+ with open(local_results) as f:
333
+ robot_data = json.load(f)
334
+
335
+ # Step 5: Analyze
336
+ print("\n[5/5] Analyzing...")
337
+
338
+ # Detect transient sounds in mic recording
339
+ detected_mic = detect_transient_onsets(captured, LAPTOP_SR, highpass_freq=2000.0)
340
+ print(f" Detected {len(detected_mic)} transients in mic at: "
341
+ f"{[f'{t:.3f}' for t in detected_mic]}")
342
+
343
+ # Expected collision times in mic coordinates
344
+ expected_mic = [robot_offset + ct for ct in COLLISION_TIMES]
345
+ print(f" Expected collision sounds at mic_t: "
346
+ f"{[f'{t:.3f}' for t in expected_mic]}")
347
+
348
+ # Match detected transients to expected collision times
349
+ print(f"\n{'='*60}")
350
+ print("Delay Analysis: Command → Audible Collision")
351
+ print(f"{'='*60}")
352
+
353
+ matched = 0
354
+ delays = []
355
+ for i, (et_mic, ct_robot) in enumerate(zip(expected_mic, COLLISION_TIMES)):
356
+ if not detected_mic:
357
+ print(f" Collision {i+1} (cmd t={ct_robot:.1f}s): NO TRANSIENT DETECTED")
358
+ continue
359
+
360
+ nearest = min(detected_mic, key=lambda d: abs(d - et_mic))
361
+ delay_ms = (nearest - et_mic) * 1000
362
+ ok = abs(nearest - et_mic) < 0.5 # 500ms tolerance
363
+ if ok:
364
+ matched += 1
365
+ delays.append(delay_ms)
366
+ status = "OK" if ok else "MISS"
367
+ print(f" Collision {i+1} (cmd t={ct_robot:.1f}s): "
368
+ f"expected mic_t={et_mic:.3f}s, detected={nearest:.3f}s, "
369
+ f"delay={delay_ms:+.0f}ms [{status}]")
370
+
371
+ if delays:
372
+ print(f"\n Matched: {matched}/{len(COLLISION_TIMES)}")
373
+ print(f" Mean delay: {np.mean(delays):+.0f}ms")
374
+ print(f" Std delay: {np.std(delays):.0f}ms")
375
+ print(f" Min/Max: {min(delays):+.0f}ms / {max(delays):+.0f}ms")
376
+ print(f"\n (Positive = sound arrived AFTER command)")
377
+ else:
378
+ print(f"\n No delays measured — no transients matched")
379
+
380
+ # Generate combined plot
381
+ plot_path = Path("tests/collision_mic_plot.png")
382
+ plot_combined(
383
+ captured, LAPTOP_SR, mic_start, mark_start,
384
+ robot_data, detected_mic, plot_path,
385
+ )
386
+
387
+ success = matched >= len(COLLISION_TIMES) - 1
388
+ print(f"\n{'='*60}")
389
+ if success:
390
+ print("RESULT: PASS — Collision sounds detected and matched")
391
+ else:
392
+ print("RESULT: FAIL — Could not reliably detect collision sounds")
393
+ print(f"{'='*60}\n")
394
+
395
+ return 0 if success else 1
396
+
397
+
398
+ if __name__ == "__main__":
399
+ sys.exit(main())
tests/test_hardware.py CHANGED
@@ -566,6 +566,18 @@ def _observe_playback(base_url, hw_reachy, duration):
566
  return observed_times, observed_frames
567
 
568
 
 
 
 
 
 
 
 
 
 
 
 
 
569
  class TestMotionAccuracy:
570
  """Play back synthetic reference recordings and compare observed poses.
571
 
@@ -768,6 +780,149 @@ class TestMotionAccuracy:
768
  httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
769
 
770
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
771
  class TestMultiDuration:
772
  """Test recording and playback across different durations.
773
 
@@ -1478,6 +1633,384 @@ class TestPlaybackWithCorruptFile:
1478
  hw_marionette._refresh_recordings()
1479
 
1480
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1481
  class TestHardwareAudio:
1482
  """Audio recording/playback tests — run last.
1483
 
@@ -1582,3 +2115,105 @@ class TestHardwareAudio:
1582
 
1583
  state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1584
  assert state["mode"] == "idle"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
566
  return observed_times, observed_frames
567
 
568
 
569
+ def _write_silent_wav(path: Path, duration: float, sample_rate: int = 48000) -> None:
570
+ """Write a silent WAV file of the given duration."""
571
+ import struct
572
+
573
+ n_frames = int(duration * sample_rate)
574
+ with wave.open(str(path), "wb") as wf:
575
+ wf.setnchannels(1)
576
+ wf.setsampwidth(2)
577
+ wf.setframerate(sample_rate)
578
+ wf.writeframes(struct.pack(f"<{n_frames}h", *([0] * n_frames)))
579
+
580
+
581
  class TestMotionAccuracy:
582
  """Play back synthetic reference recordings and compare observed poses.
583
 
 
780
  httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
781
 
782
 
783
+ class TestAudioMotionSync:
784
+ """Verify that audio and motion play back together correctly.
785
+
786
+ Creates a synthetic move with a matching WAV file, plays it back,
787
+ measures wall-clock duration and observed joint positions, and
788
+ verifies both timing and trajectory accuracy.
789
+ """
790
+
791
+ def test_audio_motion_sync_playback(
792
+ self, base_url: str, hw_marionette, hw_reachy,
793
+ ):
794
+ """Inject synthetic move + WAV, play back, verify timing and accuracy."""
795
+ import httpx
796
+ import numpy as np
797
+ from pose_utils import compare_trajectories
798
+
799
+ duration = 3.0
800
+ _ensure_idle(base_url)
801
+
802
+ # Step 1: Create synthetic move — gentle yaw oscillation
803
+ move_id = _create_synthetic_move(
804
+ hw_marionette,
805
+ label="synth-audio-sync",
806
+ duration=duration,
807
+ trajectory_fn=lambda t: (0.0, 0.0, 0.3 * np.sin(2 * np.pi * 0.5 * t)),
808
+ )
809
+
810
+ # Step 2: Write matching silent WAV alongside the JSON
811
+ wav_path = hw_marionette._dataset_dir / f"{move_id}.wav"
812
+ _write_silent_wav(wav_path, duration=duration, sample_rate=48000)
813
+
814
+ # Step 3: Refresh and verify has_audio
815
+ hw_marionette._refresh_recordings()
816
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
817
+ move = next((m for m in state["moves"] if m["id"] == move_id), None)
818
+ assert move is not None, f"Move {move_id} not in state after refresh"
819
+ assert move["has_audio"] is True, "Move should have audio (WAV exists)"
820
+
821
+ # Step 4: Load reference for comparison
822
+ ref_data = json.loads(
823
+ (hw_marionette._dataset_dir / f"{move_id}.json").read_text()
824
+ )
825
+ ref_times = ref_data["time"]
826
+ ref_frames = ref_data["set_target_data"]
827
+
828
+ # Step 5: Play back and measure wall-clock + poses
829
+ t0 = time.time()
830
+ resp = httpx.post(
831
+ f"{base_url}/api/play",
832
+ json={"move_id": move_id},
833
+ timeout=5,
834
+ )
835
+ assert resp.status_code == 200
836
+
837
+ observed_times, observed_frames = _observe_playback(
838
+ base_url, hw_reachy, duration,
839
+ )
840
+ _wait_for_mode(base_url, "idle", timeout=duration + 15)
841
+ wall_clock = time.time() - t0
842
+
843
+ # Step 6: Timing assertions — audio shouldn't cut short or hang
844
+ assert wall_clock > duration * 0.8, (
845
+ f"Playback too fast: {wall_clock:.2f}s — audio likely cut short "
846
+ f"(expected ≥{duration * 0.8:.1f}s)"
847
+ )
848
+ assert wall_clock < duration + 10, (
849
+ f"Playback too slow: {wall_clock:.2f}s — possibly stuck "
850
+ f"(expected <{duration + 10:.1f}s)"
851
+ )
852
+
853
+ # Step 7: Verify enough frames observed (robot actually moved)
854
+ assert len(observed_frames) > 10, (
855
+ f"Too few observed frames: {len(observed_frames)} — "
856
+ f"robot may not have moved"
857
+ )
858
+
859
+ # Step 8: Compare trajectories
860
+ metrics = compare_trajectories(
861
+ ref_times, ref_frames, observed_times, observed_frames,
862
+ )
863
+ print(f"\nAudio-motion sync ({len(observed_frames)} frames, "
864
+ f"wall={wall_clock:.2f}s):")
865
+ print(metrics.summary())
866
+
867
+ assert metrics.magic_mean < 50, (
868
+ f"Mean magic distance too high: {metrics.magic_mean:.1f}\n"
869
+ f"{metrics.summary()}"
870
+ )
871
+
872
+ # Cleanup
873
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
874
+
875
+ def test_audio_does_not_truncate_motion(
876
+ self, base_url: str, hw_marionette,
877
+ ):
878
+ """Verify that audio playback doesn't end before motion completes.
879
+
880
+ Creates a 5s move + 5s WAV, plays back, measures wall-clock to
881
+ ensure the full duration plays out.
882
+ """
883
+ import httpx
884
+ import numpy as np
885
+
886
+ duration = 5.0
887
+ _ensure_idle(base_url)
888
+
889
+ move_id = _create_synthetic_move(
890
+ hw_marionette,
891
+ label="synth-audio-notrim",
892
+ duration=duration,
893
+ trajectory_fn=lambda t: (0.0, 0.0, 0.2 * np.sin(2 * np.pi * 0.3 * t)),
894
+ )
895
+
896
+ wav_path = hw_marionette._dataset_dir / f"{move_id}.wav"
897
+ _write_silent_wav(wav_path, duration=duration, sample_rate=48000)
898
+ hw_marionette._refresh_recordings()
899
+
900
+ # Play and measure
901
+ _ensure_idle(base_url)
902
+ t0 = time.time()
903
+ resp = httpx.post(
904
+ f"{base_url}/api/play",
905
+ json={"move_id": move_id},
906
+ timeout=5,
907
+ )
908
+ assert resp.status_code == 200
909
+ _wait_for_mode(base_url, "idle", timeout=duration + 20)
910
+ wall_clock = time.time() - t0
911
+
912
+ print(f"\nAudio-motion no-trim: wall={wall_clock:.2f}s for {duration}s move+audio")
913
+
914
+ # Playback should last at least 80% of the duration
915
+ assert wall_clock > duration * 0.8, (
916
+ f"Playback cut short: {wall_clock:.2f}s (expected ≥{duration * 0.8:.1f}s)"
917
+ )
918
+ assert wall_clock < duration + 15, (
919
+ f"Playback hung: {wall_clock:.2f}s (expected <{duration + 15:.1f}s)"
920
+ )
921
+
922
+ # Cleanup
923
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
924
+
925
+
926
  class TestMultiDuration:
927
  """Test recording and playback across different durations.
928
 
 
1633
  hw_marionette._refresh_recordings()
1634
 
1635
 
1636
+ class TestHardwarePlaybackWithAudio:
1637
+ """Playback with uploaded audio — validates Fix 1 (audio lifecycle).
1638
+
1639
+ These tests upload a WAV file, record motion with it, then play back
1640
+ to verify that both audio and motion complete fully (not cut short).
1641
+ """
1642
+
1643
+ def test_playback_with_uploaded_audio_completes(self, base_url: str, hw_marionette):
1644
+ """Upload WAV, record with it, play back — verify full completion."""
1645
+ import httpx
1646
+ import io
1647
+ import struct
1648
+ import wave as wave_mod
1649
+
1650
+ _ensure_idle(base_url)
1651
+
1652
+ # Create a 2s WAV file in memory
1653
+ sr, dur = 44100, 2.0
1654
+ n_frames = int(dur * sr)
1655
+ buf = io.BytesIO()
1656
+ with wave_mod.open(buf, "wb") as wf:
1657
+ wf.setnchannels(1)
1658
+ wf.setsampwidth(2)
1659
+ wf.setframerate(sr)
1660
+ wf.writeframes(struct.pack(f"<{n_frames}h", *([0] * n_frames)))
1661
+ wav_bytes = buf.getvalue()
1662
+
1663
+ # Upload the WAV
1664
+ upload_resp = httpx.post(
1665
+ f"{base_url}/api/upload-audio",
1666
+ files={"file": ("test-play.wav", wav_bytes, "audio/wav")},
1667
+ timeout=10,
1668
+ )
1669
+ assert upload_resp.status_code == 200
1670
+ upload_id = upload_resp.json()["upload_id"]
1671
+
1672
+ # Record motion with uploaded audio
1673
+ _ensure_idle(base_url)
1674
+ resp = httpx.post(
1675
+ f"{base_url}/api/record",
1676
+ json={
1677
+ "duration": 2.0,
1678
+ "record_audio": False,
1679
+ "record_motion": True,
1680
+ "uploaded_audio_id": upload_id,
1681
+ "label": "hw-upload-play",
1682
+ },
1683
+ timeout=5,
1684
+ )
1685
+ assert resp.status_code == 200
1686
+ move_id = resp.json()["move_id"]
1687
+ _wait_for_mode(base_url, "idle", timeout=COUNTDOWN_SECONDS + 2 + 15)
1688
+
1689
+ # Verify recording has audio
1690
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1691
+ move = next((m for m in state["moves"] if m["id"] == move_id), None)
1692
+ assert move is not None
1693
+ assert move["has_audio"] is True
1694
+
1695
+ # Play it back and measure duration
1696
+ _ensure_idle(base_url)
1697
+ t0 = time.time()
1698
+ resp = httpx.post(
1699
+ f"{base_url}/api/play",
1700
+ json={"move_id": move_id},
1701
+ timeout=5,
1702
+ )
1703
+ assert resp.status_code == 200
1704
+ _wait_for_mode(base_url, "idle", timeout=move["duration"] + 20)
1705
+ playback_time = time.time() - t0
1706
+
1707
+ # Playback should last approximately the move duration (not 1s!)
1708
+ assert playback_time > 1.5, (
1709
+ f"Playback too short: {playback_time:.2f}s — audio/motion likely cut short"
1710
+ )
1711
+ print(f"\nPlayback with uploaded audio: {playback_time:.2f}s")
1712
+
1713
+ # Cleanup
1714
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
1715
+
1716
+ def test_multiple_playback_cycles_with_audio(self, base_url: str, hw_marionette):
1717
+ """Play the same move 3 times — verify no resource leaks."""
1718
+ import httpx
1719
+
1720
+ _ensure_idle(base_url)
1721
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1722
+ move = next((m for m in state["moves"] if m["has_audio"]), None)
1723
+ if move is None:
1724
+ pytest.skip("No audio move available for cycle test")
1725
+
1726
+ for i in range(3):
1727
+ _ensure_idle(base_url)
1728
+ t0 = time.time()
1729
+ resp = httpx.post(
1730
+ f"{base_url}/api/play",
1731
+ json={"move_id": move["id"]},
1732
+ timeout=5,
1733
+ )
1734
+ assert resp.status_code == 200, f"Cycle {i}: play failed"
1735
+ _wait_for_mode(base_url, "idle", timeout=move["duration"] + 20)
1736
+ elapsed = time.time() - t0
1737
+ print(f" Cycle {i+1}/3: {elapsed:.2f}s")
1738
+
1739
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1740
+ assert state["mode"] == "idle", f"Cycle {i}: not idle after playback"
1741
+
1742
+
1743
+ class TestHardwareAudioOnlyRecording:
1744
+ """Audio-only recording (mic, no motion) on real hardware."""
1745
+
1746
+ def test_audio_only_recording_and_playback(self, base_url: str, hw_marionette):
1747
+ """Record audio-only, verify WAV created, play it back."""
1748
+ import httpx
1749
+
1750
+ _ensure_idle(base_url)
1751
+ resp = httpx.post(
1752
+ f"{base_url}/api/record",
1753
+ json={
1754
+ "duration": 2.0,
1755
+ "record_audio": True,
1756
+ "record_motion": False,
1757
+ "label": "hw-audio-only",
1758
+ },
1759
+ timeout=5,
1760
+ )
1761
+ assert resp.status_code == 200
1762
+ move_id = resp.json()["move_id"]
1763
+
1764
+ try:
1765
+ _wait_for_mode(base_url, "idle", timeout=COUNTDOWN_SECONDS + 2 + 20)
1766
+ except TimeoutError:
1767
+ global _server_stuck
1768
+ _server_stuck = True
1769
+ pytest.skip("Audio-only recording timed out")
1770
+
1771
+ # Verify it's marked as audio_only
1772
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1773
+ move = next((m for m in state["moves"] if m["id"] == move_id), None)
1774
+ assert move is not None
1775
+ assert move["audio_only"] is True
1776
+
1777
+ # Play it back (audio-only playback)
1778
+ _ensure_idle(base_url)
1779
+ resp = httpx.post(
1780
+ f"{base_url}/api/play",
1781
+ json={"move_id": move_id},
1782
+ timeout=5,
1783
+ )
1784
+ assert resp.status_code == 200
1785
+ _wait_for_mode(base_url, "idle", timeout=15)
1786
+
1787
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1788
+ assert state["mode"] == "idle"
1789
+
1790
+ # Cleanup
1791
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
1792
+
1793
+ def test_audio_only_appears_in_robot_audio_list(self, base_url: str, hw_marionette):
1794
+ """After audio-only recording, WAV should appear in robot-audio list."""
1795
+ import httpx
1796
+
1797
+ _ensure_idle(base_url)
1798
+ resp = httpx.post(
1799
+ f"{base_url}/api/record",
1800
+ json={
1801
+ "duration": 1.5,
1802
+ "record_audio": True,
1803
+ "record_motion": False,
1804
+ "label": "hw-ao-list",
1805
+ },
1806
+ timeout=5,
1807
+ )
1808
+ assert resp.status_code == 200
1809
+ move_id = resp.json()["move_id"]
1810
+
1811
+ try:
1812
+ _wait_for_mode(base_url, "idle", timeout=COUNTDOWN_SECONDS + 1.5 + 20)
1813
+ except TimeoutError:
1814
+ global _server_stuck
1815
+ _server_stuck = True
1816
+ pytest.skip("Audio-only recording timed out")
1817
+
1818
+ # Check robot-audio endpoint
1819
+ resp = httpx.get(f"{base_url}/api/robot-audio", timeout=5)
1820
+ assert resp.status_code == 200
1821
+ names = [f["name"] for f in resp.json()["files"]]
1822
+ assert move_id in names, (
1823
+ f"Expected {move_id} in robot-audio list, got: {names}"
1824
+ )
1825
+
1826
+ # Cleanup
1827
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
1828
+
1829
+
1830
+ class TestHardwareRobotFileSelection:
1831
+ """Test the robot file selection workflow on real hardware."""
1832
+
1833
+ def test_robot_audio_list_populated(self, base_url: str):
1834
+ """GET /api/robot-audio should return files after recordings exist."""
1835
+ import httpx
1836
+
1837
+ _ensure_idle(base_url)
1838
+ resp = httpx.get(f"{base_url}/api/robot-audio", timeout=5)
1839
+ assert resp.status_code == 200
1840
+ data = resp.json()
1841
+ assert "files" in data
1842
+ # Files may or may not exist depending on test order
1843
+ for f in data["files"]:
1844
+ assert "name" in f
1845
+ assert "path" in f
1846
+
1847
+ def test_select_robot_audio_sets_upload_id(self, base_url: str):
1848
+ """POST /api/robot-audio/select with a valid file returns upload_id."""
1849
+ import httpx
1850
+
1851
+ _ensure_idle(base_url)
1852
+ # Get a file from the list
1853
+ list_resp = httpx.get(f"{base_url}/api/robot-audio", timeout=5)
1854
+ files = list_resp.json().get("files", [])
1855
+ if not files:
1856
+ pytest.skip("No robot audio files available")
1857
+
1858
+ resp = httpx.post(
1859
+ f"{base_url}/api/robot-audio/select",
1860
+ json={"path": files[0]["path"]},
1861
+ timeout=5,
1862
+ )
1863
+ assert resp.status_code == 200
1864
+ data = resp.json()
1865
+ assert "upload_id" in data
1866
+ assert data["upload_id"] # non-empty
1867
+ assert "filename" in data
1868
+
1869
+ def test_select_robot_audio_and_record(self, base_url: str):
1870
+ """Select a robot audio file, then record motion with it."""
1871
+ import httpx
1872
+
1873
+ _ensure_idle(base_url)
1874
+ list_resp = httpx.get(f"{base_url}/api/robot-audio", timeout=5)
1875
+ files = list_resp.json().get("files", [])
1876
+ if not files:
1877
+ pytest.skip("No robot audio files available")
1878
+
1879
+ # Select the first file
1880
+ select_resp = httpx.post(
1881
+ f"{base_url}/api/robot-audio/select",
1882
+ json={"path": files[0]["path"]},
1883
+ timeout=5,
1884
+ )
1885
+ assert select_resp.status_code == 200
1886
+ upload_id = select_resp.json()["upload_id"]
1887
+
1888
+ # Record with the selected audio
1889
+ resp = httpx.post(
1890
+ f"{base_url}/api/record",
1891
+ json={
1892
+ "duration": 2.0,
1893
+ "record_audio": False,
1894
+ "record_motion": True,
1895
+ "uploaded_audio_id": upload_id,
1896
+ "label": "hw-robot-file",
1897
+ },
1898
+ timeout=5,
1899
+ )
1900
+ assert resp.status_code == 200
1901
+ move_id = resp.json()["move_id"]
1902
+
1903
+ _wait_for_mode(base_url, "idle", timeout=COUNTDOWN_SECONDS + 2 + 15)
1904
+
1905
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1906
+ move = next((m for m in state["moves"] if m["id"] == move_id), None)
1907
+ assert move is not None
1908
+ assert move["has_audio"] is True
1909
+
1910
+ # Cleanup
1911
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
1912
+
1913
+
1914
+ class TestHardwareStopResponsiveness:
1915
+ """Verify stop commands respond quickly during all phases."""
1916
+
1917
+ def test_stop_recording_during_countdown(self, base_url: str):
1918
+ """Stop during countdown — should return to idle within a few seconds."""
1919
+ import httpx
1920
+
1921
+ _ensure_idle(base_url)
1922
+ resp = httpx.post(
1923
+ f"{base_url}/api/record",
1924
+ json={"duration": 10.0, "record_audio": False, "label": "stop-countdown"},
1925
+ timeout=5,
1926
+ )
1927
+ assert resp.status_code == 200
1928
+
1929
+ # Wait for countdown to start
1930
+ time.sleep(0.3)
1931
+ t0 = time.time()
1932
+ resp = httpx.post(f"{base_url}/api/record/stop", timeout=5)
1933
+ assert resp.status_code == 200
1934
+
1935
+ state = _wait_for_mode(base_url, "idle", timeout=5)
1936
+ stop_time = time.time() - t0
1937
+ assert state["mode"] == "idle"
1938
+ print(f"\nStop during countdown: {stop_time:.2f}s")
1939
+
1940
+ def test_stop_recording_during_capture(self, base_url: str):
1941
+ """Stop during active recording — should return to idle within 5s."""
1942
+ import httpx
1943
+
1944
+ _ensure_idle(base_url)
1945
+ resp = httpx.post(
1946
+ f"{base_url}/api/record",
1947
+ json={"duration": 10.0, "record_audio": False, "label": "stop-capture"},
1948
+ timeout=5,
1949
+ )
1950
+ assert resp.status_code == 200
1951
+
1952
+ # Wait for recording phase
1953
+ _wait_for_mode(base_url, "recording", timeout=COUNTDOWN_SECONDS + 5)
1954
+ time.sleep(1.0)
1955
+
1956
+ t0 = time.time()
1957
+ resp = httpx.post(f"{base_url}/api/record/stop", timeout=5)
1958
+ assert resp.status_code == 200
1959
+
1960
+ state = _wait_for_mode(base_url, "idle", timeout=5)
1961
+ stop_time = time.time() - t0
1962
+ assert state["mode"] == "idle"
1963
+ assert stop_time < 5.0, f"Stop took too long: {stop_time:.2f}s"
1964
+ print(f"\nStop during capture: {stop_time:.2f}s")
1965
+
1966
+ # Cleanup
1967
+ state = httpx.get(f"{base_url}/api/state", timeout=5).json()
1968
+ for m in state["moves"]:
1969
+ if "stop-capture" in m.get("label", ""):
1970
+ httpx.delete(f"{base_url}/api/moves/{m['id']}", timeout=5)
1971
+
1972
+ def test_stop_playback_responds_quickly(self, base_url: str, hw_marionette):
1973
+ """Stop playback mid-stream — should return to idle within 5s."""
1974
+ import httpx
1975
+
1976
+ _ensure_idle(base_url)
1977
+
1978
+ # Record a 5s move
1979
+ resp = httpx.post(
1980
+ f"{base_url}/api/record",
1981
+ json={"duration": 5.0, "record_audio": False, "label": "stop-play-resp"},
1982
+ timeout=5,
1983
+ )
1984
+ assert resp.status_code == 200
1985
+ move_id = resp.json()["move_id"]
1986
+ _wait_for_mode(base_url, "idle", timeout=COUNTDOWN_SECONDS + 5 + 10)
1987
+
1988
+ # Play it
1989
+ _ensure_idle(base_url)
1990
+ resp = httpx.post(
1991
+ f"{base_url}/api/play",
1992
+ json={"move_id": move_id},
1993
+ timeout=5,
1994
+ )
1995
+ assert resp.status_code == 200
1996
+ _wait_for_mode(base_url, "playing", timeout=15)
1997
+ time.sleep(1.0)
1998
+
1999
+ # Stop and measure
2000
+ t0 = time.time()
2001
+ resp = httpx.post(f"{base_url}/api/play/stop", timeout=5)
2002
+ assert resp.status_code == 200
2003
+
2004
+ state = _wait_for_mode(base_url, "idle", timeout=10)
2005
+ stop_time = time.time() - t0
2006
+ assert state["mode"] == "idle"
2007
+ assert stop_time < 5.0, f"Stop took too long: {stop_time:.2f}s"
2008
+ print(f"\nStop during playback: {stop_time:.2f}s")
2009
+
2010
+ # Cleanup
2011
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
2012
+
2013
+
2014
  class TestHardwareAudio:
2015
  """Audio recording/playback tests — run last.
2016
 
 
2115
 
2116
  state = httpx.get(f"{base_url}/api/state", timeout=5).json()
2117
  assert state["mode"] == "idle"
2118
+
2119
+
2120
+ class TestAntennaCollisionSync:
2121
+ """Test audio-motion sync using antenna collisions + laptop mic.
2122
+
2123
+ Generates a move where beeps are synchronized with antenna collisions.
2124
+ Replays it on the robot, records via laptop mic, and measures the
2125
+ temporal offset between beep onsets and collision transients.
2126
+
2127
+ Requires: laptop with microphone, sounddevice pip package.
2128
+ """
2129
+
2130
+ def test_playback_sync_via_mic(self, base_url: str, hw_marionette, hw_reachy):
2131
+ """Inject sync test move, play back, record mic, measure offsets."""
2132
+ import httpx
2133
+ import soundfile as sf
2134
+ from audio_analysis import (
2135
+ MicRecorder,
2136
+ detect_beep_onsets,
2137
+ detect_transient_onsets,
2138
+ generate_collision_trajectory,
2139
+ generate_sync_test_audio,
2140
+ measure_sync_offsets,
2141
+ )
2142
+
2143
+ _ensure_idle(base_url)
2144
+
2145
+ duration = 8.0
2146
+ beep_freq = 1000.0
2147
+
2148
+ # 1. Generate sync test audio + collision trajectory
2149
+ audio_data, beep_times = generate_sync_test_audio(
2150
+ duration=duration, beep_freq=beep_freq,
2151
+ )
2152
+ timestamps, collision_frames = generate_collision_trajectory(
2153
+ beep_times, duration=duration,
2154
+ )
2155
+
2156
+ # 2. Write WAV + JSON to dataset dir
2157
+ move_id = "antenna-sync-test"
2158
+ wav_path = hw_marionette._dataset_dir / f"{move_id}.wav"
2159
+ json_path = hw_marionette._dataset_dir / f"{move_id}.json"
2160
+ sf.write(str(wav_path), audio_data, 48000)
2161
+ json_path.write_text(json.dumps({
2162
+ "description": "Antenna collision sync test",
2163
+ "time": timestamps,
2164
+ "set_target_data": collision_frames,
2165
+ }), encoding="utf-8")
2166
+ hw_marionette._refresh_recordings()
2167
+
2168
+ # 3. Start laptop mic recording
2169
+ recorder = MicRecorder(sr=48000)
2170
+ recorder.start()
2171
+ # Brief delay to ensure mic is capturing
2172
+ time.sleep(0.5)
2173
+
2174
+ try:
2175
+ # 4. Trigger playback via API
2176
+ resp = httpx.post(
2177
+ f"{base_url}/api/play",
2178
+ json={"move_id": move_id},
2179
+ timeout=5,
2180
+ )
2181
+ assert resp.status_code == 200
2182
+
2183
+ # Wait for playback to finish
2184
+ _wait_for_mode(base_url, "idle", timeout=duration + 20)
2185
+ time.sleep(0.5) # Capture tail end
2186
+ finally:
2187
+ captured = recorder.stop()
2188
+
2189
+ # 5. Analyze captured audio
2190
+ print(f"\nCaptured {len(captured)} samples ({len(captured)/48000:.2f}s)")
2191
+
2192
+ beep_onsets = detect_beep_onsets(captured, 48000, freq=beep_freq)
2193
+ collision_onsets = detect_transient_onsets(captured, 48000)
2194
+
2195
+ print(f"Detected {len(beep_onsets)} beeps: {[f'{t:.3f}s' for t in beep_onsets]}")
2196
+ print(f"Detected {len(collision_onsets)} collisions: {[f'{t:.3f}s' for t in collision_onsets]}")
2197
+
2198
+ # 6. Measure sync quality
2199
+ result = measure_sync_offsets(beep_onsets, collision_onsets)
2200
+ print(f"Matched {result['n_matched']}/{result['n_beeps']} beeps")
2201
+ for bt, ct, offset in result["pairs"]:
2202
+ print(f" beep@{bt:.3f}s -> collision@{ct:.3f}s = {offset:+.1f}ms")
2203
+ print(f"Mean offset: {result['mean_offset_ms']:.1f}ms")
2204
+ print(f"Max offset: {result['max_offset_ms']:.1f}ms")
2205
+ print(f"Std offset: {result['std_offset_ms']:.1f}ms")
2206
+
2207
+ # Assertions — generous thresholds for first iteration
2208
+ assert result["n_matched"] >= 3, (
2209
+ f"Only matched {result['n_matched']}/{result['n_beeps']} beep-collision pairs"
2210
+ )
2211
+ assert abs(result["mean_offset_ms"]) < 300, (
2212
+ f"Mean sync offset {result['mean_offset_ms']:.1f}ms exceeds 300ms"
2213
+ )
2214
+ assert result["max_offset_ms"] < 500, (
2215
+ f"Max sync offset {result['max_offset_ms']:.1f}ms exceeds 500ms"
2216
+ )
2217
+
2218
+ # Cleanup
2219
+ httpx.delete(f"{base_url}/api/moves/{move_id}", timeout=5)
tests/test_marionette_sync.py ADDED
@@ -0,0 +1,614 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """End-to-end sync tests through the Marionette stack.
3
+
4
+ Tests audio-motion synchronization by playing a synthetic move through
5
+ Marionette's full playback pipeline (push_audio_sample + motion loop)
6
+ and measuring beep-to-collision intervals with a laptop microphone.
7
+
8
+ Requires:
9
+ - Marionette running on the robot (via deploy_wireless.sh or daemon)
10
+ - Laptop microphone connected and working
11
+ - Robot accessible at --host (default: reachy-mini.local)
12
+
13
+ Usage:
14
+ python tests/test_marionette_sync.py [--host reachy-mini.local]
15
+ """
16
+ from __future__ import annotations
17
+
18
+ import argparse
19
+ import json
20
+ import subprocess
21
+ import sys
22
+ import time
23
+ from pathlib import Path
24
+
25
+ import numpy as np
26
+ import requests
27
+ import sounddevice as sd
28
+ import soundfile as sf
29
+
30
+ sys.path.insert(0, str(Path(__file__).parent))
31
+ from audio_analysis import detect_beep_onsets, detect_transient_onsets
32
+
33
+ # ── Timing (same as all previous sync tests) ──────────────────────────
34
+ BEEP_TIMES = [1.0, 2.3, 4.0, 6.3, 9.4]
35
+ BEEP_COLLISION_OFFSET = 1.0
36
+ COLLISION_TIMES = [t + BEEP_COLLISION_OFFSET for t in BEEP_TIMES]
37
+
38
+ # Audio
39
+ BEEP_FREQ = 2000.0
40
+ BEEP_DURATION = 0.2
41
+ BEEP_AMPLITUDE = 0.9
42
+ ROBOT_SR = 16000
43
+
44
+ # Collision
45
+ RIGHT_REST = -0.68
46
+ LEFT_REST = 0.0
47
+ LEFT_COLLISION = 0.70
48
+ HOLD_DURATION = 0.2
49
+
50
+ MOTION_SR = 100
51
+ MOVE_ID = "sync-test-e2e"
52
+
53
+ ROBOT_USER = "pollen"
54
+ LAPTOP_SR = 48000
55
+ MIC_DURATION = 35.0 # longer: Marionette has goto + playback
56
+ MARIONETTE_PORT = 8042
57
+
58
+
59
+ def get_marionette_url(host: str) -> str:
60
+ return f"http://{host}:{MARIONETTE_PORT}"
61
+
62
+
63
+ def wait_for_mode(base_url: str, target: str, timeout: float = 30.0) -> dict:
64
+ """Poll GET /api/state until mode matches target."""
65
+ deadline = time.monotonic() + timeout
66
+ while time.monotonic() < deadline:
67
+ try:
68
+ r = requests.get(f"{base_url}/api/state", timeout=3)
69
+ state = r.json()
70
+ if state.get("mode") == target:
71
+ return state
72
+ except Exception:
73
+ pass
74
+ time.sleep(0.3)
75
+ raise TimeoutError(f"Mode never reached '{target}' within {timeout}s")
76
+
77
+
78
+ def ensure_idle(base_url: str) -> dict:
79
+ """Make sure Marionette is idle, stopping anything in progress."""
80
+ try:
81
+ state = requests.get(f"{base_url}/api/state", timeout=3).json()
82
+ except Exception as exc:
83
+ raise RuntimeError(f"Cannot reach Marionette at {base_url}: {exc}") from exc
84
+
85
+ mode = state.get("mode", "unknown")
86
+ if mode == "idle":
87
+ return state
88
+
89
+ # Try to stop whatever is running
90
+ if mode == "playing":
91
+ requests.post(f"{base_url}/api/play/stop", timeout=3)
92
+ elif mode in {"recording", "countdown", "preparing"}:
93
+ requests.post(f"{base_url}/api/record/stop", timeout=3)
94
+ return wait_for_mode(base_url, "idle", timeout=10)
95
+
96
+
97
+ def generate_move_json() -> dict:
98
+ """Generate Marionette-format move data with collision trajectory."""
99
+ total_duration = max(COLLISION_TIMES) + HOLD_DURATION + 1.0
100
+ dt = 1.0 / MOTION_SR
101
+ n_frames = int(total_duration * MOTION_SR)
102
+
103
+ identity_head = np.eye(4).tolist()
104
+ left_targets = np.full(n_frames, LEFT_REST, dtype=np.float64)
105
+ for ct in COLLISION_TIMES:
106
+ start = int(ct * MOTION_SR)
107
+ end = min(int((ct + HOLD_DURATION) * MOTION_SR), n_frames)
108
+ left_targets[start:end] = LEFT_COLLISION
109
+
110
+ timestamps = []
111
+ frames = []
112
+ for i in range(n_frames):
113
+ t = round(i * dt, 4)
114
+ timestamps.append(t)
115
+ frames.append({
116
+ "head": identity_head,
117
+ "antennas": [float(left_targets[i]), RIGHT_REST],
118
+ "body_yaw": 0.0,
119
+ "check_collision": False,
120
+ })
121
+
122
+ return {
123
+ "description": "E2E sync test: beeps + antenna collisions",
124
+ "time": timestamps,
125
+ "set_target_data": frames,
126
+ }
127
+
128
+
129
+ def generate_move_wav(path: Path) -> None:
130
+ """Generate WAV with beeps at known times."""
131
+ total_duration = max(COLLISION_TIMES) + HOLD_DURATION + 1.0
132
+ n_audio = int(total_duration * ROBOT_SR)
133
+ audio = np.zeros(n_audio, dtype=np.float32)
134
+ for bt in BEEP_TIMES:
135
+ start = int(bt * ROBOT_SR)
136
+ n_beep = int(BEEP_DURATION * ROBOT_SR)
137
+ if start + n_beep > n_audio:
138
+ continue
139
+ t_arr = np.arange(n_beep, dtype=np.float32) / ROBOT_SR
140
+ beep = BEEP_AMPLITUDE * np.sin(2 * np.pi * BEEP_FREQ * t_arr).astype(np.float32)
141
+ fade = int(0.005 * ROBOT_SR)
142
+ if fade > 0 and 2 * fade < n_beep:
143
+ beep[:fade] *= np.linspace(0, 1, fade, dtype=np.float32)
144
+ beep[-fade:] *= np.linspace(1, 0, fade, dtype=np.float32)
145
+ audio[start:start + n_beep] += beep
146
+ sf.write(str(path), audio, ROBOT_SR)
147
+
148
+
149
+ def inject_move(host: str, base_url: str) -> None:
150
+ """Inject the synthetic move into the running Marionette's dataset."""
151
+ # Get the active dataset path from Marionette
152
+ state = requests.get(f"{base_url}/api/state", timeout=3).json()
153
+ dataset_path = state["config"]["active_dataset_path"]
154
+ active_id = state.get("datasets", {}).get("active_id")
155
+ print(f" Active dataset path: {dataset_path}")
156
+
157
+ # Generate move files locally
158
+ import tempfile
159
+ with tempfile.TemporaryDirectory() as tmpdir:
160
+ tmpdir = Path(tmpdir)
161
+ json_path = tmpdir / f"{MOVE_ID}.json"
162
+ wav_path = tmpdir / f"{MOVE_ID}.wav"
163
+
164
+ move_data = generate_move_json()
165
+ json_path.write_text(json.dumps(move_data), encoding="utf-8")
166
+ generate_move_wav(wav_path)
167
+
168
+ total_duration = max(COLLISION_TIMES) + HOLD_DURATION + 1.0
169
+ print(f" Generated move: {total_duration:.1f}s, {len(BEEP_TIMES)} beeps, {len(COLLISION_TIMES)} collisions")
170
+
171
+ # SCP to robot's dataset directory
172
+ for local_file in [json_path, wav_path]:
173
+ remote_target = f"{ROBOT_USER}@{host}:{dataset_path}/{local_file.name}"
174
+ result = subprocess.run(
175
+ ["scp", "-o", "ConnectTimeout=5", str(local_file), remote_target],
176
+ capture_output=True, text=True, timeout=15,
177
+ )
178
+ if result.returncode != 0:
179
+ raise RuntimeError(f"SCP failed: {result.stderr}")
180
+
181
+ # Trigger Marionette to re-scan recordings by re-selecting the active dataset
182
+ if active_id:
183
+ requests.post(
184
+ f"{base_url}/api/datasets/select",
185
+ json={"dataset_id": active_id},
186
+ timeout=5,
187
+ )
188
+ time.sleep(0.5)
189
+
190
+ # Verify the move is visible
191
+ state = requests.get(f"{base_url}/api/state", timeout=3).json()
192
+ move_ids = [m["id"] for m in state.get("moves", [])]
193
+ if MOVE_ID not in move_ids:
194
+ raise RuntimeError(
195
+ f"Move '{MOVE_ID}' not found after injection. "
196
+ f"Available: {move_ids}"
197
+ )
198
+ print(f" Move '{MOVE_ID}' injected and visible in Marionette")
199
+
200
+
201
+ def cleanup_move(host: str, base_url: str) -> None:
202
+ """Remove the synthetic test move."""
203
+ try:
204
+ requests.delete(f"{base_url}/api/moves/{MOVE_ID}", timeout=5)
205
+ except Exception:
206
+ pass
207
+
208
+
209
+ def plot_results(
210
+ mic_audio, mic_sr, detected_beeps, detected_collisions, pairs, output_path,
211
+ ):
212
+ import matplotlib
213
+ matplotlib.use("Agg")
214
+ import matplotlib.pyplot as plt
215
+
216
+ mic_t = np.arange(len(mic_audio)) / mic_sr
217
+
218
+ fig, ax = plt.subplots(1, 1, figsize=(18, 6))
219
+ ax.plot(mic_t, mic_audio, "k-", linewidth=0.3, alpha=0.5)
220
+ ax.set_ylabel("Mic amplitude")
221
+ ax.set_title("Marionette E2E Playback Sync Test — Laptop Mic Recording")
222
+ ax.grid(True, alpha=0.3)
223
+
224
+ for i, bt in enumerate(detected_beeps):
225
+ ax.axvline(bt, color="blue", linestyle="-", linewidth=1.2, alpha=0.7,
226
+ label="Detected beep" if i == 0 else None)
227
+ for i, ct in enumerate(detected_collisions):
228
+ ax.axvline(ct, color="red", linestyle="-", linewidth=1.2, alpha=0.7,
229
+ label="Detected collision" if i == 0 else None)
230
+ for p in pairs:
231
+ mid = (p["beep_t"] + p["collision_t"]) / 2
232
+ ax.annotate(f'{p["interval_ms"]:.0f}ms', xy=(mid, 0),
233
+ ha="center", fontsize=9, color="purple", fontweight="bold",
234
+ bbox=dict(boxstyle="round,pad=0.2", facecolor="lightyellow", alpha=0.8))
235
+
236
+ ax.legend(loc="upper right", fontsize=9)
237
+
238
+ all_events = detected_beeps + detected_collisions
239
+ if all_events:
240
+ ax.set_xlim(min(all_events) - 1.0, max(all_events) + 1.0)
241
+ ax.set_xlabel("Time since mic start (s)")
242
+
243
+ fig.tight_layout()
244
+ fig.savefig(str(output_path), dpi=150)
245
+ plt.close(fig)
246
+ print(f" Plot saved to {output_path}")
247
+
248
+
249
+ def run_playback_test(host: str) -> dict:
250
+ """Run the Marionette playback sync test.
251
+
252
+ Returns a results dict with pairs, errors, and verdict.
253
+ """
254
+ base_url = get_marionette_url(host)
255
+ print(f"\n{'='*60}")
256
+ print("Marionette E2E Playback Sync Test")
257
+ print(f"{'='*60}")
258
+ print(f" Server: {base_url}")
259
+ print(f" Beep times: {BEEP_TIMES}")
260
+ print(f" Collision times: {COLLISION_TIMES}")
261
+ print(f" Expected interval: {BEEP_COLLISION_OFFSET*1000:.0f}ms\n")
262
+
263
+ # Step 1: Ensure idle
264
+ print("[1/6] Ensuring Marionette is idle...")
265
+ ensure_idle(base_url)
266
+ print(" Idle.")
267
+
268
+ # Step 2: Inject synthetic move
269
+ print("[2/6] Injecting synthetic move into dataset...")
270
+ inject_move(host, base_url)
271
+
272
+ # Step 3: Start mic recording
273
+ print(f"\n[3/6] Starting mic recording ({MIC_DURATION}s)...")
274
+ mic_start = time.monotonic()
275
+ mic_data = sd.rec(
276
+ int(MIC_DURATION * LAPTOP_SR),
277
+ samplerate=LAPTOP_SR, channels=1, dtype="float32",
278
+ )
279
+
280
+ # Step 4: Trigger playback
281
+ time.sleep(0.3)
282
+ print("[4/6] Triggering playback via API...")
283
+ play_start = time.monotonic()
284
+ r = requests.post(f"{base_url}/api/play", json={"move_id": MOVE_ID}, timeout=5)
285
+ if r.status_code != 200:
286
+ sd.stop()
287
+ raise RuntimeError(f"Play failed: {r.status_code} {r.text}")
288
+ print(f" Play accepted at mic_t={play_start - mic_start:.3f}s")
289
+
290
+ # Step 5: Wait for playback to finish
291
+ print("[5/6] Waiting for playback to finish...")
292
+ try:
293
+ state = wait_for_mode(base_url, "idle", timeout=30)
294
+ play_end = time.monotonic()
295
+ print(f" Playback done at mic_t={play_end - mic_start:.3f}s")
296
+ except TimeoutError:
297
+ print(" WARNING: Playback did not finish in time")
298
+
299
+ sd.wait()
300
+ captured = mic_data.flatten()
301
+ print(f" Mic recording done ({len(captured)/LAPTOP_SR:.1f}s)")
302
+
303
+ mic_path = Path("tests/marionette_sync_mic.wav")
304
+ sf.write(str(mic_path), captured, LAPTOP_SR)
305
+ print(f" Saved to {mic_path}")
306
+
307
+ # Step 6: Analyze
308
+ print(f"\n[6/6] Analyzing...")
309
+ detected_beeps = detect_beep_onsets(
310
+ captured, LAPTOP_SR, freq=BEEP_FREQ, bandwidth=150.0,
311
+ threshold_db=-12.0, min_separation=1.0,
312
+ )
313
+ detected_collisions = detect_transient_onsets(
314
+ captured, LAPTOP_SR, highpass_freq=3000.0,
315
+ )
316
+ print(f" Detected {len(detected_beeps)} beeps at: {[f'{t:.3f}' for t in detected_beeps]}")
317
+ print(f" Detected {len(detected_collisions)} collisions at: {[f'{t:.3f}' for t in detected_collisions]}")
318
+
319
+ # Match pairs
320
+ print(f"\n{'='*60}")
321
+ print("Beep → Collision Interval Analysis (Marionette E2E)")
322
+ print(f" (Expected interval: {BEEP_COLLISION_OFFSET*1000:.0f}ms)")
323
+ print(f"{'='*60}")
324
+
325
+ pairs = []
326
+ for i, bt in enumerate(detected_beeps):
327
+ candidates = [ct for ct in detected_collisions if 0.3 < (ct - bt) < 2.0]
328
+ if not candidates:
329
+ print(f" Beep {i+1} at {bt:.3f}s: NO COLLISION FOUND")
330
+ continue
331
+ nearest = min(candidates, key=lambda ct: abs((ct - bt) - BEEP_COLLISION_OFFSET))
332
+ interval_ms = (nearest - bt) * 1000
333
+ error_ms = interval_ms - BEEP_COLLISION_OFFSET * 1000
334
+ pairs.append({
335
+ "beep_t": bt,
336
+ "collision_t": nearest,
337
+ "interval_ms": interval_ms,
338
+ "error_ms": error_ms,
339
+ })
340
+ print(f" Pair {len(pairs)}: beep {bt:.3f}s → collision {nearest:.3f}s = "
341
+ f"{interval_ms:.0f}ms (error {error_ms:+.0f}ms)")
342
+
343
+ result = {
344
+ "test": "marionette_playback_sync",
345
+ "n_beeps_detected": len(detected_beeps),
346
+ "n_collisions_detected": len(detected_collisions),
347
+ "n_pairs": len(pairs),
348
+ "pairs": pairs,
349
+ }
350
+
351
+ if pairs:
352
+ errors = [p["error_ms"] for p in pairs]
353
+ intervals = [p["interval_ms"] for p in pairs]
354
+ result["mean_interval_ms"] = float(np.mean(intervals))
355
+ result["mean_error_ms"] = float(np.mean(errors))
356
+ result["std_error_ms"] = float(np.std(errors))
357
+ result["min_error_ms"] = float(min(errors))
358
+ result["max_error_ms"] = float(max(errors))
359
+ print(f"\n Pairs matched: {len(pairs)}/{len(BEEP_TIMES)}")
360
+ print(f" Mean interval: {np.mean(intervals):.0f}ms (expected {BEEP_COLLISION_OFFSET*1000:.0f}ms)")
361
+ print(f" Mean error: {np.mean(errors):+.0f}ms")
362
+ print(f" Std error: {np.std(errors):.0f}ms")
363
+ print(f" Min/Max error: {min(errors):+.0f}ms / {max(errors):+.0f}ms")
364
+
365
+ # Plot
366
+ plot_path = Path("tests/marionette_sync_plot.png")
367
+ plot_results(captured, LAPTOP_SR, detected_beeps, detected_collisions, pairs, plot_path)
368
+
369
+ # Cleanup
370
+ cleanup_move(host, base_url)
371
+
372
+ success = len(pairs) >= len(BEEP_TIMES) - 1
373
+ result["success"] = success
374
+
375
+ print(f"\n{'='*60}")
376
+ if success:
377
+ print("RESULT: PASS — Beep-collision pairs detected via Marionette E2E")
378
+ else:
379
+ print("RESULT: FAIL — Could not reliably detect pairs")
380
+ print(f"{'='*60}\n")
381
+
382
+ return result
383
+
384
+
385
+ def run_recording_test(host: str) -> dict:
386
+ """Test the 3-2-1 countdown timing accuracy.
387
+
388
+ Triggers a recording with a known audio file, records with the
389
+ laptop mic, and measures:
390
+ - Countdown beep timing (3 beeps at 440Hz, 1s apart)
391
+ - "Go" beep timing (880Hz)
392
+ - When the uploaded audio actually starts playing
393
+ """
394
+ base_url = get_marionette_url(host)
395
+ print(f"\n{'='*60}")
396
+ print("Marionette Recording Countdown Sync Test")
397
+ print(f"{'='*60}")
398
+ print(f" Server: {base_url}\n")
399
+
400
+ # Step 1: Ensure idle
401
+ print("[1/5] Ensuring Marionette is idle...")
402
+ ensure_idle(base_url)
403
+
404
+ # Step 2: Upload a known audio file (a single 2kHz beep at t=0.5)
405
+ # This beep will play after the countdown, so we can measure
406
+ # the delay from "go" beep to actual audio playback start.
407
+ print("[2/5] Generating and uploading test audio...")
408
+ import tempfile
409
+ with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
410
+ test_wav = Path(f.name)
411
+
412
+ # 3-second audio with a single 2kHz marker beep at t=0.2
413
+ # (early, so it's clearly after the go beep)
414
+ marker_time = 0.2
415
+ audio_duration = 3.0
416
+ n_audio = int(audio_duration * 48000) # 48kHz for upload
417
+ audio = np.zeros(n_audio, dtype=np.float32)
418
+ start = int(marker_time * 48000)
419
+ n_beep = int(BEEP_DURATION * 48000)
420
+ t_arr = np.arange(n_beep, dtype=np.float32) / 48000
421
+ beep = BEEP_AMPLITUDE * np.sin(2 * np.pi * BEEP_FREQ * t_arr).astype(np.float32)
422
+ fade = int(0.005 * 48000)
423
+ if fade > 0:
424
+ beep[:fade] *= np.linspace(0, 1, fade, dtype=np.float32)
425
+ beep[-fade:] *= np.linspace(1, 0, fade, dtype=np.float32)
426
+ audio[start:start + n_beep] = beep
427
+ sf.write(str(test_wav), audio, 48000)
428
+ print(f" Test audio: {audio_duration}s, marker beep at {marker_time}s (2kHz)")
429
+
430
+ # Upload via Marionette API
431
+ with open(test_wav, "rb") as f:
432
+ r = requests.post(
433
+ f"{base_url}/api/upload-audio",
434
+ files={"file": ("countdown_test.wav", f, "audio/wav")},
435
+ timeout=10,
436
+ )
437
+ test_wav.unlink()
438
+ if r.status_code != 200:
439
+ raise RuntimeError(f"Upload failed: {r.status_code} {r.text}")
440
+ upload_info = r.json()
441
+ upload_id = upload_info["upload_id"]
442
+ print(f" Uploaded: id={upload_id}")
443
+
444
+ # Step 3: Start mic recording
445
+ print(f"\n[3/5] Starting mic recording (15s)...")
446
+ mic_duration = 15.0
447
+ mic_start = time.monotonic()
448
+ mic_data = sd.rec(
449
+ int(mic_duration * LAPTOP_SR),
450
+ samplerate=LAPTOP_SR, channels=1, dtype="float32",
451
+ )
452
+
453
+ # Step 4: Trigger recording
454
+ time.sleep(0.3)
455
+ print("[4/5] Triggering recording with audio...")
456
+ rec_start = time.monotonic()
457
+ r = requests.post(
458
+ f"{base_url}/api/record",
459
+ json={
460
+ "label": "countdown-test",
461
+ "duration": 3.0,
462
+ "record_audio": False,
463
+ "record_motion": True,
464
+ "uploaded_audio_id": upload_id,
465
+ },
466
+ timeout=5,
467
+ )
468
+ if r.status_code != 200:
469
+ sd.stop()
470
+ raise RuntimeError(f"Record failed: {r.status_code} {r.text}")
471
+ rec_move_id = r.json().get("move_id")
472
+ print(f" Recording accepted at mic_t={rec_start - mic_start:.3f}s, move_id={rec_move_id}")
473
+
474
+ # Wait for recording to finish (countdown ~3s + recording ~3s)
475
+ print("[5/5] Waiting for recording to finish...")
476
+ try:
477
+ wait_for_mode(base_url, "idle", timeout=20)
478
+ rec_end = time.monotonic()
479
+ print(f" Recording done at mic_t={rec_end - mic_start:.3f}s")
480
+ except TimeoutError:
481
+ print(" WARNING: Recording did not finish in time")
482
+
483
+ sd.wait()
484
+ captured = mic_data.flatten()
485
+
486
+ mic_path = Path("tests/marionette_countdown_mic.wav")
487
+ sf.write(str(mic_path), captured, LAPTOP_SR)
488
+ print(f" Saved to {mic_path}")
489
+
490
+ # Analyze: detect countdown beeps (440Hz) and go beep (880Hz) and marker (2kHz)
491
+ print(f"\n{'='*60}")
492
+ print("Countdown Timing Analysis")
493
+ print(f"{'='*60}")
494
+
495
+ # Detect countdown beeps at 440Hz
496
+ countdown_beeps = detect_beep_onsets(
497
+ captured, LAPTOP_SR, freq=440.0, bandwidth=80.0,
498
+ threshold_db=-12.0, min_separation=0.8,
499
+ )
500
+ print(f" Countdown beeps (440Hz): {len(countdown_beeps)} at {[f'{t:.3f}' for t in countdown_beeps]}")
501
+
502
+ # Detect "go" beep at 880Hz
503
+ go_beeps = detect_beep_onsets(
504
+ captured, LAPTOP_SR, freq=880.0, bandwidth=80.0,
505
+ threshold_db=-12.0, min_separation=0.5,
506
+ )
507
+ print(f" Go beep (880Hz): {len(go_beeps)} at {[f'{t:.3f}' for t in go_beeps]}")
508
+
509
+ # Detect marker beep at 2kHz (from uploaded audio)
510
+ marker_beeps = detect_beep_onsets(
511
+ captured, LAPTOP_SR, freq=BEEP_FREQ, bandwidth=150.0,
512
+ threshold_db=-12.0, min_separation=0.5,
513
+ )
514
+ print(f" Marker beep (2kHz): {len(marker_beeps)} at {[f'{t:.3f}' for t in marker_beeps]}")
515
+
516
+ result = {
517
+ "test": "marionette_recording_countdown",
518
+ "countdown_beeps": countdown_beeps,
519
+ "go_beeps": go_beeps,
520
+ "marker_beeps": marker_beeps,
521
+ }
522
+
523
+ # Analyze countdown spacing
524
+ if len(countdown_beeps) >= 2:
525
+ gaps = [countdown_beeps[i+1] - countdown_beeps[i] for i in range(len(countdown_beeps)-1)]
526
+ print(f"\n Countdown gaps: {[f'{g:.3f}s' for g in gaps]} (expected ~1.0s each)")
527
+ result["countdown_gaps"] = gaps
528
+ gap_errors = [abs(g - 1.0) * 1000 for g in gaps]
529
+ print(f" Gap errors: {[f'{e:.0f}ms' for e in gap_errors]}")
530
+
531
+ # Analyze go-to-marker delay
532
+ # Filter go beeps: discard any that overlap with countdown beeps (440Hz
533
+ # harmonic leaks into the 880Hz band). The real "go" beep comes AFTER the
534
+ # last countdown beep.
535
+ if countdown_beeps and go_beeps:
536
+ last_cd = countdown_beeps[-1]
537
+ go_beeps_filtered = [t for t in go_beeps if t > last_cd + 0.3]
538
+ print(f" Go beeps after countdown: {[f'{t:.3f}' for t in go_beeps_filtered]}")
539
+ else:
540
+ go_beeps_filtered = go_beeps
541
+
542
+ if go_beeps_filtered and marker_beeps:
543
+ go_t = go_beeps_filtered[0]
544
+ marker_t = marker_beeps[0]
545
+ delay_ms = (marker_t - go_t) * 1000
546
+ print(f"\n Go beep → Marker beep: {delay_ms:.0f}ms")
547
+ print(f" (Marker is at {marker_time}s in audio file)")
548
+ print(f" Expected if no pipeline latency: ~{marker_time*1000:.0f}ms")
549
+ print(f" Extra delay (pipeline latency): ~{delay_ms - marker_time*1000:.0f}ms")
550
+ result["go_to_marker_ms"] = delay_ms
551
+ result["estimated_pipeline_latency_ms"] = delay_ms - marker_time * 1000
552
+
553
+ # Cleanup: delete the test recording
554
+ if rec_move_id:
555
+ try:
556
+ requests.delete(f"{base_url}/api/moves/{rec_move_id}", timeout=5)
557
+ except Exception:
558
+ pass
559
+
560
+ success = len(countdown_beeps) >= 2 and len(marker_beeps) >= 1
561
+ result["success"] = success
562
+
563
+ print(f"\n{'='*60}")
564
+ if success:
565
+ print("RESULT: PASS — Countdown and marker beeps detected")
566
+ else:
567
+ print("RESULT: FAIL — Could not detect expected beeps")
568
+ print(f"{'='*60}\n")
569
+
570
+ return result
571
+
572
+
573
+ def main():
574
+ parser = argparse.ArgumentParser(description="Marionette E2E sync tests")
575
+ parser.add_argument("--host", default="reachy-mini.local")
576
+ parser.add_argument("--test", choices=["playback", "recording", "both"],
577
+ default="both", help="Which test to run")
578
+ args = parser.parse_args()
579
+
580
+ results = {}
581
+
582
+ if args.test in ("playback", "both"):
583
+ results["playback"] = run_playback_test(args.host)
584
+
585
+ if args.test in ("recording", "both"):
586
+ results["recording"] = run_recording_test(args.host)
587
+
588
+ # Save results
589
+ results_path = Path("tests/marionette_sync_results.json")
590
+ results_path.write_text(json.dumps(results, indent=2, default=str))
591
+ print(f"\nResults saved to {results_path}")
592
+
593
+ # Summary
594
+ print(f"\n{'='*60}")
595
+ print("SUMMARY")
596
+ print(f"{'='*60}")
597
+ for name, r in results.items():
598
+ status = "PASS" if r.get("success") else "FAIL"
599
+ if name == "playback" and "mean_error_ms" in r:
600
+ print(f" {name}: {status} — mean error {r['mean_error_ms']:+.0f}ms, "
601
+ f"std {r['std_error_ms']:.0f}ms")
602
+ elif name == "recording" and "go_to_marker_ms" in r:
603
+ print(f" {name}: {status} — go→marker {r['go_to_marker_ms']:.0f}ms, "
604
+ f"pipeline latency ~{r.get('estimated_pipeline_latency_ms', 0):.0f}ms")
605
+ else:
606
+ print(f" {name}: {status}")
607
+ print(f"{'='*60}\n")
608
+
609
+ all_pass = all(r.get("success", False) for r in results.values())
610
+ return 0 if all_pass else 1
611
+
612
+
613
+ if __name__ == "__main__":
614
+ sys.exit(main())
tests/test_move_sync.py ADDED
@@ -0,0 +1,464 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Test beep+collision sync using a Marionette-format move played via SDK.
3
+
4
+ Intermediate test between the direct-control script and the full Marionette
5
+ frontend. Creates a synthetic move (JSON + WAV) matching the beep+collision
6
+ test, plays it on the robot via reachy_mini SDK's play_move(), and records
7
+ with the laptop mic.
8
+
9
+ This tests the SDK's built-in audio-motion sync mechanism (play_sound +
10
+ motion loop) rather than our manual push_audio_sample approach.
11
+
12
+ Usage:
13
+ python tests/test_move_sync.py [--host reachy-mini.local]
14
+ """
15
+ from __future__ import annotations
16
+
17
+ import argparse
18
+ import json
19
+ import subprocess
20
+ import sys
21
+ import tempfile
22
+ import time
23
+ from pathlib import Path
24
+
25
+ import numpy as np
26
+ import sounddevice as sd
27
+ import soundfile as sf
28
+
29
+ sys.path.insert(0, str(Path(__file__).parent))
30
+ from audio_analysis import detect_beep_onsets, detect_transient_onsets
31
+
32
+ # Timing — same as test_beep_collision_sync.py
33
+ BEEP_TIMES = [1.0, 2.3, 4.0, 6.3, 9.4]
34
+ BEEP_COLLISION_OFFSET = 1.0
35
+ COLLISION_TIMES = [t + BEEP_COLLISION_OFFSET for t in BEEP_TIMES]
36
+
37
+ # Audio
38
+ BEEP_FREQ = 2000.0
39
+ BEEP_DURATION = 0.2
40
+ BEEP_AMPLITUDE = 0.9
41
+ ROBOT_SR = 16000
42
+
43
+ # Collision
44
+ RIGHT_REST = -0.68
45
+ LEFT_REST = 0.0
46
+ LEFT_COLLISION = 0.70
47
+ HOLD_DURATION = 0.2
48
+
49
+ MOTION_SR = 100 # 100Hz motion sampling, standard for Marionette
50
+
51
+ ROBOT_USER = "pollen"
52
+ ROBOT_PYTHON = "/venvs/apps_venv/bin/python"
53
+ LAPTOP_SR = 48000
54
+ MIC_DURATION = 30.0
55
+ REMOTE_DIR = "/tmp/sync_test_move"
56
+ REMOTE_RESULTS = "/tmp/move_sync_positions.json"
57
+
58
+
59
+ def generate_move_files(tmpdir: Path) -> tuple[Path, Path]:
60
+ """Generate Marionette-format JSON + WAV for the beep+collision test."""
61
+ total_duration = max(COLLISION_TIMES) + HOLD_DURATION + 1.0
62
+ dt = 1.0 / MOTION_SR
63
+ n_frames = int(total_duration * MOTION_SR)
64
+
65
+ identity_head = np.eye(4).tolist()
66
+
67
+ # Build collision timeline
68
+ left_targets = np.full(n_frames, LEFT_REST, dtype=np.float64)
69
+ for ct in COLLISION_TIMES:
70
+ start = int(ct * MOTION_SR)
71
+ end = int((ct + HOLD_DURATION) * MOTION_SR)
72
+ end = min(end, n_frames)
73
+ left_targets[start:end] = LEFT_COLLISION
74
+
75
+ timestamps = []
76
+ frames = []
77
+ for i in range(n_frames):
78
+ t = i * dt
79
+ timestamps.append(round(t, 4))
80
+ frames.append({
81
+ "head": identity_head,
82
+ "antennas": [float(left_targets[i]), RIGHT_REST],
83
+ "body_yaw": 0.0,
84
+ "check_collision": False,
85
+ })
86
+
87
+ move_data = {
88
+ "description": "Sync test: beeps + antenna collisions",
89
+ "time": timestamps,
90
+ "set_target_data": frames,
91
+ }
92
+
93
+ json_path = tmpdir / "sync-test.json"
94
+ json_path.write_text(json.dumps(move_data), encoding="utf-8")
95
+
96
+ # Generate WAV with beeps
97
+ n_audio = int(total_duration * ROBOT_SR)
98
+ audio = np.zeros(n_audio, dtype=np.float32)
99
+ for bt in BEEP_TIMES:
100
+ start = int(bt * ROBOT_SR)
101
+ n_beep = int(BEEP_DURATION * ROBOT_SR)
102
+ if start + n_beep > n_audio:
103
+ continue
104
+ t_arr = np.arange(n_beep, dtype=np.float32) / ROBOT_SR
105
+ beep = BEEP_AMPLITUDE * np.sin(2 * np.pi * BEEP_FREQ * t_arr).astype(np.float32)
106
+ fade = int(0.005 * ROBOT_SR)
107
+ if fade > 0 and 2 * fade < n_beep:
108
+ beep[:fade] *= np.linspace(0, 1, fade, dtype=np.float32)
109
+ beep[-fade:] *= np.linspace(1, 0, fade, dtype=np.float32)
110
+ audio[start:start + n_beep] += beep
111
+
112
+ wav_path = tmpdir / "sync-test.wav"
113
+ sf.write(str(wav_path), audio, ROBOT_SR)
114
+
115
+ print(f" Generated move: {n_frames} frames at {MOTION_SR}Hz, {total_duration:.1f}s")
116
+ print(f" Generated WAV: {n_audio} samples at {ROBOT_SR}Hz, {len(BEEP_TIMES)} beeps")
117
+ return json_path, wav_path
118
+
119
+
120
+ def scp_to_robot(local_path: Path, remote_path: str, host: str) -> None:
121
+ target = f"{ROBOT_USER}@{host}:{remote_path}"
122
+ result = subprocess.run(
123
+ ["scp", "-o", "ConnectTimeout=5", str(local_path), target],
124
+ capture_output=True, text=True, timeout=15,
125
+ )
126
+ if result.returncode != 0:
127
+ raise RuntimeError(f"SCP failed: {result.stderr}")
128
+
129
+
130
+ def scp_from_robot(remote_path: str, local_path: Path, host: str) -> None:
131
+ source = f"{ROBOT_USER}@{host}:{remote_path}"
132
+ result = subprocess.run(
133
+ ["scp", "-o", "ConnectTimeout=5", source, str(local_path)],
134
+ capture_output=True, text=True, timeout=15,
135
+ )
136
+ if result.returncode != 0:
137
+ raise RuntimeError(f"SCP failed: {result.stderr}")
138
+
139
+
140
+ # Robot-side playback script: loads the move, plays via SDK, records positions
141
+ ROBOT_PLAY_SCRIPT = """\
142
+ import json, os, sys, time
143
+ import numpy as np
144
+ from pathlib import Path
145
+
146
+ move_dir = sys.argv[1]
147
+ results_path = sys.argv[2]
148
+
149
+ print("robot: connecting to ReachyMini", flush=True)
150
+ from reachy_mini import ReachyMini
151
+ from reachy_mini.motion.recorded_move import RecordedMove
152
+
153
+ r = ReachyMini()
154
+
155
+ # Load move
156
+ json_path = Path(move_dir) / "sync-test.json"
157
+ wav_path = Path(move_dir) / "sync-test.wav"
158
+ move_data = json.loads(json_path.read_text())
159
+ sound_path = wav_path if wav_path.exists() else None
160
+ move = RecordedMove(move_data, sound_path=sound_path)
161
+ print(f"robot: loaded move: {move.duration:.1f}s, sound={sound_path is not None}", flush=True)
162
+
163
+ # Record present positions during playback at 50Hz
164
+ # We do this in a background thread while play_move runs
165
+ import threading
166
+
167
+ timestamps = []
168
+ left_present = []
169
+ right_present = []
170
+ recording = True
171
+
172
+ def record_positions():
173
+ t0 = time.monotonic()
174
+ while recording:
175
+ pos = r.get_present_antenna_joint_positions()
176
+ timestamps.append(time.monotonic() - t0)
177
+ left_present.append(pos[0])
178
+ right_present.append(pos[1])
179
+ time.sleep(0.02) # 50Hz
180
+
181
+ recorder = threading.Thread(target=record_positions, daemon=True)
182
+
183
+ print("robot: MARK_START", flush=True)
184
+ recorder.start()
185
+ r.play_move(move, initial_goto_duration=1.0)
186
+ recording = False
187
+ recorder.join(timeout=1.0)
188
+
189
+ print(f"robot: playback done, recorded {len(timestamps)} position samples", flush=True)
190
+
191
+ # Save results
192
+ results = {
193
+ "beep_times": move_data.get("_beep_times", []),
194
+ "collision_times": move_data.get("_collision_times", []),
195
+ "timestamps": timestamps,
196
+ "left_present": left_present,
197
+ "right_present": right_present,
198
+ }
199
+ with open(results_path, "w") as f:
200
+ json.dump(results, f)
201
+ print(f"robot: saved to {results_path}", flush=True)
202
+ print("robot: done", flush=True)
203
+ os._exit(0)
204
+ """
205
+
206
+
207
+ def start_robot(host: str) -> subprocess.Popen:
208
+ with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
209
+ f.write(ROBOT_PLAY_SCRIPT)
210
+ local_script = Path(f.name)
211
+
212
+ remote_script = "/tmp/move_sync_play.py"
213
+ try:
214
+ target = f"{ROBOT_USER}@{host}:{remote_script}"
215
+ subprocess.run(
216
+ ["scp", "-o", "ConnectTimeout=5", str(local_script), target],
217
+ capture_output=True, text=True, timeout=15, check=True,
218
+ )
219
+ finally:
220
+ local_script.unlink()
221
+
222
+ args_str = f"{ROBOT_PYTHON} {remote_script} {REMOTE_DIR} {REMOTE_RESULTS}"
223
+ proc = subprocess.Popen(
224
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{host}", args_str],
225
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
226
+ )
227
+ return proc
228
+
229
+
230
+ def plot_combined(
231
+ mic_audio, mic_sr, mic_start, mark_start,
232
+ robot_data, detected_beeps, detected_collisions, pairs,
233
+ output_path,
234
+ ):
235
+ import matplotlib
236
+ matplotlib.use("Agg")
237
+ import matplotlib.pyplot as plt
238
+
239
+ mic_t = np.arange(len(mic_audio)) / mic_sr
240
+ robot_offset = mark_start - mic_start
241
+
242
+ fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(18, 10), sharex=True)
243
+
244
+ # Mic waveform
245
+ ax1.plot(mic_t, mic_audio, "k-", linewidth=0.3, alpha=0.5)
246
+ ax1.set_ylabel("Mic amplitude")
247
+ ax1.set_title("Move Sync Test (SDK play_move) — Laptop Mic Recording")
248
+ ax1.grid(True, alpha=0.3)
249
+
250
+ for i, bt in enumerate(detected_beeps):
251
+ ax1.axvline(bt, color="blue", linestyle="-", linewidth=1.2, alpha=0.7,
252
+ label="Detected beep" if i == 0 else None)
253
+ for i, ct in enumerate(detected_collisions):
254
+ ax1.axvline(ct, color="red", linestyle="-", linewidth=1.2, alpha=0.7,
255
+ label="Detected collision" if i == 0 else None)
256
+ for p in pairs:
257
+ mid = (p["beep_mic_t"] + p["collision_mic_t"]) / 2
258
+ ax1.annotate(f'{p["interval_ms"]:.0f}ms', xy=(mid, 0),
259
+ ha="center", fontsize=9, color="purple", fontweight="bold",
260
+ bbox=dict(boxstyle="round,pad=0.2", facecolor="lightyellow", alpha=0.8))
261
+ ax1.legend(loc="upper right", fontsize=9)
262
+
263
+ # Robot trajectory
264
+ if robot_data.get("timestamps"):
265
+ robot_ts = np.array(robot_data["timestamps"])
266
+ left_pos = np.array(robot_data["left_present"])
267
+ right_pos = np.array(robot_data["right_present"])
268
+ # Note: robot timestamps start from MARK_START, which includes goto_duration
269
+ ax2.plot(robot_ts + robot_offset, left_pos, "b-", linewidth=1.5,
270
+ label="Left antenna (present)")
271
+ ax2.plot(robot_ts + robot_offset, right_pos, "r-", linewidth=1.5,
272
+ label="Right antenna (present)")
273
+
274
+ # Command times (approximate, via MARK_START)
275
+ for i, bt in enumerate(BEEP_TIMES):
276
+ # Beep times are relative to move start, not MARK_START
277
+ # play_move does a 1.0s goto first, so beeps start ~1s after MARK_START
278
+ mic_bt = robot_offset + 1.0 + bt # +1.0 for initial_goto_duration
279
+ ax2.axvline(mic_bt, color="blue", linestyle="--", linewidth=0.8, alpha=0.4,
280
+ label="Beep cmd (+goto)" if i == 0 else None)
281
+ for i, ct in enumerate(COLLISION_TIMES):
282
+ mic_ct = robot_offset + 1.0 + ct
283
+ ax2.axvline(mic_ct, color="red", linestyle="--", linewidth=0.8, alpha=0.4,
284
+ label="Collision cmd (+goto)" if i == 0 else None)
285
+
286
+ for bt in detected_beeps:
287
+ ax2.axvline(bt, color="blue", linestyle="-", linewidth=0.8, alpha=0.4)
288
+ for ct in detected_collisions:
289
+ ax2.axvline(ct, color="red", linestyle="-", linewidth=0.8, alpha=0.4)
290
+
291
+ ax2.set_xlabel("Time since mic start (s)")
292
+ ax2.set_ylabel("Position (rad)")
293
+ ax2.set_title("Robot Antenna Trajectory (aligned to mic clock)")
294
+ ax2.legend(loc="upper right", fontsize=9)
295
+ ax2.grid(True, alpha=0.3)
296
+
297
+ # Auto-zoom to active region
298
+ all_events = detected_beeps + detected_collisions
299
+ if all_events:
300
+ ax1.set_xlim(min(all_events) - 1.0, max(all_events) + 1.0)
301
+
302
+ fig.tight_layout()
303
+ fig.savefig(str(output_path), dpi=150)
304
+ plt.close(fig)
305
+ print(f" Plot saved to {output_path}")
306
+
307
+
308
+ def main():
309
+ parser = argparse.ArgumentParser(description="Move sync test via SDK play_move")
310
+ parser.add_argument("--host", default="reachy-mini.local")
311
+ args = parser.parse_args()
312
+
313
+ print(f"\n{'='*60}")
314
+ print("Move Sync Test (SDK play_move)")
315
+ print(f"{'='*60}")
316
+ print(f" Beep times: {BEEP_TIMES}")
317
+ print(f" Collision times: {COLLISION_TIMES}")
318
+ print(f" Expected interval: {BEEP_COLLISION_OFFSET:.1f}s\n")
319
+
320
+ # Step 1: Stop running apps
321
+ print("[1/6] Stopping any running app...")
322
+ subprocess.run(
323
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{args.host}",
324
+ "curl -sf -X POST http://127.0.0.1:8000/api/apps/stop-current-app >/dev/null 2>&1 || true"],
325
+ capture_output=True, timeout=10,
326
+ )
327
+ time.sleep(1)
328
+
329
+ # Step 2: Generate move files
330
+ print("[2/6] Generating Marionette-format move...")
331
+ with tempfile.TemporaryDirectory() as tmpdir:
332
+ tmpdir = Path(tmpdir)
333
+ json_path, wav_path = generate_move_files(tmpdir)
334
+
335
+ # Embed beep/collision times in JSON for the robot script to pass back
336
+ move_data = json.loads(json_path.read_text())
337
+ move_data["_beep_times"] = BEEP_TIMES
338
+ move_data["_collision_times"] = COLLISION_TIMES
339
+ json_path.write_text(json.dumps(move_data))
340
+
341
+ # Step 3: SCP to robot
342
+ print("[3/6] Copying move files to robot...")
343
+ subprocess.run(
344
+ ["ssh", "-o", "ConnectTimeout=5", f"{ROBOT_USER}@{args.host}",
345
+ f"mkdir -p {REMOTE_DIR}"],
346
+ capture_output=True, timeout=10,
347
+ )
348
+ scp_to_robot(json_path, f"{REMOTE_DIR}/sync-test.json", args.host)
349
+ scp_to_robot(wav_path, f"{REMOTE_DIR}/sync-test.wav", args.host)
350
+ print(" Done")
351
+
352
+ # Step 4: Start mic recording
353
+ print(f"[4/6] Starting mic recording ({MIC_DURATION}s)...")
354
+ mic_start = time.monotonic()
355
+ mic_data = sd.rec(
356
+ int(MIC_DURATION * LAPTOP_SR),
357
+ samplerate=LAPTOP_SR, channels=1, dtype="float32",
358
+ )
359
+
360
+ # Step 5: Start robot playback
361
+ time.sleep(0.3)
362
+ print("[5/6] Starting SDK play_move on robot...")
363
+ proc = start_robot(args.host)
364
+
365
+ mark_start = None
366
+ print("\n--- Robot output ---")
367
+ for line in iter(proc.stdout.readline, ""):
368
+ line = line.rstrip()
369
+ if not line:
370
+ continue
371
+ laptop_time = time.monotonic()
372
+ print(f" {line}")
373
+ if "MARK_START" in line:
374
+ mark_start = laptop_time
375
+ proc.wait()
376
+ print("--- End robot output ---")
377
+
378
+ sd.wait()
379
+ captured = mic_data.flatten()
380
+ print(f"\n Mic recording done")
381
+
382
+ if mark_start is None:
383
+ print("\nFAILED: Never received MARK_START")
384
+ return 1
385
+
386
+ robot_offset = mark_start - mic_start
387
+ print(f" MARK_START at mic_t={robot_offset:.3f}s")
388
+
389
+ mic_path = Path("tests/move_sync_mic.wav")
390
+ sf.write(str(mic_path), captured, LAPTOP_SR)
391
+ print(f" Saved mic to {mic_path}")
392
+
393
+ # Fetch robot position data
394
+ print("\n[6/6] Fetching robot data + analyzing...")
395
+ local_results = Path("tests/move_sync_positions.json")
396
+ scp_from_robot(REMOTE_RESULTS, local_results, args.host)
397
+ with open(local_results) as f:
398
+ robot_data = json.load(f)
399
+
400
+ # Detect beeps and collisions
401
+ detected_beeps = detect_beep_onsets(
402
+ captured, LAPTOP_SR, freq=BEEP_FREQ, bandwidth=150.0, threshold_db=-12.0,
403
+ min_separation=1.0,
404
+ )
405
+ detected_collisions = detect_transient_onsets(
406
+ captured, LAPTOP_SR, highpass_freq=3000.0,
407
+ )
408
+ print(f" Detected {len(detected_beeps)} beeps at: "
409
+ f"{[f'{t:.3f}' for t in detected_beeps]}")
410
+ print(f" Detected {len(detected_collisions)} collisions at: "
411
+ f"{[f'{t:.3f}' for t in detected_collisions]}")
412
+
413
+ # Match pairs
414
+ print(f"\n{'='*60}")
415
+ print("Beep → Collision Interval Analysis (SDK play_move)")
416
+ print(f" (Expected interval: {BEEP_COLLISION_OFFSET*1000:.0f}ms)")
417
+ print(f"{'='*60}")
418
+
419
+ pairs = []
420
+ for i, bt in enumerate(detected_beeps):
421
+ candidates = [ct for ct in detected_collisions if 0.3 < (ct - bt) < 2.0]
422
+ if not candidates:
423
+ print(f" Beep {i+1} at {bt:.3f}s: NO COLLISION FOUND")
424
+ continue
425
+ nearest = min(candidates, key=lambda ct: abs((ct - bt) - BEEP_COLLISION_OFFSET))
426
+ interval_ms = (nearest - bt) * 1000
427
+ error_ms = interval_ms - BEEP_COLLISION_OFFSET * 1000
428
+ pairs.append({
429
+ "beep_mic_t": bt,
430
+ "collision_mic_t": nearest,
431
+ "interval_ms": interval_ms,
432
+ "error_ms": error_ms,
433
+ })
434
+ print(f" Pair {len(pairs)}: beep {bt:.3f}s → collision {nearest:.3f}s = "
435
+ f"{interval_ms:.0f}ms (error {error_ms:+.0f}ms)")
436
+
437
+ if pairs:
438
+ errors = [p["error_ms"] for p in pairs]
439
+ intervals = [p["interval_ms"] for p in pairs]
440
+ print(f"\n Pairs matched: {len(pairs)}/{len(BEEP_TIMES)}")
441
+ print(f" Mean interval: {np.mean(intervals):.0f}ms (expected {BEEP_COLLISION_OFFSET*1000:.0f}ms)")
442
+ print(f" Mean error: {np.mean(errors):+.0f}ms")
443
+ print(f" Std error: {np.std(errors):.0f}ms")
444
+ print(f" Min/Max error: {min(errors):+.0f}ms / {max(errors):+.0f}ms")
445
+
446
+ plot_path = Path("tests/move_sync_plot.png")
447
+ plot_combined(
448
+ captured, LAPTOP_SR, mic_start, mark_start,
449
+ robot_data, detected_beeps, detected_collisions, pairs, plot_path,
450
+ )
451
+
452
+ success = len(pairs) >= len(BEEP_TIMES) - 1
453
+ print(f"\n{'='*60}")
454
+ if success:
455
+ print("RESULT: PASS — Beep-collision pairs detected via SDK play_move")
456
+ else:
457
+ print("RESULT: FAIL — Could not reliably detect pairs")
458
+ print(f"{'='*60}\n")
459
+
460
+ return 0 if success else 1
461
+
462
+
463
+ if __name__ == "__main__":
464
+ sys.exit(main())