Spaces:
Running on Zero
Running on Zero
docs(01-03): complete the avatar stage and transport seam plan
Browse files
.planning/ROADMAP.md
CHANGED
|
@@ -35,7 +35,7 @@ Decimal phases appear between their surrounding integers in numeric order.
|
|
| 35 |
- [x] 01-02-PLAN.md β Toolchain, LFS arming, package skeleton, test scaffolding (local-only) [wave 1]
|
| 36 |
- [x] 01-01-PLAN.md β Hosting decision + Space creation + VRM sourcing (user-gated) [wave 2]
|
| 37 |
- [x] 01-04-PLAN.md β VOICEVOX TTS, requirements.txt, ground-truth AudioQuery fixtures [wave 2]
|
| 38 |
-
- [
|
| 39 |
- [ ] 01-05-PLAN.md β Space manifest, deploy the spike, first deployed E2E, verdict (user-gated) [wave 4]
|
| 40 |
- [ ] 01-06-PLAN.md β Mora-to-viseme timeline builder, test-first [wave 4]
|
| 41 |
- [ ] 01-07-PLAN.md β Push-to-talk mic gate + tiered browser ASR [wave 4]
|
|
|
|
| 35 |
- [x] 01-02-PLAN.md β Toolchain, LFS arming, package skeleton, test scaffolding (local-only) [wave 1]
|
| 36 |
- [x] 01-01-PLAN.md β Hosting decision + Space creation + VRM sourcing (user-gated) [wave 2]
|
| 37 |
- [x] 01-04-PLAN.md β VOICEVOX TTS, requirements.txt, ground-truth AudioQuery fixtures [wave 2]
|
| 38 |
+
- [x] 01-03-PLAN.md β Avatar stage, shared facade + turn loop, both transports [wave 3]
|
| 39 |
- [ ] 01-05-PLAN.md β Space manifest, deploy the spike, first deployed E2E, verdict (user-gated) [wave 4]
|
| 40 |
- [ ] 01-06-PLAN.md β Mora-to-viseme timeline builder, test-first [wave 4]
|
| 41 |
- [ ] 01-07-PLAN.md β Push-to-talk mic gate + tiered browser ASR [wave 4]
|
.planning/STATE.md
CHANGED
|
@@ -3,13 +3,13 @@ gsd_state_version: 1.0
|
|
| 3 |
milestone: v1.0
|
| 4 |
milestone_name: milestone
|
| 5 |
status: executing
|
| 6 |
-
stopped_at: 01-03
|
| 7 |
-
last_updated: "2026-08-
|
| 8 |
progress:
|
| 9 |
total_phases: 6
|
| 10 |
completed_phases: 0
|
| 11 |
total_plans: 10
|
| 12 |
-
completed_plans:
|
| 13 |
---
|
| 14 |
|
| 15 |
# Project State
|
|
@@ -24,26 +24,28 @@ See: .planning/PROJECT.md (updated 2026-08-08)
|
|
| 24 |
## Current Position
|
| 25 |
|
| 26 |
Phase: 01 (voice-avatar-loop-skeleton) β EXECUTING
|
| 27 |
-
Plan:
|
| 28 |
|
| 29 |
-
> **01-03 IS
|
| 30 |
>
|
| 31 |
-
> | Task | Deliverable |
|
| 32 |
-
> |------|-------------|-------|
|
| 33 |
-
> | 1 | `vrm-stage.js`, `lipsync.js`, `audio-queue.js` |
|
| 34 |
-
> | 2 | `facade.js`, `turn-loop.js` |
|
| 35 |
-
> | 3 | `avatar.js`, `avatar-iframe.js`, `stage.html`, `demo-konnichiwa.wav`, `avatar_component.py`, `app.py` |
|
| 36 |
-
> | 4 | `
|
| 37 |
>
|
| 38 |
-
> **
|
| 39 |
>
|
| 40 |
-
> **Two
|
| 41 |
-
> 1. **`
|
| 42 |
-
> 2. **
|
| 43 |
>
|
| 44 |
-
>
|
| 45 |
>
|
| 46 |
-
> **
|
|
|
|
|
|
|
| 47 |
|
| 48 |
## Performance Metrics
|
| 49 |
|
|
@@ -73,6 +75,7 @@ Plan: 4 of 10 (3 complete, 01-03 in flight at Task 3 of 4)
|
|
| 73 |
- Trend: 01-01 was fast because its two expensive tasks were human checkpoints answered outside the executor; the executor's own work was verification and recording
|
| 74 |
|
| 75 |
*Updated after each plan completion*
|
|
|
|
| 76 |
|
| 77 |
## Accumulated Context
|
| 78 |
|
|
@@ -95,6 +98,10 @@ Recent decisions affecting current work:
|
|
| 95 |
- [Phase 01]: 1 of 2 free ZeroGPU slots consumed by WolfDavid/japanese-learning-avatar; the remaining slot is the last free Gradio Space this account can create without PRO.
|
| 96 |
- [Phase 01]: tutor.vrm is VRM1_Constraint_Twist_Sample (pixiv Inc., VRM 1.0) adopted as a TEMPORARY dev asset; embedded VRMC_vrm.meta grants redistribution/avatar-use/modification with creditNotation unnecessary. Plan 01-10 must re-raise the sourcing gate.
|
| 97 |
- [Phase 01]: Audit Space hardware via runtime.hardware.requested, not .current β .current is null for every SLEEPING Space, which made six of eight look unallocated.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
### Pending Todos
|
| 100 |
|
|
@@ -113,8 +120,8 @@ None yet.
|
|
| 113 |
|
| 114 |
## Session Continuity
|
| 115 |
|
| 116 |
-
Last session: 2026-08-
|
| 117 |
-
Stopped at: 01-03
|
| 118 |
-
Resume file: None
|
| 119 |
|
| 120 |
-
**Do not push to the `space` remote yet.** It is the only remote this repo has, and pushing triggers a Space rebuild. `app.py`
|
|
|
|
| 3 |
milestone: v1.0
|
| 4 |
milestone_name: milestone
|
| 5 |
status: executing
|
| 6 |
+
stopped_at: Completed 01-03-PLAN.md (Tasks 3-4 executed after crash recovery); nothing pushed to the space remote
|
| 7 |
+
last_updated: "2026-08-27T12:09:36.232Z"
|
| 8 |
progress:
|
| 9 |
total_phases: 6
|
| 10 |
completed_phases: 0
|
| 11 |
total_plans: 10
|
| 12 |
+
completed_plans: 4
|
| 13 |
---
|
| 14 |
|
| 15 |
# Project State
|
|
|
|
| 24 |
## Current Position
|
| 25 |
|
| 26 |
Phase: 01 (voice-avatar-loop-skeleton) β EXECUTING
|
| 27 |
+
Plan: 5 of 10 (4 complete: 01-01, 01-02, 01-03, 01-04)
|
| 28 |
|
| 29 |
+
> **01-03 IS COMPLETE.** The crash-interrupted plan was resumed at Task 3 on 2026-08-27 and both remaining tasks executed and committed. Full detail in `01-03-SUMMARY.md`; the highlights that change what later plans should assume are below.
|
| 30 |
>
|
| 31 |
+
> | Task | Deliverable | Commit |
|
| 32 |
+
> |------|-------------|--------|
|
| 33 |
+
> | 1 | `vrm-stage.js`, `lipsync.js`, `audio-queue.js` | `94a5b15` |
|
| 34 |
+
> | 2 | `facade.js`, `turn-loop.js` | `7146e75` |
|
| 35 |
+
> | 3 | `avatar.js`, `avatar-iframe.js`, `stage.html`, `demo-konnichiwa.wav`, `avatar_component.py`, `app.py` | `f822421` |
|
| 36 |
+
> | 4 | `test_transport_seam.py`, `test_stage_standalone.py`, `test_facade_parity.py` | `d6ea280` |
|
| 37 |
>
|
| 38 |
+
> **The Gradio spike works locally, which materially de-risks plan 01-05.** three.js + `@pixiv/three-vrm` render inside a real Gradio 6.22.0 `gr.HTML` component with `threeInstanceCount === 1` and `mountCount === 1`, and the iframe fallback boots identically under `AVATAR_TRANSPORT=iframe`. 43 tests green: 36 static seam guards, 4 standalone browser tests, 3 live parity tests. Both transports expose byte-identical method sets and `__debug` keys.
|
| 39 |
>
|
| 40 |
+
> **Two Gradio 6 API traps were found the hard way β `01-RESEARCH.md`'s `gr.HTML` section is NOT reliable on these two points. Copy from `src/japanese_avatar/ui/avatar_component.py`, not from research:**
|
| 41 |
+
> 1. **`js_on_load` cannot use top-level `await`.** Gradio 6.22.0 compiles it with a plain, non-async `Function('element','trigger','props','server','upload','watch', body)`. The documented snippet throws `SyntaxError` and the component silently never boots. Wrap the body in an async IIFE with its own `.catch`.
|
| 42 |
+
> 2. **Custom props are `**kwargs`, not a `props=` dict.** `gr.HTML(props={"vrmUrl": ...})` creates one prop literally named `props`, so `props.vrmUrl` is `undefined` in the browser and `GLTFLoader` throws inside `extractUrlBase`. Pass `vrmUrl=...` directly.
|
| 43 |
>
|
| 44 |
+
> **Recorded for plan 01-09:** observed `vrmMetaTitle` is `VRM1_Constraint_Twist_Sample`, VRM spec version `1` (so no facing rotation is applied and licence fields live under `meta.name`), `threeInstanceCount` is `1`. Standalone VRM load: 607 ms cold context / ~335 ms warm on loopback.
|
| 45 |
>
|
| 46 |
+
> **AVTR-01 was deliberately NOT marked complete** even though it is in 01-03's `requirements:` frontmatter. Its acceptance tests (`tests/e2e/test_avatar_loop.py::test_vrm_ready` / `::test_idle_life` / `::test_no_remount`) belong to plan 01-09 and run against the deployed Space; they do not exist yet. This follows the standing warning below about over-claiming plan frontmatter.
|
| 47 |
+
>
|
| 48 |
+
> **Still do not push to the `space` remote.** It is the only remote this repo has and pushing triggers a Space rebuild. `app.py` now exists, but deployment (and the Space README front-matter) is plan 01-05's job.
|
| 49 |
|
| 50 |
## Performance Metrics
|
| 51 |
|
|
|
|
| 75 |
- Trend: 01-01 was fast because its two expensive tasks were human checkpoints answered outside the executor; the executor's own work was verification and recording
|
| 76 |
|
| 77 |
*Updated after each plan completion*
|
| 78 |
+
| Phase 01 P03 | 62 | 4 tasks | 20 files |
|
| 79 |
|
| 80 |
## Accumulated Context
|
| 81 |
|
|
|
|
| 98 |
- [Phase 01]: 1 of 2 free ZeroGPU slots consumed by WolfDavid/japanese-learning-avatar; the remaining slot is the last free Gradio Space this account can create without PRO.
|
| 99 |
- [Phase 01]: tutor.vrm is VRM1_Constraint_Twist_Sample (pixiv Inc., VRM 1.0) adopted as a TEMPORARY dev asset; embedded VRMC_vrm.meta grants redistribution/avatar-use/modification with creditNotation unnecessary. Plan 01-10 must re-raise the sourcing gate.
|
| 100 |
- [Phase 01]: Audit Space hardware via runtime.hardware.requested, not .current β .current is null for every SLEEPING Space, which made six of eight look unallocated.
|
| 101 |
+
- [Phase 01]: gr.HTML js_on_load must be wrapped in an async IIFE: Gradio 6.22.0 compiles it with a plain non-async Function(), so the documented top-level await is a SyntaxError and the component silently never boots
|
| 102 |
+
- [Phase 01]: Custom gr.HTML props are **kwargs, not a props= dict; the dict form creates one prop literally named 'props' and props.vrmUrl arrives undefined in the browser
|
| 103 |
+
- [Phase 01]: Gradio's auto-generated .pyi component stub is build output: gitignored and ruff-excluded, because create_or_modify_pyi runs unconditionally with no opt-out
|
| 104 |
+
- [Phase 01]: Browser assertions sample in-page at animation-frame rate; a 250 ms Python poll cannot reliably observe a 120 ms blink and would be flaky by construction
|
| 105 |
|
| 106 |
### Pending Todos
|
| 107 |
|
|
|
|
| 120 |
|
| 121 |
## Session Continuity
|
| 122 |
|
| 123 |
+
Last session: 2026-08-27T12:09:28.441Z
|
| 124 |
+
Stopped at: Completed 01-03-PLAN.md (Tasks 3-4 executed after crash recovery); nothing pushed to the space remote
|
| 125 |
+
Resume file: None
|
| 126 |
|
| 127 |
+
**Do not push to the `space` remote yet.** It is the only remote this repo has, and pushing triggers a Space rebuild. `app.py` now exists (01-03), but the Space README front-matter and the deploy itself are owned by plan 01-05. The Space correctly serves 503 (`NO_APP_FILE`) until then. Nothing from 01-03 was pushed.
|
.planning/phases/01-voice-avatar-loop-skeleton/01-03-SUMMARY.md
ADDED
|
@@ -0,0 +1,277 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
phase: 01-voice-avatar-loop-skeleton
|
| 3 |
+
plan: 03
|
| 4 |
+
subsystem: ui
|
| 5 |
+
tags: [three.js, three-vrm, vrm, webaudio, gradio, playwright, postmessage, esm]
|
| 6 |
+
|
| 7 |
+
# Dependency graph
|
| 8 |
+
requires:
|
| 9 |
+
- phase: 01-01
|
| 10 |
+
provides: avatar/assets/tutor.vrm (VRM 1.0, Git LFS) and its recorded licence facts
|
| 11 |
+
- phase: 01-02
|
| 12 |
+
provides: pyproject pytest/ruff config, tests/conftest.py, marker conventions
|
| 13 |
+
provides:
|
| 14 |
+
- "avatar/vrm-stage.js β three.js scene, single-instance module load, VRM mount, idle blink/breathe/sway, numeric __debug"
|
| 15 |
+
- "avatar/lipsync.js β AudioContext-clocked viseme player, no timers, no amplitude analysis"
|
| 16 |
+
- "avatar/audio-queue.js β WebAudio decode/schedule, speech-start/speech-end off the SCHEDULED start"
|
| 17 |
+
- "avatar/facade.js β AVATAR_SURFACE + installFacade, the only assignment of window.Avatar"
|
| 18 |
+
- "avatar/turn-loop.js β the only turn implementation, shared by both transports"
|
| 19 |
+
- "avatar/avatar.js β inline gr.HTML transport (145 lines, plumbing only)"
|
| 20 |
+
- "avatar/avatar-iframe.js β postMessage transport (155 lines, plumbing only)"
|
| 21 |
+
- "avatar/stage.html β iframe target AND standalone debug harness with ?demo=1"
|
| 22 |
+
- "avatar/assets/demo-konnichiwa.wav β 24000 Hz mono 16-bit, 1.129 s canned utterance"
|
| 23 |
+
- "src/japanese_avatar/ui/avatar_component.py β VrmStage, AVATAR_TRANSPORT read in one place"
|
| 24 |
+
- "app.py β Blocks assembly, gr.set_static_paths([\"avatar\"]), DISABLE_GPU switch"
|
| 25 |
+
- "tests/test_transport_seam.py β 36 static seam guards"
|
| 26 |
+
- "tests/e2e/test_stage_standalone.py β 4 browser tests against stage.html"
|
| 27 |
+
- "tests/e2e/test_facade_parity.py β 3 live-object parity tests across both transports"
|
| 28 |
+
affects: [01-05 deployment spike, 01-06 viseme timeline, 01-07 mic and ASR, 01-08 turn loop, 01-09 avatar loop E2E]
|
| 29 |
+
|
| 30 |
+
# Tech tracking
|
| 31 |
+
tech-stack:
|
| 32 |
+
added: [three@0.185.1 via esm.sh, "@pixiv/three-vrm@3.5.5 via esm.sh ?deps= pinning", playwright chromium]
|
| 33 |
+
patterns:
|
| 34 |
+
- "stagePort: the six-method renderer boundary; the ONLY thing that differs between transports"
|
| 35 |
+
- "AVATAR_SURFACE as data β the facade is built by iterating it and throws at boot on an unresolved name"
|
| 36 |
+
- "deferred event bus in each transport so events emitted during mount survive until installFacade runs"
|
| 37 |
+
- "in-page frame-rate sampling for browser assertions instead of Python-side polling"
|
| 38 |
+
|
| 39 |
+
key-files:
|
| 40 |
+
created:
|
| 41 |
+
- avatar/avatar.js
|
| 42 |
+
- avatar/avatar-iframe.js
|
| 43 |
+
- avatar/stage.html
|
| 44 |
+
- avatar/assets/demo-konnichiwa.wav
|
| 45 |
+
- src/japanese_avatar/ui/avatar_component.py
|
| 46 |
+
- app.py
|
| 47 |
+
- tests/test_transport_seam.py
|
| 48 |
+
- tests/e2e/conftest.py
|
| 49 |
+
- tests/e2e/test_stage_standalone.py
|
| 50 |
+
- tests/e2e/test_facade_parity.py
|
| 51 |
+
modified:
|
| 52 |
+
- avatar/vrm-stage.js
|
| 53 |
+
- .gitignore
|
| 54 |
+
- pyproject.toml
|
| 55 |
+
|
| 56 |
+
key-decisions:
|
| 57 |
+
- "gr.HTML js_on_load must be wrapped in an async IIFE: Gradio 6.22.0 compiles it with a plain non-async Function(), so the documented top-level await is a SyntaxError and the component silently never boots"
|
| 58 |
+
- "Custom gr.HTML props are **kwargs, not a props= dict; the dict form creates one prop literally named 'props' and props.vrmUrl arrives undefined"
|
| 59 |
+
- "Gradio's auto-generated .pyi component stub is build output: gitignored and excluded from ruff, because create_or_modify_pyi runs unconditionally with no opt-out"
|
| 60 |
+
- "Browser assertions sample in-page at animation-frame rate; a 250 ms poll cannot reliably observe a 120 ms blink"
|
| 61 |
+
- "The host-token seam guards are case-insensitive so a capitalised mention cannot pass by accident"
|
| 62 |
+
|
| 63 |
+
patterns-established:
|
| 64 |
+
- "One facade, one turn loop, N transports: a transport is message plumbing, never behaviour"
|
| 65 |
+
- "Every visual property is published as a number in __debug; the canvas is never screenshotted"
|
| 66 |
+
- "Deferred methods throw an error naming the plan that implements them, so a premature call fails loudly"
|
| 67 |
+
|
| 68 |
+
requirements-completed: []
|
| 69 |
+
|
| 70 |
+
# Metrics
|
| 71 |
+
duration: 62min
|
| 72 |
+
completed: 2026-08-27
|
| 73 |
+
---
|
| 74 |
+
|
| 75 |
+
# Phase 01 Plan 03: Avatar Stage and Transport Seam Summary
|
| 76 |
+
|
| 77 |
+
**A VRM 1.0 avatar that blinks, breathes, sways and lip-syncs a canned Japanese utterance off the AudioContext clock, behind a single `window.Avatar` facade that two independently-booting transports are proven β in a running browser β to expose identically.**
|
| 78 |
+
|
| 79 |
+
## Performance
|
| 80 |
+
|
| 81 |
+
- **Duration:** ~62 min for Tasks 3-4 (this session); Tasks 1-2 were executed before a session crash
|
| 82 |
+
- **Started:** 2026-08-27T07:45:00Z (resume at Task 3)
|
| 83 |
+
- **Completed:** 2026-08-27T08:47:00Z
|
| 84 |
+
- **Tasks:** 4 of 4 (Tasks 1-2 pre-existing and committed; Tasks 3-4 executed here)
|
| 85 |
+
- **Files modified:** 20 across four task commits
|
| 86 |
+
|
| 87 |
+
## Required Measurements
|
| 88 |
+
|
| 89 |
+
| Measurement | Observed value |
|
| 90 |
+
|---|---|
|
| 91 |
+
| `threeInstanceCount` | **1** β in the standalone harness, in the inline Gradio app, and in the iframe Gradio app. `performance.getEntriesByType('resource')` also reports exactly **one** `three.mjs` entry, and no `Multiple instances of Three.js` console warning appeared in any run. |
|
| 92 |
+
| `vrmMetaTitle` | **`VRM1_Constraint_Twist_Sample`** β plan 01-09's `test_vrm_meta_matches_licenses` compares against this string. It matches `docs/ASSETS.md` `meta.name` exactly. |
|
| 93 |
+
| VRM spec version detected | **VRM 1.0** β `vrm.meta.metaVersion === '1'`, surfaced as `__debug.vrmSpecVersion`. Confirms `docs/ASSETS.md` (`extensions.VRMC_vrm.specVersion = 1.0`). Consequence: **no facing rotation is applied** (`vrm.scene.rotation.y = Math.PI` is the VRM 0.0 branch only), and the licence fields live under `meta.name`, not `meta.title`. |
|
| 94 |
+
| VRM load time, standalone harness | Navigation β `__stageDebug.ready === true`: **607 ms** on a cold browser context, **334 ms / 338 ms** on the two following contexts. The 10.28 MiB `tutor.vrm` fetch itself was **33-56 ms** (served from `python -m http.server` on loopback), and the single `three.mjs` fetch **34-37 ms** from esm.sh. Loopback + a warm browser HTTP cache for the CDN modules β the deployed number is 01-05's to measure, and the E2E `READY_TIMEOUT_MS` is set to 30 s to leave room for it. |
|
| 95 |
+
| `avatar/avatar.js` line count | **145** (budget: under 150) |
|
| 96 |
+
| `avatar/avatar-iframe.js` line count | **155** (budget: under 180) |
|
| 97 |
+
| Deliberate seam break | **Confirmed.** Appending `window.Avatar = {};` to `avatar/avatar-iframe.js` made `test_only_facade_assigns_window_avatar` fail with `AssertionError: avatar-iframe.js assigns window.Avatar; only facade.js may` and `assert not <re.Match object; span=(5384, 5399), match='window.Avatar ='>`. The file was reverted with `git checkout --` and the full 36-case suite is green again. |
|
| 98 |
+
|
| 99 |
+
## Accomplishments
|
| 100 |
+
|
| 101 |
+
- **The stage core works in a real browser with no Python at all.** `avatar/stage.html?demo=1` loads the VRM, runs blink/breathe/sway, and lip-syncs `γγγ«γ‘γ―` against a hand-authored placeholder timeline. Observed viseme peaks: `aa = 1.0`, `ih = 1.0`, `oh = 1.0`, with `ou` and `ee` correctly at `0` (γγγ«γ‘γ― contains no `u` or `e` vowel), and all five back to `0` after `speech-end`.
|
| 102 |
+
- **The Wave 0 spike question is now substantially answered locally, and the answer is positive.** three.js + `@pixiv/three-vrm` load and render inside a Gradio 6.22.0 `gr.HTML` component via dynamic `import()` from `/gradio_api/file=avatar/...`, with `threeInstanceCount === 1` and `mountCount === 1`. `01-RESEARCH.md` recorded that nobody had publicly done this; it now works on this machine. Plan 01-05 still owns the deployed verdict.
|
| 103 |
+
- **The iframe fallback is not a plan, it is a running program.** Both transports boot the same app under a single env var and expose byte-identical surfaces, proven mechanically rather than asserted in prose.
|
| 104 |
+
- **Two real Gradio 6 API defects were found and fixed** (see Deviations) β both would have made the deployed spike fail in ways that look like "three.js does not work in Gradio", which is precisely the wrong conclusion.
|
| 105 |
+
|
| 106 |
+
## Task Commits
|
| 107 |
+
|
| 108 |
+
1. **Task 1: Gradio-free stage core** β `94a5b15` (feat) *(pre-existing; committed before the crash)*
|
| 109 |
+
2. **Task 2: The single facade and the single turn loop** β `7146e75` (feat) *(pre-existing; recovered and committed at resume)*
|
| 110 |
+
3. **Task 3: Two thin transports, the standalone host page, and the Gradio component** β `f822421` (feat)
|
| 111 |
+
4. **Task 4: Enforce the seam β static guards, standalone proof, mechanical parity** β `d6ea280` (test)
|
| 112 |
+
|
| 113 |
+
## Files Created/Modified
|
| 114 |
+
|
| 115 |
+
- `avatar/avatar.js` β inline transport: builds a direct stagePort, memoised single mount, deferred emit, delegates everything else
|
| 116 |
+
- `avatar/avatar-iframe.js` β iframe transport: same shape, six postMessage round trips, id-correlated replies with a 60 s bound
|
| 117 |
+
- `avatar/stage.html` β standalone harness and iframe target; `DEMO_TIMELINE`, `window.__stageDebug`, `window.__stageEvents`, the full postMessage protocol
|
| 118 |
+
- `avatar/assets/demo-konnichiwa.wav` β 24000 Hz mono 16-bit, 27096 frames, 1.129000 s; one 220 Hz raised-cosine burst per voiced mora at the `DEMO_TIMELINE` offsets (Git LFS)
|
| 119 |
+
- `avatar/vrm-stage.js` β reworded the import-map comment so the stage core names no host framework
|
| 120 |
+
- `src/japanese_avatar/ui/avatar_component.py` β `VrmStage`, `AVATAR_TRANSPORT`, one shared `_BOOT_JS` template
|
| 121 |
+
- `app.py` β `gr.set_static_paths(["avatar"])`, `build_app()`, `gpu_disabled()` for SC-4
|
| 122 |
+
- `tests/test_transport_seam.py` β 36 static cases
|
| 123 |
+
- `tests/e2e/conftest.py` β static-file server, per-transport Gradio subprocess, Chromium autoplay flag
|
| 124 |
+
- `tests/e2e/test_stage_standalone.py` β 4 tests
|
| 125 |
+
- `tests/e2e/test_facade_parity.py` β 3 tests
|
| 126 |
+
- `.gitignore`, `pyproject.toml` β exclude Gradio's generated `.pyi` stub from git and from the lint gate
|
| 127 |
+
|
| 128 |
+
## Decisions Made
|
| 129 |
+
|
| 130 |
+
- **`js_on_load` bodies are wrapped in an async IIFE with its own `.catch`.** The IIFE escapes Gradio's internal `try/catch`, so a boot failure has to report itself; that console line is what plan 01-05 will read.
|
| 131 |
+
- **The demo WAV is a synthetic tone burst per voiced mora, not synthesised speech.** It only has to be audible and correctly timed; plan 01-06 replaces both the audio and the timeline from a real `AudioQuery`. The generator snippet is recorded below so the asset is reproducible.
|
| 132 |
+
- **`window.__stageEvents` was added alongside `window.__stageDebug`.** Opened standalone, the stage has no parent to post to, so the event stream would be unobservable and `test_demo_timeline_drives_visemes` would have to guess at timing instead of waiting for `speech-end`.
|
| 133 |
+
- **Host-token guards are case-insensitive.** The state file flagged that `vrm-stage.js` passed only because the offending word was capitalised. The comment is reworded *and* the guard is now case-insensitive, so the pass is honest rather than lucky.
|
| 134 |
+
|
| 135 |
+
## Deviations from Plan
|
| 136 |
+
|
| 137 |
+
### Auto-fixed Issues
|
| 138 |
+
|
| 139 |
+
**1. [Rule 1 - Bug] `js_on_load` top-level `await` is a SyntaxError in Gradio 6.22.0**
|
| 140 |
+
|
| 141 |
+
- **Found during:** Task 4 (writing the parity suite, which boots the real app)
|
| 142 |
+
- **Issue:** The plan's `_INLINE_JS` / `_IFRAME_JS` (taken from `01-RESEARCH.md`, which took them from Gradio's docs) use top-level `await`. Gradio 6.22.0's HTML component compiles the string with `Function('element','trigger','props','server','upload','watch', js_on_load)` β a plain, non-async function. The browser console showed `Error executing js_on_load: SyntaxError: await is only valid in async functions and the top level bodies of modules`, and `window.Avatar` never existed. Verified by reading `gradio/templates/frontend/assets/HTML-DtO81iEW.js`.
|
| 143 |
+
- **Fix:** One shared `_BOOT_JS` template wrapping the body in `(async () => { ... })().catch(err => console.error('avatar boot failed:', err))`.
|
| 144 |
+
- **Files modified:** `src/japanese_avatar/ui/avatar_component.py`
|
| 145 |
+
- **Verification:** `window.Avatar` present and `__debug.ready` true under both transports; `test_facade_parity.py` 3 passed.
|
| 146 |
+
- **Committed in:** `d6ea280`
|
| 147 |
+
|
| 148 |
+
**2. [Rule 1 - Bug] Custom `gr.HTML` props are `**kwargs`, not a `props=` dict**
|
| 149 |
+
|
| 150 |
+
- **Found during:** Task 4 (same run, immediately after fix 1)
|
| 151 |
+
- **Issue:** The plan passes `props={"vrmUrl": VRM_URL, "transport": AVATAR_TRANSPORT}`. `gradio/components/html.py` declares `**props: Any`, so that call creates a single prop literally named `props`. In the browser `props.vrmUrl` was `undefined`, which reached `GLTFLoader.load` and threw `TypeError: Cannot read properties of undefined (reading 'lastIndexOf')` inside `extractUrlBase`. `01-RESEARCH.md`'s "verified signature" line `props: Any = {}` is a misreading of `**props`.
|
| 152 |
+
- **Fix:** Pass `vrmUrl=VRM_URL, transport=AVATAR_TRANSPORT` as keyword arguments, with a comment recording the trap.
|
| 153 |
+
- **Files modified:** `src/japanese_avatar/ui/avatar_component.py`
|
| 154 |
+
- **Verification:** VRM fetched (`RESP 200 /gradio_api/file=avatar/assets/tutor.vrm`) and `__debug.ready` true under both transports.
|
| 155 |
+
- **Committed in:** `d6ea280`
|
| 156 |
+
|
| 157 |
+
**3. [Rule 3 - Blocking] Gradio writes an un-lintable `.pyi` stub next to any component subclass**
|
| 158 |
+
|
| 159 |
+
- **Found during:** Task 3 (first `ruff check .` after creating `VrmStage`)
|
| 160 |
+
- **Issue:** `gradio/component_meta.py` calls `create_or_modify_pyi()` unconditionally at class creation β no env switch, no opt-out. It generated `src/japanese_avatar/ui/avatar_component.pyi`, which fails `E402`, `F401` and `UP035`. The lint gate is a plan verification command, so this blocked the task, and the file regenerates on every import.
|
| 161 |
+
- **Fix:** `*.pyi` added to `.gitignore` and to ruff's `extend-exclude`, each with a comment explaining why.
|
| 162 |
+
- **Files modified:** `.gitignore`, `pyproject.toml`
|
| 163 |
+
- **Verification:** `uv run ruff check . && uv run ruff format --check .` exits 0; `git status` clean after a test run.
|
| 164 |
+
- **Committed in:** `f822421`
|
| 165 |
+
|
| 166 |
+
**4. [Rule 1 - Bug] The 250 ms idle-life poll the plan specifies is flaky by construction**
|
| 167 |
+
|
| 168 |
+
- **Found during:** Task 4 (writing `test_idle_life_values_change`)
|
| 169 |
+
- **Issue:** The plan says sample `{blinkValue, breathValue}` every 250 ms for 12 s and assert `max(blink_samples) > 0.5`. A blink is a 120 ms triangular ramp on a 1.8-5.8 s schedule, so the window where `blinkValue > 0.5` is roughly 60 ms per blink β under 2% of the wall clock. 48 polls over 12 s would land inside it about 0.7 times on average: the test would fail on most runs for a perfectly healthy avatar.
|
| 170 |
+
- **Fix:** The sampler runs *inside the page* on `requestAnimationFrame` for the same 12 s window, returning the full series to Python. Same measurement, same assertions, ~60Γ the sampling rate, no flakiness. The same technique is used for viseme peaks in `test_demo_timeline_drives_visemes`.
|
| 171 |
+
- **Files modified:** `tests/e2e/test_stage_standalone.py`
|
| 172 |
+
- **Verification:** Suite run repeatedly, 4 passed each time; observed blink peak 0.868, 69 distinct breath values in a 3 s window alone.
|
| 173 |
+
- **Committed in:** `d6ea280`
|
| 174 |
+
|
| 175 |
+
**5. [Rule 2 - Missing Critical] `window.__stageEvents` added to the standalone harness**
|
| 176 |
+
|
| 177 |
+
- **Found during:** Task 4
|
| 178 |
+
- **Issue:** `stage.html` emits `speech-start` / `speech-end` / `error` by `postMessage` to its parent. Opened standalone there is no parent, so the entire event stream is unobservable and a browser test can only guess when the utterance ended.
|
| 179 |
+
- **Fix:** `emit()` also appends `{event, data, at}` to `window.__stageEvents`, mirroring the existing `window.__stageDebug` convention.
|
| 180 |
+
- **Files modified:** `avatar/stage.html`
|
| 181 |
+
- **Verification:** `test_demo_timeline_drives_visemes` waits on `speech-end` and asserts the mouth shut afterwards; passes.
|
| 182 |
+
- **Committed in:** `d6ea280`
|
| 183 |
+
|
| 184 |
+
**6. [Rule 2 - Missing Critical] Deferred event bus in both transports**
|
| 185 |
+
|
| 186 |
+
- **Found during:** Task 3 (flagged in advance by STATE.md, not rediscovered)
|
| 187 |
+
- **Issue:** The plan's `boot` calls `mountStage(canvas, props.vrmUrl, emit)` at step 2 but `emit` only exists after `installFacade` returns it at step 5. Events emitted during mount β notably the `error` raised when a VRM has no `expressionManager`, the exact symptom of a double three.js instance β would be dropped.
|
| 188 |
+
- **Fix:** A ~12-line `deferredBus()` in each transport buffers `(name, data)` pairs and replays them in order into the real bus at `emit.connect(installed.emit)`. Duplicated rather than shared because the alternative was editing a frozen Task 2 file; it is plumbing, not behaviour, so it is outside what the seam guards protect.
|
| 189 |
+
- **Files modified:** `avatar/avatar.js`, `avatar/avatar-iframe.js`
|
| 190 |
+
- **Verification:** `test_transports_only_do_plumbing` passes for both files; both transports boot green.
|
| 191 |
+
- **Committed in:** `f822421`
|
| 192 |
+
|
| 193 |
+
**7. [Rule 3 - Blocking] `test_transport_seam.py` resolves `avatar/` from `__file__`, not the cwd**
|
| 194 |
+
|
| 195 |
+
- **Found during:** Task 4
|
| 196 |
+
- **Issue:** The plan's code block uses `AVATAR = Path("avatar")`, which silently makes every assertion a function of the process working directory.
|
| 197 |
+
- **Fix:** `AVATAR = Path(__file__).resolve().parent.parent / "avatar"`. Every assertion is otherwise verbatim from the plan.
|
| 198 |
+
- **Files modified:** `tests/test_transport_seam.py`
|
| 199 |
+
- **Verification:** 36 passed from the repo root.
|
| 200 |
+
- **Committed in:** `d6ea280`
|
| 201 |
+
|
| 202 |
+
**8. [Rule 3 - Blocking] `tests/e2e/conftest.py` created (not in `files_modified`)**
|
| 203 |
+
|
| 204 |
+
- **Found during:** Task 4
|
| 205 |
+
- **Issue:** Both browser suites need a static-file server, a per-transport Gradio subprocess and a Chromium launched with `--autoplay-policy=no-user-gesture-required`. Without the autoplay flag an ungestured `AudioContext` stays suspended, `currentTime` never advances, and the lip-sync test would be measuring Chromium's autoplay policy rather than the player.
|
| 206 |
+
- **Fix:** One `tests/e2e/conftest.py` holding those three fixtures, following plan 01-02's conventions.
|
| 207 |
+
- **Files modified:** `tests/e2e/conftest.py`
|
| 208 |
+
- **Verification:** Both suites green.
|
| 209 |
+
- **Committed in:** `d6ea280`
|
| 210 |
+
|
| 211 |
+
### Acceptance criteria not satisfiable as literally written
|
| 212 |
+
|
| 213 |
+
Recorded rather than fixed, because satisfying the literal text would make the code worse:
|
| 214 |
+
|
| 215 |
+
- **Task 2's `grep -ci "gradio\|element\|props\|THREE" facade.js turn-loop.js` returning 0** is impossible: `threeInstanceCount` is a mandated `__debug` key and contains `three`. The enforced test (`test_shared_modules_are_gradio_free`) checks `gradio`, `gradio_api`, `server.` and `trigger(`, all genuinely absent from both files, and now case-insensitively. This was pre-flagged in STATE.md.
|
| 216 |
+
- **`grep -c "installFacade" avatar/avatar.js` returning exactly 1** β returns 4 here, because the module header and JSDoc name it. The enforced test asserts presence, not count. Same for `grep -c 'key="vrm-stage"'` in `avatar_component.py` returning 2 rather than 1, which the plan's own code block also does (the docstring names it).
|
| 217 |
+
|
| 218 |
+
---
|
| 219 |
+
|
| 220 |
+
**Total deviations:** 8 auto-fixed (4 bugs, 2 missing-critical, 2 blocking) + 2 unsatisfiable criteria recorded.
|
| 221 |
+
**Impact on plan:** No scope creep. Deviations 1 and 2 are the substantive ones: without them the component never boots and the spike would have been read as "three.js does not work in Gradio 6". Everything else is test robustness or lint plumbing.
|
| 222 |
+
|
| 223 |
+
## Issues Encountered
|
| 224 |
+
|
| 225 |
+
- **The published Gradio `gr.HTML` recipe does not run on 6.22.0.** Both the top-level `await` and the `props=` dict come straight from the docs/research and both are wrong for this version. The fix required reading Gradio's compiled frontend bundle and `components/html.py` directly. Recorded in code comments at both sites so a future editor cannot re-introduce either.
|
| 226 |
+
- **`three-vrm` emits two deprecation warnings on load** β `VRMUtils.removeUnnecessaryJoints is deprecated` (we already call `combineSkeletons` first and the call is guarded with `?.`) and `THREE.Clock: This module has been deprecated. Please use THREE.Timer instead.` Neither is an error and neither affects behaviour. Logged as a cleanup candidate; deliberately not fixed here, as changing the clock source is a stage-core change with no test demanding it.
|
| 227 |
+
- **Playwright + Chromium were already installed** and `uv run playwright install chromium` was a no-op. No environment blockers; both browser suites ran to completion in this environment.
|
| 228 |
+
|
| 229 |
+
## Known Stubs
|
| 230 |
+
|
| 231 |
+
Intentional, each named in the plan and each failing loudly rather than silently:
|
| 232 |
+
|
| 233 |
+
- `avatar/turn-loop.js` β `startListening` / `stopListening` throw `Avatar.<name>() is not wired yet - plan 01-07 implements it`; `dispatchTurn` / `requestSlower` throw the same naming **01-08**. `test_deferred_methods_fail_loudly_not_silently` asserts this in a live browser under both transports and is deleted by 01-08.
|
| 234 |
+
- `avatar/stage.html` `DEMO_TIMELINE` and `avatar/assets/demo-konnichiwa.wav` β hand-authored placeholder, marked `PLACEHOLDER` in source. Plan 01-06 replaces both from a real VOICEVOX `AudioQuery`.
|
| 235 |
+
- `app.py` mounts only `VrmStage()`; there is no chat UI, mic control or credits footer yet. Those belong to 01-07, 01-08 and 01-10.
|
| 236 |
+
|
| 237 |
+
None of these block the plan's goal: the seam, the renderer and the parity proof are all real.
|
| 238 |
+
|
| 239 |
+
## Reproducing the demo asset
|
| 240 |
+
|
| 241 |
+
`avatar/assets/demo-konnichiwa.wav` was generated from `DEMO_TIMELINE`'s voiced-mora offsets β 24000 Hz mono 16-bit, 27096 frames, a 220 Hz carrier under a raised-cosine (Hann) envelope at amplitude 0.35, one burst per voiced mora at `(0.164, 0.128)`, `(0.484, 0.117)`, `(0.676, 0.107)`, `(0.826, 0.203)`.
|
| 242 |
+
|
| 243 |
+
## Verification Results
|
| 244 |
+
|
| 245 |
+
| Command | Result |
|
| 246 |
+
|---|---|
|
| 247 |
+
| `uv run pytest tests/test_transport_seam.py -q` | **36 passed** (plan requires β₯30 collected) |
|
| 248 |
+
| `uv run pytest tests/e2e/test_stage_standalone.py -q` | **4 passed** |
|
| 249 |
+
| `uv run pytest tests/e2e/test_facade_parity.py -q` | **3 passed** |
|
| 250 |
+
| `uv run pytest tests/ -x -q --ignore=tests/e2e` | **49 passed in 8.3 s wall clock** (budget: under 15 s) |
|
| 251 |
+
| `uv run ruff check . && uv run ruff format --check .` | **exit 0**, 19 files formatted |
|
| 252 |
+
| `uv run python -c "import app; app.build_app()"` | **exit 0** |
|
| 253 |
+
| `AVATAR_TRANSPORT=iframe uv run python -c "import app; app.build_app()"` | **exit 0** |
|
| 254 |
+
| `grep -rc "screenshot" tests/e2e/` | **0** in every file |
|
| 255 |
+
| `grep -rc "importmap" avatar/` | **0** in every file |
|
| 256 |
+
| Seam break: `window.Avatar = {}` appended to `avatar-iframe.js` | `test_only_facade_assigns_window_avatar` **FAILED** as designed, then reverted and green |
|
| 257 |
+
|
| 258 |
+
## User Setup Required
|
| 259 |
+
|
| 260 |
+
None β no external service configuration required. Nothing was pushed to the `space` remote; deployment remains plan 01-05's job and the Space still correctly serves 503 `NO_APP_FILE` until then.
|
| 261 |
+
|
| 262 |
+
## Next Phase Readiness
|
| 263 |
+
|
| 264 |
+
- **Plan 01-05 (deploy spike) is materially de-risked.** The inline transport already runs against a real Gradio 6.22.0 server locally with `mountCount === 1` and `threeInstanceCount === 1`. What 01-05 still has to prove is deployment-specific: CDN reachability from `*.hf.space`, `set_static_paths` behaviour behind the Space proxy, cold-start time, and mobile. If it fails, `AVATAR_TRANSPORT=iframe` is a one-variable flip that is already tested green.
|
| 265 |
+
- **Plan 01-06** replaces `DEMO_TIMELINE` and `demo-konnichiwa.wav` with generated output; the player contract (`{t, dur, viseme, weight}`, `'closed'` meaning all five at zero) is fixed and exercised.
|
| 266 |
+
- **Plans 01-07 and 01-08** must add their methods to `avatar/turn-loop.js`. `test_turn_surface_lives_in_the_shared_module` fails if they land in a transport instead, and `test_transports_expose_identical_surfaces` fails if the live objects drift.
|
| 267 |
+
- **Plan 01-09** should read `vrmMetaTitle = 'VRM1_Constraint_Twist_Sample'` and `__debug.vrmMeta` (a structured-clone-safe copy β `thumbnailImage` is deliberately stripped) for `test_vrm_meta_matches_licenses`.
|
| 268 |
+
- **Concern, unchanged:** esm.sh is a volunteer-run CDN and every module load in this plan depends on it. `01-RESEARCH.md`'s recommendation to vendor into `avatar/vendor/` before the phase exits still stands and is not yet owned by any plan.
|
| 269 |
+
- **Concern, new:** the two Gradio 6 API traps above mean `01-RESEARCH.md`'s `gr.HTML` section is not fully reliable. Any later plan copying `js_on_load` or `props` usage from it should copy from `avatar_component.py` instead.
|
| 270 |
+
|
| 271 |
+
---
|
| 272 |
+
*Phase: 01-voice-avatar-loop-skeleton*
|
| 273 |
+
*Completed: 2026-08-27*
|
| 274 |
+
|
| 275 |
+
## Self-Check: PASSED
|
| 276 |
+
|
| 277 |
+
All 11 claimed files exist on disk and all four claimed commits (`94a5b15`, `7146e75`, `f822421`, `d6ea280`) are present in the git history.
|