--- title: Japanese Learning Avatar emoji: ๐Ÿ—พ colorFrom: pink colorTo: indigo sdk: gradio sdk_version: 6.22.0 python_version: 3.12.12 app_file: app.py pinned: false license: mit short_description: Talk to a lip-synced VRM avatar in Japanese --- # Japanese Learning Avatar An animated 3D avatar tutor that teaches Japanese through spoken conversation. A VRM anime-style character listens to you speak Japanese and answers aloud, with lip-sync driven by real per-mora phoneme timings rather than by microphone-style mouth flapping. The avatar is rendered **in your browser** with three.js and `@pixiv/three-vrm`. Nothing about the character is generated server-side, so it paints and starts breathing while the Python backend is still waking up. ## Scope of this phase **Phase 1 has no AI tutoring in it yet.** This phase exists to prove the transport โ€” that a VRM avatar can live inside a Gradio `gr.HTML` component on a real Hugging Face Space, that speech can be synthesised with per-mora timings, and that the microphone round trip works without spending any GPU quota. The tutoring brain, level gating and progress tracking come in later phases. What is live here: - A VRM avatar with idle life โ€” blinking, breathing, a slight sway โ€” and a visible thinking pose while a reply is being synthesised. - Japanese speech synthesis with mora-accurate visemes (VOICEVOX, CPU only). Type Japanese and press Enter, or click "Say hello", and the avatar says it back with lip-sync. - Push-to-talk capture with in-browser speech recognition (Whisper via transformers.js), which costs **zero** ZeroGPU quota and therefore does not eat into a visitor's daily allowance. Hold the button, speak, and the transcript is echoed back aloud. - Replay (instant, no server round trip) and Slower (re-synthesised at 0.75x with a rebuilt lip-sync timeline), with per-stage latency shown under the controls. **The avatar repeats what you say.** There is no tutor yet; that is Phase 3. ## Run it locally ```bash uv sync --extra dev uv run python app.py ``` Requires Python 3.12.12 (see `.python-version`). The Space itself is pinned to Gradio 6.22.0 and Python 3.12.12 โ€” ZeroGPU only provides 3.10.13 and 3.12.12, so those pins are a platform constraint, not a preference. Tests: ```bash uv run pytest tests/ -q --ignore=tests/e2e # quick loop uv run pytest tests/e2e/ -q # browser suite (run it whole, never per-file) ``` ## Credits The in-app footer and About panel are the binding credit surfaces; this section mirrors them so the Space's own page carries the same strings. - **Voice: VOICEVOX:้›จๆ™ดใฏใ†** โ€” speech is synthesised with [VOICEVOX CORE](https://voicevox.hiroshiba.jp/term/) (software terms) using the ้›จๆ™ดใฏใ† voice by Amehare Project ([character terms](https://amehau.com/?page_id=225)). *Synthesised audio is provided under the VOICEVOX and VOICEVOX:้›จๆ™ดใฏใ† terms of use; by using it you agree to comply with them.* - **Avatar: VRM1_Constraint_Twist_Sample (c) 2022 pixiv Inc. โ€” VRM Public License 1.0** ([terms](https://vrm.dev/licenses/1.0/)), from the official VRM specification samples. The file's embedded `VRMC_vrm.meta` grants redistribution, avatar use by everyone, modification and commercial use, and requires no credit; it is credited anyway. - **Dictionary: JMdict (EDRDG, CC BY-SA 4.0)** โ€” Dictionary data: **JMdict** โ€” Copyright (c) James William Breen and The Electronic Dictionary Research and Development Group, used under the Creative Commons Attribution-ShareAlike Licence (V4.0). [JMdict-EDICT Dictionary Project](https://www.edrdg.org/wiki/index.php/JMdict-EDICT_Dictionary_Project) ยท [licence](https://www.edrdg.org/edrdg/licence.html). The glosses in the word-lookup card come from a compact projection of JMdict built by this project; that projection is a derivative and is itself CC BY-SA 4.0. JSON packaging by [scriptin/jmdict-simplified](https://github.com/scriptin/jmdict-simplified), release 3.6.2+20260831182826, whose derived files carry the EDRDG licence. - **JLPT levels: Jonathan Waller's JLPT Resources via stephenmk/yomitan-jlpt-vocab** โ€” levels come from [Jonathan Waller's JLPT Resources](https://www.tanos.co.uk/jlpt/) (Creative Commons BY) via [stephenmk/yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab) (CC BY-SA 4.0), which added the JMdict entry id to every word; kanji levels come from [davidluzgouveia/kanji-data](https://github.com/davidluzgouveia/kanji-data) (MIT), which took its levels from the same Waller lists. **There is no official JLPT vocabulary list** โ€” the upstream project says so itself โ€” so every level shown is an estimate, and a word on no list is labelled "N1+ / beyond lists" rather than given a level the data does not claim. - **Translation: OPUS-MT (Helsinki-NLP, Apache-2.0)** โ€” English on demand comes from [Helsinki-NLP/opus-mt-ja-en](https://huggingface.co/Helsinki-NLP/opus-mt-ja-en) (Apache-2.0), shipped as a [CTranslate2](https://github.com/OpenNMT/CTranslate2) (MIT) int8 conversion tokenised with [SentencePiece](https://github.com/google/sentencepiece) (Apache-2.0). It runs on the Space's CPU, no GPU, so it costs a visitor none of their daily quota. - **Japanese text analysis** โ€” [SudachiPy](https://github.com/WorksApplications/sudachi.rs) with [SudachiDict-core](https://github.com/WorksApplications/SudachiDict) (both Apache-2.0, Works Applications Co., Ltd.) and [jaconv](https://github.com/ikegami-yukino/jaconv) (MIT). - **Open JTalk dictionary** `open_jtalk_dic_utf_8-1.11` โ€” BSD-3-Clause, (c) 2009 Nara Institute of Science and Technology. Open JTalk itself is by the Nagoya Institute of Technology and the HTS Working Group, also under a modified BSD licence. - **Runtime libraries** โ€” [three.js](https://threejs.org/) (MIT), [@pixiv/three-vrm](https://github.com/pixiv/three-vrm) (MIT), [@huggingface/transformers](https://github.com/huggingface/transformers.js) (Apache-2.0). `LICENSES.md` is the project-wide record: every asset's version, rights holder, licence URL, the permission relied on and how it is honoured, plus the verification record naming the primary source read for each one. `docs/ASSETS.md` and `docs/VOICEVOX-SETUP.md` hold the Phase 1 provenance detail, and `docs/LANGUAGE.md` records the language core itself. ## Licence MIT for this repository's own code. Third-party assets and models carry their own terms โ€” see above.