Automatic Speech Recognition
LiteRT-LM
LiteRT
litertlm
on-device
edge
asr
speech-recognition
audio
qwen3-asr
Instructions to use litert-community/Qwen3-ASR-1.7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/Qwen3-ASR-1.7B with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/Qwen3-ASR-1.7B \ --prompt="Write me a poem"
- LiteRT
How to use litert-community/Qwen3-ASR-1.7B with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
android/: the Android sample app (model picker, hold to talk, three clips, device check); card: the Sample app line with the Galaxy S26 numbers
Browse files- .gitattributes +3 -0
- README.md +2 -0
- android/.gitignore +10 -0
- android/LICENSE +21 -0
- android/README.md +189 -0
- android/app/build.gradle.kts +39 -0
- android/app/src/androidTest/kotlin/com/asrlitertlm/AsrDeviceCheck.kt +417 -0
- android/app/src/main/AndroidManifest.xml +28 -0
- android/app/src/main/kotlin/com/asrlitertlm/AsrEngine.kt +181 -0
- android/app/src/main/kotlin/com/asrlitertlm/AsrSession.kt +248 -0
- android/app/src/main/kotlin/com/asrlitertlm/ClipPlayer.kt +99 -0
- android/app/src/main/kotlin/com/asrlitertlm/MainActivity.kt +712 -0
- android/app/src/main/kotlin/com/asrlitertlm/MicRecorder.kt +123 -0
- android/app/src/main/kotlin/com/asrlitertlm/ModelProfile.kt +108 -0
- android/app/src/main/kotlin/com/asrlitertlm/PlayBar.kt +35 -0
- android/app/src/main/kotlin/com/asrlitertlm/TextMatch.kt +64 -0
- android/app/src/main/kotlin/com/asrlitertlm/Wav.kt +140 -0
- android/app/src/main/res/raw/fleurs_cmn_1698.wav +3 -0
- android/app/src/main/res/raw/fleurs_en_1698.wav +3 -0
- android/app/src/main/res/raw/fleurs_ja_1698.wav +3 -0
- android/app/src/main/res/values/strings.xml +4 -0
- android/app/src/main/res/values/themes.xml +15 -0
- android/app/src/test/kotlin/com/asrlitertlm/AsrLogicTest.kt +104 -0
- android/build.gradle.kts +9 -0
- android/gradle.properties +7 -0
- android/gradle/wrapper/gradle-wrapper.jar +0 -0
- android/gradle/wrapper/gradle-wrapper.properties +7 -0
- android/gradlew +248 -0
- android/settings.gradle.kts +20 -0
.gitattributes
CHANGED
|
@@ -36,3 +36,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 36 |
Qwen3-ASR-1.7B.litertlm filter=lfs diff=lfs merge=lfs -text
|
| 37 |
assets/demo_s26.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
assets/demo_s26.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
Qwen3-ASR-1.7B.litertlm filter=lfs diff=lfs merge=lfs -text
|
| 37 |
assets/demo_s26.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
assets/demo_s26.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
android/app/src/main/res/raw/fleurs_cmn_1698.wav filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
android/app/src/main/res/raw/fleurs_en_1698.wav filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
android/app/src/main/res/raw/fleurs_ja_1698.wav filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -153,6 +153,8 @@ print(raw.partition("<asr_text>")[2].strip())
|
|
| 153 |
|
| 154 |
For the Chinese example clip `raw` is `language Chinese<asr_text>开放时间:早上九点至下午五点。` and the script prints the text after the tag. `backend=GPU()` (from `litert_lm.interfaces`) puts the language model on the GPU. Keep `audio_backend=CPU()`: the audio encoder does not run on the GPU delegate (`create_conversation` fails). The Kotlin Conversation API takes the same calls. In the Galaxy S26 app above (`litertlm-android` 0.17.1, CPU), the three clips gave the same answers, rendered prompts and token counts as the Python API on the Mac (0.17.1, CPU) in all 5 app runs. The Swift API was not run for this file.
|
| 155 |
|
|
|
|
|
|
|
| 156 |
On Android the S26 rows above came from the command-line tool: `litert_lm_advanced_main --backend=cpu --audio_backend=cpu --sampler_backend=cpu --num_cpu_threads=4 --model_path=Qwen3-ASR-1.7B.litertlm --cache_dir=cache --max_num_tokens=1024 --max_output_tokens=256 --benchmark --input_prompt='[audio:/path/clip.wav]'`.
|
| 157 |
|
| 158 |
## License
|
|
|
|
| 153 |
|
| 154 |
For the Chinese example clip `raw` is `language Chinese<asr_text>开放时间:早上九点至下午五点。` and the script prints the text after the tag. `backend=GPU()` (from `litert_lm.interfaces`) puts the language model on the GPU. Keep `audio_backend=CPU()`: the audio encoder does not run on the GPU delegate (`create_conversation` fails). The Kotlin Conversation API takes the same calls. In the Galaxy S26 app above (`litertlm-android` 0.17.1, CPU), the three clips gave the same answers, rendered prompts and token counts as the Python API on the Mac (0.17.1, CPU) in all 5 app runs. The Swift API was not run for this file.
|
| 155 |
|
| 156 |
+
Sample app (Android): [`android/`](https://huggingface.co/litert-community/Qwen3-ASR-1.7B/tree/main/android) — one app for this file and two sibling bundles (Qwen3-ASR-1.7B, Fun-ASR-Nano-2512, Confucius4-R2T2): a model picker, hold-to-talk up to 30 s and three FLEURS clips, with a device check. Galaxy S26, language model and audio encoder on the CPU (4 threads), the 6.4 s English clip in 2.49 s and the 9.7 s Mandarin clip in 2.38 s (2026-10-09, CPU at 3.51 / 4.19 GHz); a 9 s microphone recording in 2.97 s with no character error, measured by the app's device check on 2026-10-09.
|
| 157 |
+
|
| 158 |
On Android the S26 rows above came from the command-line tool: `litert_lm_advanced_main --backend=cpu --audio_backend=cpu --sampler_backend=cpu --num_cpu_threads=4 --model_path=Qwen3-ASR-1.7B.litertlm --cache_dir=cache --max_num_tokens=1024 --max_output_tokens=256 --benchmark --input_prompt='[audio:/path/clip.wav]'`.
|
| 159 |
|
| 160 |
## License
|
android/.gitignore
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.gradle/
|
| 2 |
+
.kotlin/
|
| 3 |
+
build/
|
| 4 |
+
app/build/
|
| 5 |
+
local.properties
|
| 6 |
+
.gradle-user-home/
|
| 7 |
+
# The .litertlm bundles are pushed to the phone (see README), never committed.
|
| 8 |
+
*.litertlm
|
| 9 |
+
# hf download --local-dir in the README writes the bundles (and its own metadata) here.
|
| 10 |
+
bundles/
|
android/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Daisuke Majima
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
android/README.md
ADDED
|
@@ -0,0 +1,189 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ASR LiteRT-LM — speech to text on Android with three LiteRT-LM bundles
|
| 2 |
+
|
| 3 |
+
Hold the button, speak for up to 30 s, and the app writes what you said. On the CPU of a Galaxy S26
|
| 4 |
+
(4 threads, measured 2026-10-09), Qwen3-ASR-1.7B turned a 6.4 s English clip into text in 2.49 s and
|
| 5 |
+
Fun-ASR-Nano-2512 in 1.19 s. The app runs three speech recognition models in the LiteRT-LM `.litertlm`
|
| 6 |
+
format through one code path: pick a model, then talk or play one of three bundled clips. Everything
|
| 7 |
+
runs on the phone; the app asks for the microphone and nothing else (no network permission).
|
| 8 |
+
|
| 9 |
+
## Models
|
| 10 |
+
|
| 11 |
+
| Model | Hugging Face | File | Size | License | Languages |
|
| 12 |
+
|---|---|---|---|---|---|
|
| 13 |
+
| Qwen3-ASR-1.7B | [litert-community/Qwen3-ASR-1.7B](https://huggingface.co/litert-community/Qwen3-ASR-1.7B) | `Qwen3-ASR-1.7B.litertlm` | 2,693,064,192 B | Apache-2.0 | 30 languages (the card scores English, Mandarin, Japanese) |
|
| 14 |
+
| Fun-ASR-Nano-2512 | [litert-community/Fun-ASR-Nano-2512](https://huggingface.co/litert-community/Fun-ASR-Nano-2512) | `Fun-ASR-Nano-2512.litertlm` | 1,255,894,736 B | Apache-2.0 | Chinese, English, Japanese |
|
| 15 |
+
| Confucius4-R2T2 | [mlboydaisuke/Confucius4-R2T2-LiteRT](https://huggingface.co/mlboydaisuke/Confucius4-R2T2-LiteRT) | `Confucius4-R2T2.litertlm` | 2,693,064,192 B | [NetEase Youdao Model Use License Agreement](https://huggingface.co/mlboydaisuke/Confucius4-R2T2-LiteRT/blob/main/MODEL_LICENSE) | Chinese, English; a fine-tune of Qwen3-ASR-1.7B |
|
| 16 |
+
|
| 17 |
+
Each file holds the audio encoder, the language model, the tokenizer and the prompt template. Read
|
| 18 |
+
the Confucius4-R2T2 agreement before you use that file: it has user-count and revenue thresholds
|
| 19 |
+
(clause 2.2) and limits on using the model to improve other models (clause 3.4).
|
| 20 |
+
|
| 21 |
+
## Run it in 5 lines
|
| 22 |
+
|
| 23 |
+
From this directory, with the phone connected over adb:
|
| 24 |
+
|
| 25 |
+
```bash
|
| 26 |
+
hf download litert-community/Qwen3-ASR-1.7B Qwen3-ASR-1.7B.litertlm --local-dir bundles
|
| 27 |
+
hf download litert-community/Fun-ASR-Nano-2512 Fun-ASR-Nano-2512.litertlm --local-dir bundles
|
| 28 |
+
hf download mlboydaisuke/Confucius4-R2T2-LiteRT Confucius4-R2T2.litertlm --local-dir bundles
|
| 29 |
+
./gradlew :app:installDebug && adb shell am start -n com.asrlitertlm/.MainActivity
|
| 30 |
+
adb push bundles/*.litertlm /sdcard/Android/data/com.asrlitertlm/files/
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
Reopen the app, tap a model in the picker and hold **Hold to talk**. The first launch creates the
|
| 34 |
+
folder the push writes to. One bundle is enough; the picker marks the others `missing`.
|
| 35 |
+
|
| 36 |
+
- Storage: the three files take 6.6 GB. The first load of each file writes a runtime cache into the
|
| 37 |
+
app's cache directory: 3.0 GB for each Qwen3-ASR file, 1.5 GB for Fun-ASR.
|
| 38 |
+
- Run `adb shell touch /sdcard/Android/data/com.asrlitertlm/files/*.litertlm` after the push. adb push
|
| 39 |
+
writes whole-second modification times, and with those a later launch can miss the runtime cache
|
| 40 |
+
and write it again ([LiteRT-LM #3772](https://github.com/google-ai-edge/LiteRT-LM/issues/3772)).
|
| 41 |
+
|
| 42 |
+
## Galaxy S26 numbers
|
| 43 |
+
|
| 44 |
+
Galaxy S26 (SM-S942Q, Android 16), `litertlm-android` 0.17.1, language model and audio encoder on the
|
| 45 |
+
CPU with 4 threads, 2026-10-09. "Transcribe" is the wall time of `sendMessage()`, from sending the
|
| 46 |
+
clip to the full answer; RTF is that time over the clip's length. The three clips are the same sentence
|
| 47 |
+
in three languages (below).
|
| 48 |
+
|
| 49 |
+
| Model | zh, 9.7 s | en, 6.4 s | ja, 11.0 s | Microphone, 9 s | CPU limit at the start (big / prime cores) |
|
| 50 |
+
|---|---|---|---|---|---|
|
| 51 |
+
| Qwen3-ASR-1.7B | 2.38 s, RTF 0.25 | 2.49 s, RTF 0.39 | 3.56 s, RTF 0.32 | 2.97 s, CER 0 | 3.51 / 4.19 GHz |
|
| 52 |
+
| Confucius4-R2T2 | 3.48 s, RTF 0.36 | 2.59 s, RTF 0.40 | 3.54 s, RTF 0.32 | 3.17 s, CER 0 | 3.63 / 4.74 GHz (no limit) |
|
| 53 |
+
| Fun-ASR-Nano-2512 | 1.95 s, RTF 0.20 | 1.19 s, RTF 0.19 | 1.83 s, RTF 0.17 | 1.44 s, CER 0 | 3.63 / 4.74 GHz (no limit) |
|
| 54 |
+
|
| 55 |
+
| Model | Engine load, first (writes the cache) | Engine load, cache in place | Runtime cache | Peak resident memory, first load |
|
| 56 |
+
|---|---|---|---|---|
|
| 57 |
+
| Qwen3-ASR-1.7B | 7.35 s | 0.72 s | 3.0 GB | 5.6 GB |
|
| 58 |
+
| Confucius4-R2T2 | 4.27 s | 1.33 s | 3.0 GB | 5.7 GB |
|
| 59 |
+
| Fun-ASR-Nano-2512 | 1.39 s | 1.13 s | 1.5 GB | 3.5 GB |
|
| 60 |
+
|
| 61 |
+
- Every clip's text equals the Mac CPU answer of the same file (LiteRT-LM 0.17.1, Python) after
|
| 62 |
+
removing punctuation and spaces. The three models write the same sentences for these clips.
|
| 63 |
+
- The last column is the CPU limit at the start of the run (the maximum is 3.63 / 4.74 GHz; a warm phone
|
| 64 |
+
lowers it). The Confucius4-R2T2 and Fun-ASR rows are a second run of the check on the same day with no
|
| 65 |
+
limit (Confucius4-R2T2 at thermal status 0, Fun-ASR at status 1, each file pushed alone). The first run,
|
| 66 |
+
with the limit lowered by heat, gave Confucius4-R2T2 4.03 / 3.06 / 3.83 s at 2.75 / 2.67 GHz and
|
| 67 |
+
Fun-ASR 1.26 / 1.60 / 2.29 s at 2.23 / 2.23 GHz. Confucius4-R2T2 has the same architecture and file
|
| 68 |
+
size as Qwen3-ASR-1.7B. In a run that started at the maximum, Qwen3-ASR-1.7B took 2.13 / 2.14 / 3.19 s
|
| 69 |
+
for the three clips.
|
| 70 |
+
- "Engine load" is `Engine.initialize()` plus the first conversation, which creates the audio encoder.
|
| 71 |
+
The Qwen3-ASR-1.7B cached load is a new launch; the other two are a second load in the same process.
|
| 72 |
+
- Microphone: the phone plays clip en through its speaker and records it with its microphone
|
| 73 |
+
(media volume 3 of 15), then transcribes the recording. CER is the character error rate against
|
| 74 |
+
the clip's sentence.
|
| 75 |
+
- Language model on the GPU (launch extra `--es backend gpu`), Fun-ASR-Nano-2512: 1.69 / 1.68 / 2.29 s,
|
| 76 |
+
the same text. All nodes of the prefill and decode graphs ran on the GPU (LITERT_CL); the audio
|
| 77 |
+
encoder stays on the CPU (the model cards: the GPU delegate does not take it). Engine load 2.61 s.
|
| 78 |
+
It was not faster than the CPU here.
|
| 79 |
+
|
| 80 |
+
## How the app calls a model
|
| 81 |
+
|
| 82 |
+
The three bundles share one path ([`AsrEngine.kt`](app/src/main/kotlin/com/asrlitertlm/AsrEngine.kt)):
|
| 83 |
+
one `Engine`, a new `Conversation` per clip, and one user message that holds only the audio file. Each
|
| 84 |
+
bundle's template adds the rest: Fun-ASR's default instruction `语音转写:`, Qwen3-ASR's audio markers.
|
| 85 |
+
|
| 86 |
+
```kotlin
|
| 87 |
+
val engine = Engine(EngineConfig(
|
| 88 |
+
modelPath = bundle.path, // /sdcard/Android/data/com.asrlitertlm/files/<file>
|
| 89 |
+
backend = Backend.CPU(threadCount = 4), // the language model
|
| 90 |
+
audioBackend = Backend.CPU(threadCount = 4), // the audio encoder
|
| 91 |
+
cacheDir = context.cacheDir.path,
|
| 92 |
+
))
|
| 93 |
+
engine.initialize()
|
| 94 |
+
|
| 95 |
+
val raw = engine.createConversation(ConversationConfig(
|
| 96 |
+
samplerConfig = SamplerConfig(topK = 1, topP = 1.0, temperature = 0.0),
|
| 97 |
+
maxOutputToken = 512,
|
| 98 |
+
)).use { conversation ->
|
| 99 |
+
conversation.sendMessage(Message.user(Contents.of(Content.AudioFile(wav.absolutePath)))).toString()
|
| 100 |
+
}
|
| 101 |
+
// Qwen3-ASR-1.7B and Confucius4-R2T2: "language English<asr_text>Everything in the universe ..."
|
| 102 |
+
// Fun-ASR-Nano-2512: "Everything in the universe ..."
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
Only the answer differs: [`ModelProfile.kt`](app/src/main/kotlin/com/asrlitertlm/ModelProfile.kt)
|
| 106 |
+
splits the Qwen3-ASR answer at `<asr_text>` and shows the named language; Fun-ASR's answer is the text.
|
| 107 |
+
|
| 108 |
+
- One file is loaded at a time: picking another model closes the engine first. A first load of a
|
| 109 |
+
Qwen3-ASR file peaked at 5.6–5.7 GB resident (VmHWM, the memory-mapped file included) on this phone
|
| 110 |
+
with 11.4 GB of memory.
|
| 111 |
+
- `Engine.close()` throws when called twice, so the app drops its reference before closing; releasing
|
| 112 |
+
twice is harmless.
|
| 113 |
+
- All model calls run on one worker thread.
|
| 114 |
+
|
| 115 |
+
## Microphone
|
| 116 |
+
|
| 117 |
+
Hold **Hold to talk** and speak; releasing the button ends the recording, and so does the 30 s limit.
|
| 118 |
+
Each model reads one window of about 30 s per message, so the app sends at most 30 s. The app records
|
| 119 |
+
16 kHz mono 16-bit PCM (the `VOICE_RECOGNITION` source) and writes it as a WAV file for the runtime.
|
| 120 |
+
A recording quieter than −60 dBFS RMS is reported as "No speech heard" and is not sent to the model.
|
| 121 |
+
Without the microphone permission the app shows a sentence that says how to allow it; the clips still work.
|
| 122 |
+
|
| 123 |
+
## The three clips
|
| 124 |
+
|
| 125 |
+
`app/src/main/res/raw/` holds FLEURS test sentence 1698 read in Mandarin, English and Japanese
|
| 126 |
+
([google/fleurs](https://huggingface.co/datasets/google/fleurs) revision `70bb2e84`, test split, id 1698;
|
| 127 |
+
FLEURS, Conneau et al. 2022, licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)). Changes:
|
| 128 |
+
converted to 16-bit PCM and given one linear gain each, so that all three have an active RMS of −20 dBFS.
|
| 129 |
+
The English reading is "Everything in the Universe is made of matter. All matter is made of tiny
|
| 130 |
+
particles called atoms."
|
| 131 |
+
|
| 132 |
+
## Device check
|
| 133 |
+
|
| 134 |
+
[`AsrDeviceCheck.kt`](app/src/androidTest/kotlin/com/asrlitertlm/AsrDeviceCheck.kt) runs on a phone
|
| 135 |
+
with at least one bundle pushed (as above). Turn the media volume up first: the microphone step plays a
|
| 136 |
+
clip through the speaker.
|
| 137 |
+
|
| 138 |
+
```bash
|
| 139 |
+
./gradlew :app:installDebug :app:assembleDebugAndroidTest
|
| 140 |
+
adb install -r -t app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
|
| 141 |
+
adb shell pm revoke com.asrlitertlm android.permission.RECORD_AUDIO
|
| 142 |
+
adb shell am instrument -w -e class com.asrlitertlm.AsrDeviceCheck#check com.asrlitertlm.test/androidx.test.runner.AndroidJUnitRunner
|
| 143 |
+
adb logcat -d -s asr-check | grep RESULT
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
It prints one line per step, `RESULT step=<name> ok=<true|false> model=<id> ...`, then
|
| 147 |
+
`RESULT ok=<all>`. It starts with the microphone permission revoked and checks the sentence the app
|
| 148 |
+
shows; then it grants the permission. For each bundle on the phone: load, the three clips (text equal to
|
| 149 |
+
the expected sentence after removing punctuation and spaces), the microphone (CER ≤ 0.15), release
|
| 150 |
+
twice, and a second load. Then a 33.6 s clip (cut to 30 s), a silent recording and a file that is not
|
| 151 |
+
on the phone. With two or more bundles it also switches models without releasing. Add
|
| 152 |
+
`-e class com.asrlitertlm.AsrDeviceCheck#gpu -e gpu_model fun-asr-nano-2512` for the GPU run.
|
| 153 |
+
|
| 154 |
+
## Launch extras
|
| 155 |
+
|
| 156 |
+
For recordings and tests; the app works without them.
|
| 157 |
+
|
| 158 |
+
| Extra | Effect |
|
| 159 |
+
|---|---|
|
| 160 |
+
| `--es model <id>` | `qwen3-asr-1.7b`, `fun-asr-nano-2512` or `confucius4-r2t2` (default: the model used last) |
|
| 161 |
+
| `--es backend gpu` | language model on the GPU; the audio encoder stays on the CPU |
|
| 162 |
+
| `--ez autoplay true` | after loading: clips zh, en, ja, each played and then transcribed (`--ei delay_ms`, `--ei gap_ms`) |
|
| 163 |
+
| `--ei mic_test_ms 9000` | after loading and `delay_ms`: record that long, then transcribe (`--ez mic_test_play true` plays clip en meanwhile) |
|
| 164 |
+
|
| 165 |
+
The app logs one `STATE <name>` line per screen state and `LOAD` / `TRANSCRIBED` lines with the times
|
| 166 |
+
under the tag `AsrLitertlm`.
|
| 167 |
+
|
| 168 |
+
## Files
|
| 169 |
+
|
| 170 |
+
| File | Role |
|
| 171 |
+
|---|---|
|
| 172 |
+
| `app/src/main/kotlin/com/asrlitertlm/AsrEngine.kt` | Engine and one Conversation per clip: load, transcribe (cut at 30 s), release (safe twice) |
|
| 173 |
+
| `app/src/main/kotlin/com/asrlitertlm/ModelProfile.kt` | the three bundles: file name, Hugging Face id, answer format |
|
| 174 |
+
| `app/src/main/kotlin/com/asrlitertlm/AsrSession.kt` | the app's one engine on one worker thread, events for the screen, the no-speech check |
|
| 175 |
+
| `app/src/main/kotlin/com/asrlitertlm/MainActivity.kt` | picker, hold to talk, clip buttons, launch extras |
|
| 176 |
+
| `app/src/main/kotlin/com/asrlitertlm/MicRecorder.kt` | 16 kHz mono recording, up to 30 s |
|
| 177 |
+
| `app/src/main/kotlin/com/asrlitertlm/ClipPlayer.kt`, `PlayBar.kt` | clip playback through the speaker and its progress bar |
|
| 178 |
+
| `app/src/main/kotlin/com/asrlitertlm/Wav.kt`, `TextMatch.kt` | WAV read, write and cut; transcript comparison (normalization and CER) |
|
| 179 |
+
| `app/src/main/res/raw/fleurs_*_1698.wav` | the three clips |
|
| 180 |
+
| `app/src/androidTest/kotlin/com/asrlitertlm/AsrDeviceCheck.kt` | device check |
|
| 181 |
+
| `app/src/test/kotlin/com/asrlitertlm/AsrLogicTest.kt` | JVM tests: answer parsing, transcript comparison, WAV cut and levels |
|
| 182 |
+
|
| 183 |
+
Build: AGP 9.3.1 with Kotlin 2.4.0 (the `litertlm-android` 0.17.1 AAR carries Kotlin 2.4 metadata),
|
| 184 |
+
compileSdk 36, minSdk 31, arm64-v8a. The runtime version is pinned in `gradle.properties`.
|
| 185 |
+
|
| 186 |
+
## License
|
| 187 |
+
|
| 188 |
+
The app code is under the repository's MIT license. The clips are CC BY 4.0 (FLEURS, see above). Each
|
| 189 |
+
model file has its own license (table above); none of them is in this directory.
|
android/app/build.gradle.kts
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// On-device speech recognition with three LiteRT-LM bundles (Qwen3-ASR-1.7B, Fun-ASR-Nano-2512, Confucius4-R2T2):
|
| 2 |
+
// one Engine at a time, a new Conversation per clip, the language model and the audio encoder on the CPU.
|
| 3 |
+
plugins { id("com.android.application") }
|
| 4 |
+
|
| 5 |
+
val litertlmVersion: String = providers.gradleProperty("litertlmVersion").get()
|
| 6 |
+
|
| 7 |
+
android {
|
| 8 |
+
namespace = "com.asrlitertlm"
|
| 9 |
+
compileSdk = 36
|
| 10 |
+
|
| 11 |
+
defaultConfig {
|
| 12 |
+
applicationId = "com.asrlitertlm"
|
| 13 |
+
minSdk = 31
|
| 14 |
+
targetSdk = 36
|
| 15 |
+
versionCode = 1
|
| 16 |
+
versionName = "1.0"
|
| 17 |
+
ndk { abiFilters += setOf("arm64-v8a") }
|
| 18 |
+
buildConfigField("String", "LITERTLM_VERSION", "\"$litertlmVersion\"")
|
| 19 |
+
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
|
| 20 |
+
}
|
| 21 |
+
|
| 22 |
+
buildFeatures { buildConfig = true }
|
| 23 |
+
|
| 24 |
+
compileOptions {
|
| 25 |
+
sourceCompatibility = JavaVersion.VERSION_17
|
| 26 |
+
targetCompatibility = JavaVersion.VERSION_17
|
| 27 |
+
}
|
| 28 |
+
}
|
| 29 |
+
|
| 30 |
+
dependencies {
|
| 31 |
+
// Brings gson, kotlin-reflect and kotlinx-coroutines-android (its POM).
|
| 32 |
+
implementation("com.google.ai.edge.litertlm:litertlm-android:$litertlmVersion")
|
| 33 |
+
|
| 34 |
+
testImplementation("junit:junit:4.13.2")
|
| 35 |
+
|
| 36 |
+
// Device check (app/src/androidTest): README "Device check" runs it with am instrument.
|
| 37 |
+
androidTestImplementation("androidx.test:runner:1.6.2")
|
| 38 |
+
androidTestImplementation("androidx.test.ext:junit:1.2.1")
|
| 39 |
+
}
|
android/app/src/androidTest/kotlin/com/asrlitertlm/AsrDeviceCheck.kt
ADDED
|
@@ -0,0 +1,417 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import android.Manifest
|
| 4 |
+
import android.content.Intent
|
| 5 |
+
import android.content.pm.PackageManager
|
| 6 |
+
import android.util.Log
|
| 7 |
+
import androidx.test.ext.junit.runners.AndroidJUnit4
|
| 8 |
+
import androidx.test.platform.app.InstrumentationRegistry
|
| 9 |
+
import java.io.File
|
| 10 |
+
import java.util.Locale
|
| 11 |
+
import java.util.concurrent.CountDownLatch
|
| 12 |
+
import java.util.concurrent.ExecutionException
|
| 13 |
+
import java.util.concurrent.Future
|
| 14 |
+
import java.util.concurrent.TimeUnit
|
| 15 |
+
import java.util.concurrent.atomic.AtomicReference
|
| 16 |
+
import org.junit.Assert.assertTrue
|
| 17 |
+
import org.junit.Assume.assumeTrue
|
| 18 |
+
import org.junit.Test
|
| 19 |
+
import org.junit.runner.RunWith
|
| 20 |
+
|
| 21 |
+
/**
|
| 22 |
+
* Device check for the app's paths on a phone with the bundles pushed. One line per step under the log tag
|
| 23 |
+
* `asr-check`: `RESULT step=<name> ok=<bool> model=<id> ...`, then `RESULT ok=<all> scope=<test>`.
|
| 24 |
+
*
|
| 25 |
+
* [check] expects RECORD_AUDIO to be revoked when it starts (`adb shell pm revoke com.asrlitertlm
|
| 26 |
+
* android.permission.RECORD_AUDIO`): its first step is the microphone path without the permission; it then grants the
|
| 27 |
+
* permission itself. Per bundle on the phone: load, the three clips (the answer's text after normalization equals the
|
| 28 |
+
* expected sentence; the CER is printed when it does not), the microphone (clip en played through the speaker and
|
| 29 |
+
* picked up by the microphone, CER <= 0.15), release (a second release is harmless) and, when the load wrote the
|
| 30 |
+
* runtime cache, a second load. Then the failure paths: switching models, a clip longer than 30 s, a silent
|
| 31 |
+
* recording and a bundle that is not on the phone.
|
| 32 |
+
*
|
| 33 |
+
* [gpu] runs only with `-e gpu_model <id>`: the language model on the GPU, the audio encoder on the CPU.
|
| 34 |
+
*/
|
| 35 |
+
@RunWith(AndroidJUnit4::class)
|
| 36 |
+
class AsrDeviceCheck {
|
| 37 |
+
private val instrumentation = InstrumentationRegistry.getInstrumentation()
|
| 38 |
+
private val context = instrumentation.targetContext
|
| 39 |
+
private val outcomes = mutableListOf<Boolean>()
|
| 40 |
+
|
| 41 |
+
@Test
|
| 42 |
+
fun check() {
|
| 43 |
+
AsrSession.init(context)
|
| 44 |
+
val present = ORDER.filter { AsrSession.isPresent(it) }
|
| 45 |
+
report(
|
| 46 |
+
"bundles",
|
| 47 |
+
present.isNotEmpty(),
|
| 48 |
+
null,
|
| 49 |
+
"present" to present.joinToString(",") { it.id }.ifEmpty { "none" },
|
| 50 |
+
"missing" to ORDER.filterNot { it in present }.joinToString(",") { it.id }.ifEmpty { "none" },
|
| 51 |
+
"dir" to AsrSession.modelDir.path,
|
| 52 |
+
)
|
| 53 |
+
if (present.isEmpty()) return finish("check")
|
| 54 |
+
|
| 55 |
+
val first = present.first()
|
| 56 |
+
val firstCache = cacheFiles(first)
|
| 57 |
+
val screen = launch(launchIntent(first))
|
| 58 |
+
try {
|
| 59 |
+
val firstLoad = awaitLoad(checkNotNull(AsrSession.lastLoad) { "the screen did not start a load" }, first)
|
| 60 |
+
val loads = mutableMapOf(first to (firstLoad to firstCache))
|
| 61 |
+
|
| 62 |
+
permissionStep(screen, first)
|
| 63 |
+
instrumentation.uiAutomation.grantRuntimePermission(context.packageName, Manifest.permission.RECORD_AUDIO)
|
| 64 |
+
report("grant", hasMicPermission(), null, "granted" to hasMicPermission())
|
| 65 |
+
|
| 66 |
+
for (profile in present) {
|
| 67 |
+
val load = loads[profile] ?: run {
|
| 68 |
+
val before = cacheFiles(profile)
|
| 69 |
+
awaitLoad(onScreen(screen) { it.selectModel(profile) }, profile)?.let { it to before }
|
| 70 |
+
}
|
| 71 |
+
if (load?.first == null) continue
|
| 72 |
+
reportLoad("load", profile, load.first!!, load.second)
|
| 73 |
+
clipSteps(profile, "clip")
|
| 74 |
+
micStep(screen, profile)
|
| 75 |
+
releaseStep(profile)
|
| 76 |
+
if (cacheState(profile, load.second) == "written") reloadStep(profile)
|
| 77 |
+
}
|
| 78 |
+
failurePaths(screen, present)
|
| 79 |
+
} finally {
|
| 80 |
+
close(screen)
|
| 81 |
+
}
|
| 82 |
+
finish("check")
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
@Test
|
| 86 |
+
fun gpu() {
|
| 87 |
+
val profile = ModelProfile.byId(InstrumentationRegistry.getArguments().getString("gpu_model"))
|
| 88 |
+
assumeTrue("run with -e gpu_model <id>", profile != null)
|
| 89 |
+
AsrSession.init(context)
|
| 90 |
+
val model = profile!!
|
| 91 |
+
if (!AsrSession.isPresent(model)) {
|
| 92 |
+
report("gpu-load", false, model, "reason" to "bundle not on the phone")
|
| 93 |
+
return finish("gpu")
|
| 94 |
+
}
|
| 95 |
+
val before = cacheFiles(model)
|
| 96 |
+
val screen = launch(launchIntent(model).putExtra(MainActivity.EXTRA_BACKEND, "gpu"))
|
| 97 |
+
try {
|
| 98 |
+
val info = awaitLoad(checkNotNull(AsrSession.lastLoad), model, step = "gpu-load")
|
| 99 |
+
if (info != null) {
|
| 100 |
+
reportLoad("gpu-load", model, info, before)
|
| 101 |
+
clipSteps(model, "gpu-clip")
|
| 102 |
+
releaseStep(model, step = "gpu-release")
|
| 103 |
+
val beforeReload = cacheFiles(model)
|
| 104 |
+
val reloaded = awaitLoad(AsrSession.load(model, LmBackend.GPU), model, step = "gpu-reload")
|
| 105 |
+
if (reloaded != null) reportLoad("gpu-reload", model, reloaded, beforeReload)
|
| 106 |
+
AsrSession.release().get(TIMEOUT_S, TimeUnit.SECONDS)
|
| 107 |
+
}
|
| 108 |
+
} finally {
|
| 109 |
+
close(screen)
|
| 110 |
+
}
|
| 111 |
+
finish("gpu")
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
// ---- steps --------------------------------------------------------------------------------------------------
|
| 115 |
+
|
| 116 |
+
/** The microphone path with RECORD_AUDIO revoked returns a sentence, records nothing, and a clip still works. */
|
| 117 |
+
private fun permissionStep(screen: MainActivity, profile: ModelProfile) {
|
| 118 |
+
val deniedAtStart = !hasMicPermission()
|
| 119 |
+
val result = micTest(screen, durationMs = 2000, play = false, waitS = 10)
|
| 120 |
+
val sentence = (result as? MicTestResult.Failed)?.message.orEmpty()
|
| 121 |
+
val clip = Clip.ALL[1]
|
| 122 |
+
val clipText = transcribeText(AsrSession.clipFile(clip), "clip:${clip.id}")
|
| 123 |
+
val clipOk = clipText != null && TextMatch.matches(EXPECTED.getValue(clip.id), clipText.answer.text)
|
| 124 |
+
report(
|
| 125 |
+
"permission",
|
| 126 |
+
deniedAtStart && sentence == MicRecorder.PERMISSION_MESSAGE && clipOk,
|
| 127 |
+
profile,
|
| 128 |
+
"denied_at_start" to deniedAtStart,
|
| 129 |
+
"message" to sentence.ifEmpty { result.toString() },
|
| 130 |
+
"clip_en_while_denied" to clipOk,
|
| 131 |
+
)
|
| 132 |
+
}
|
| 133 |
+
|
| 134 |
+
private fun clipSteps(profile: ModelProfile, step: String) {
|
| 135 |
+
for (clip in Clip.ALL) {
|
| 136 |
+
val expected = EXPECTED.getValue(clip.id)
|
| 137 |
+
val transcript = transcribeText(AsrSession.clipFile(clip), "clip:${clip.id}")
|
| 138 |
+
if (transcript == null) {
|
| 139 |
+
report(step, false, profile, "clip" to clip.label, "reason" to "no transcript")
|
| 140 |
+
continue
|
| 141 |
+
}
|
| 142 |
+
val text = transcript.answer.text
|
| 143 |
+
val match = TextMatch.matches(expected, text)
|
| 144 |
+
report(
|
| 145 |
+
step,
|
| 146 |
+
match,
|
| 147 |
+
profile,
|
| 148 |
+
"clip" to clip.label,
|
| 149 |
+
"backend" to transcript.lmBackend,
|
| 150 |
+
"audio_s" to fmt(transcript.sentSeconds),
|
| 151 |
+
"send_s" to fmt(transcript.sendMs / 1000),
|
| 152 |
+
"rtf" to fmt(transcript.rtf, digits = 3),
|
| 153 |
+
"conversation_s" to fmt(transcript.createConversationMs / 1000),
|
| 154 |
+
"language" to transcript.answer.language.ifEmpty { "-" },
|
| 155 |
+
"match" to match,
|
| 156 |
+
"cer" to fmt(TextMatch.cer(expected, text), digits = 4),
|
| 157 |
+
"text" to text,
|
| 158 |
+
)
|
| 159 |
+
}
|
| 160 |
+
}
|
| 161 |
+
|
| 162 |
+
/** Clip en through the speaker, picked up by the microphone (the mic_test launch extras' path). */
|
| 163 |
+
private fun micStep(screen: MainActivity, profile: ModelProfile) {
|
| 164 |
+
val result = micTest(screen, durationMs = MIC_TEST_MS, play = true, waitS = 90)
|
| 165 |
+
val outcome = (result as? MicTestResult.Finished)?.outcome
|
| 166 |
+
val transcript = (outcome as? TranscribeOutcome.Text)?.transcript
|
| 167 |
+
if (transcript == null) {
|
| 168 |
+
report("mic", false, profile, "result" to result.toString())
|
| 169 |
+
return
|
| 170 |
+
}
|
| 171 |
+
val expected = EXPECTED.getValue(Clip.ALL[1].id)
|
| 172 |
+
val cer = TextMatch.cer(expected, transcript.answer.text)
|
| 173 |
+
report(
|
| 174 |
+
"mic",
|
| 175 |
+
cer <= MIC_MAX_CER,
|
| 176 |
+
profile,
|
| 177 |
+
"rec_s" to fmt(transcript.audioSeconds),
|
| 178 |
+
"send_s" to fmt(transcript.sendMs / 1000),
|
| 179 |
+
"rtf" to fmt(transcript.rtf, digits = 3),
|
| 180 |
+
"cer" to fmt(cer, digits = 4),
|
| 181 |
+
"language" to transcript.answer.language.ifEmpty { "-" },
|
| 182 |
+
"text" to transcript.answer.text,
|
| 183 |
+
)
|
| 184 |
+
}
|
| 185 |
+
|
| 186 |
+
private fun releaseStep(profile: ModelProfile, step: String = "release") {
|
| 187 |
+
val first = runCatching { AsrSession.release().get(TIMEOUT_S, TimeUnit.SECONDS) }
|
| 188 |
+
val second = runCatching { AsrSession.release().get(TIMEOUT_S, TimeUnit.SECONDS) }
|
| 189 |
+
report(
|
| 190 |
+
step,
|
| 191 |
+
first.getOrNull() == true && second.getOrNull() == false && AsrSession.current == null,
|
| 192 |
+
profile,
|
| 193 |
+
"first" to (first.getOrNull() ?: first.exceptionOrNull()),
|
| 194 |
+
"second" to (second.getOrNull() ?: second.exceptionOrNull()),
|
| 195 |
+
"vmrss_mb" to statusMb("VmRSS"),
|
| 196 |
+
)
|
| 197 |
+
}
|
| 198 |
+
|
| 199 |
+
/** A second load, now that the first wrote the runtime cache: the load time a user sees from then on. */
|
| 200 |
+
private fun reloadStep(profile: ModelProfile) {
|
| 201 |
+
val before = cacheFiles(profile)
|
| 202 |
+
val info = awaitLoad(AsrSession.load(profile), profile, step = "reload") ?: return
|
| 203 |
+
reportLoad("reload", profile, info, before)
|
| 204 |
+
AsrSession.release().get(TIMEOUT_S, TimeUnit.SECONDS)
|
| 205 |
+
}
|
| 206 |
+
|
| 207 |
+
private fun failurePaths(screen: MainActivity, present: List<ModelProfile>) {
|
| 208 |
+
// Switching: load one bundle, then pick another without releasing; the screen's path closes the first.
|
| 209 |
+
val to = present.first()
|
| 210 |
+
val from = present.firstOrNull { it != to }
|
| 211 |
+
if (from == null) {
|
| 212 |
+
// Not a result: the phone holds one bundle (with its runtime cache) at a time when storage is short.
|
| 213 |
+
Log.i(TAG, "NOTE step=switch not_run=\"one bundle on the phone\" model=${to.id}")
|
| 214 |
+
} else {
|
| 215 |
+
awaitLoad(onScreen(screen) { it.selectModel(from) }, from, step = "switch")
|
| 216 |
+
val switched = awaitLoad(onScreen(screen) { it.selectModel(to) }, to, step = "switch")
|
| 217 |
+
val clip = Clip.ALL[1]
|
| 218 |
+
val text = transcribeText(AsrSession.clipFile(clip), "clip:${clip.id}")?.answer?.text
|
| 219 |
+
report(
|
| 220 |
+
"switch",
|
| 221 |
+
switched != null && AsrSession.current?.profile == to && text != null &&
|
| 222 |
+
TextMatch.matches(EXPECTED.getValue(clip.id), text),
|
| 223 |
+
to,
|
| 224 |
+
"from" to from.id,
|
| 225 |
+
"load_s" to (switched?.let { fmt(it.loadMs / 1000) } ?: "-"),
|
| 226 |
+
"vmrss_mb" to statusMb("VmRSS"),
|
| 227 |
+
"text" to (text ?: "-"),
|
| 228 |
+
)
|
| 229 |
+
}
|
| 230 |
+
if (AsrSession.current == null) awaitLoad(AsrSession.load(to), to, step = "long-clip")
|
| 231 |
+
|
| 232 |
+
// Longer than one window: the app sends the first 30 s and does not crash.
|
| 233 |
+
val long = File(context.cacheDir, "long-check.wav")
|
| 234 |
+
val pcm = (Clip.ALL + Clip.ALL[1]).map { Wav.pcm16(AsrSession.clipFile(it)).data }
|
| 235 |
+
Wav.writeMono16(long, pcm.fold(ByteArray(0)) { all, part -> all + part }, MicRecorder.SAMPLE_RATE)
|
| 236 |
+
val longText = transcribeText(long, "long-clip")
|
| 237 |
+
report(
|
| 238 |
+
"long-clip",
|
| 239 |
+
longText != null && longText.audioSeconds > 31.0 && kotlin.math.abs(longText.sentSeconds - 30.0) < 0.01 &&
|
| 240 |
+
longText.answer.text.isNotEmpty(),
|
| 241 |
+
AsrSession.current?.profile,
|
| 242 |
+
"audio_s" to fmt(longText?.audioSeconds ?: Wav.seconds(long)),
|
| 243 |
+
"sent_s" to (longText?.let { fmt(it.sentSeconds) } ?: "-"),
|
| 244 |
+
"send_s" to (longText?.let { fmt(it.sendMs / 1000) } ?: "-"),
|
| 245 |
+
"text" to (longText?.answer?.text ?: "-"),
|
| 246 |
+
)
|
| 247 |
+
|
| 248 |
+
// A silent recording: "no speech", and the model is not called.
|
| 249 |
+
val silent = File(context.cacheDir, "silence-check.wav")
|
| 250 |
+
Wav.writeMono16(silent, ByteArray(MicRecorder.SAMPLE_RATE * 2 * 3), MicRecorder.SAMPLE_RATE)
|
| 251 |
+
val silence = runCatching { AsrSession.transcribe(silent, "silence").get(TIMEOUT_S, TimeUnit.SECONDS) }
|
| 252 |
+
report(
|
| 253 |
+
"silence",
|
| 254 |
+
silence.getOrNull() is TranscribeOutcome.NoSpeech,
|
| 255 |
+
AsrSession.current?.profile,
|
| 256 |
+
"result" to (silence.getOrNull() ?: silence.exceptionOrNull()),
|
| 257 |
+
)
|
| 258 |
+
|
| 259 |
+
// A bundle that is not on the phone: a sentence naming the file and where to push it; the loaded model stays.
|
| 260 |
+
val loadedBefore = AsrSession.current?.profile
|
| 261 |
+
val ghost = ModelProfile.QWEN3_ASR_1_7B.copy(fileName = "not-on-this-phone.litertlm")
|
| 262 |
+
val missing =
|
| 263 |
+
runCatching { AsrSession.load(ghost, bundle = AsrSession.bundleFile(ghost)).get(TIMEOUT_S, TimeUnit.SECONDS) }
|
| 264 |
+
val message = ((missing.exceptionOrNull() as? ExecutionException)?.cause as? AsrException)?.message.orEmpty()
|
| 265 |
+
report(
|
| 266 |
+
"missing-bundle",
|
| 267 |
+
missing.isFailure && message.contains("not-on-this-phone.litertlm") &&
|
| 268 |
+
message.contains(AsrSession.modelDir.path) && AsrSession.current?.profile == loadedBefore,
|
| 269 |
+
loadedBefore,
|
| 270 |
+
"message" to message.ifEmpty { missing.toString() },
|
| 271 |
+
"still_loaded" to (AsrSession.current?.profile?.id ?: "none"),
|
| 272 |
+
)
|
| 273 |
+
releaseStep(loadedBefore ?: to, step = "release-final")
|
| 274 |
+
long.delete()
|
| 275 |
+
silent.delete()
|
| 276 |
+
}
|
| 277 |
+
|
| 278 |
+
// ---- helpers ------------------------------------------------------------------------------------------------
|
| 279 |
+
|
| 280 |
+
private fun launchIntent(profile: ModelProfile) =
|
| 281 |
+
Intent(context, MainActivity::class.java)
|
| 282 |
+
.putExtra(MainActivity.EXTRA_MODEL, profile.id)
|
| 283 |
+
.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
|
| 284 |
+
|
| 285 |
+
/** Starts the screen in the foreground (the microphone records only for a visible app) and waits until it runs. */
|
| 286 |
+
private fun launch(intent: Intent): MainActivity = instrumentation.startActivitySync(intent) as MainActivity
|
| 287 |
+
|
| 288 |
+
private fun close(screen: MainActivity) {
|
| 289 |
+
instrumentation.runOnMainSync { screen.finish() }
|
| 290 |
+
instrumentation.waitForIdleSync()
|
| 291 |
+
}
|
| 292 |
+
|
| 293 |
+
private fun <T> onScreen(screen: MainActivity, block: (MainActivity) -> T): T {
|
| 294 |
+
val value = AtomicReference<T>()
|
| 295 |
+
instrumentation.runOnMainSync { value.set(block(screen)) }
|
| 296 |
+
return value.get()
|
| 297 |
+
}
|
| 298 |
+
|
| 299 |
+
private fun awaitLoad(future: Future<LoadInfo>?, profile: ModelProfile, step: String = "load"): LoadInfo? {
|
| 300 |
+
if (future == null) {
|
| 301 |
+
report(step, false, profile, "reason" to "the screen did not load it")
|
| 302 |
+
return null
|
| 303 |
+
}
|
| 304 |
+
return try {
|
| 305 |
+
future.get(LOAD_TIMEOUT_S, TimeUnit.SECONDS)
|
| 306 |
+
} catch (e: ExecutionException) {
|
| 307 |
+
report(step, false, profile, "message" to (e.cause?.message ?: e.toString()))
|
| 308 |
+
null
|
| 309 |
+
}
|
| 310 |
+
}
|
| 311 |
+
|
| 312 |
+
private fun transcribeText(wav: File, source: String): Transcript? =
|
| 313 |
+
try {
|
| 314 |
+
(AsrSession.transcribe(wav, source).get(TIMEOUT_S, TimeUnit.SECONDS) as? TranscribeOutcome.Text)?.transcript
|
| 315 |
+
} catch (e: ExecutionException) {
|
| 316 |
+
Log.e(TAG, "transcription failed: $source", e.cause)
|
| 317 |
+
null
|
| 318 |
+
}
|
| 319 |
+
|
| 320 |
+
private fun micTest(screen: MainActivity, durationMs: Long, play: Boolean, waitS: Long): MicTestResult? {
|
| 321 |
+
val done = CountDownLatch(1)
|
| 322 |
+
val result = AtomicReference<MicTestResult>()
|
| 323 |
+
instrumentation.runOnMainSync {
|
| 324 |
+
screen.runMicTest(durationMs, play) {
|
| 325 |
+
result.set(it)
|
| 326 |
+
done.countDown()
|
| 327 |
+
}
|
| 328 |
+
}
|
| 329 |
+
done.await(waitS, TimeUnit.SECONDS)
|
| 330 |
+
return result.get()
|
| 331 |
+
}
|
| 332 |
+
|
| 333 |
+
private fun reportLoad(step: String, profile: ModelProfile, info: LoadInfo, before: Map<String, CacheFile>) {
|
| 334 |
+
report(
|
| 335 |
+
step,
|
| 336 |
+
true,
|
| 337 |
+
profile,
|
| 338 |
+
"backend" to info.lmBackend,
|
| 339 |
+
"load_s" to fmt(info.loadMs / 1000),
|
| 340 |
+
"initialize_s" to fmt(info.initializeMs / 1000),
|
| 341 |
+
"first_conversation_s" to fmt(info.firstConversationMs / 1000),
|
| 342 |
+
"cache" to cacheState(profile, before),
|
| 343 |
+
"cache_mb" to cacheFiles(profile).values.sumOf { it.bytes } / 1_000_000,
|
| 344 |
+
"threads" to AsrEngine.CPU_THREADS,
|
| 345 |
+
"vmhwm_mb" to statusMb("VmHWM"),
|
| 346 |
+
"vmrss_mb" to statusMb("VmRSS"),
|
| 347 |
+
"mem_available_mb" to memAvailableMb(),
|
| 348 |
+
)
|
| 349 |
+
}
|
| 350 |
+
|
| 351 |
+
/** One runtime cache file: the runtime names them after the bundle file. */
|
| 352 |
+
private data class CacheFile(val bytes: Long, val modified: Long)
|
| 353 |
+
|
| 354 |
+
private fun cacheFiles(profile: ModelProfile): Map<String, CacheFile> =
|
| 355 |
+
context.cacheDir
|
| 356 |
+
.listFiles { file -> file.name.startsWith(profile.fileName) }
|
| 357 |
+
.orEmpty()
|
| 358 |
+
.associate { it.name to CacheFile(it.length(), it.lastModified()) }
|
| 359 |
+
|
| 360 |
+
/** "loaded" when the load left the cache files as they were, "written" when it created or rewrote them. */
|
| 361 |
+
private fun cacheState(profile: ModelProfile, before: Map<String, CacheFile>): String =
|
| 362 |
+
if (cacheFiles(profile) == before && before.isNotEmpty()) "loaded" else "written"
|
| 363 |
+
|
| 364 |
+
private fun hasMicPermission(): Boolean =
|
| 365 |
+
context.checkSelfPermission(Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED
|
| 366 |
+
|
| 367 |
+
private fun statusMb(key: String): Long =
|
| 368 |
+
File("/proc/self/status").readLines().firstOrNull { it.startsWith("$key:") }
|
| 369 |
+
?.trim()?.split(Regex("\\s+"))?.getOrNull(1)?.toLongOrNull()?.div(1024) ?: -1
|
| 370 |
+
|
| 371 |
+
private fun memAvailableMb(): Long =
|
| 372 |
+
File("/proc/meminfo").readLines().firstOrNull { it.startsWith("MemAvailable:") }
|
| 373 |
+
?.trim()?.split(Regex("\\s+"))?.getOrNull(1)?.toLongOrNull()?.div(1024) ?: -1
|
| 374 |
+
|
| 375 |
+
private fun report(step: String, ok: Boolean, profile: ModelProfile?, vararg fields: Pair<String, Any?>) {
|
| 376 |
+
outcomes += ok
|
| 377 |
+
val line = StringBuilder("RESULT step=$step ok=$ok model=${profile?.id ?: "-"}")
|
| 378 |
+
for ((key, value) in fields) line.append(' ').append(key).append('=').append(quote(value.toString()))
|
| 379 |
+
Log.i(TAG, line.toString())
|
| 380 |
+
}
|
| 381 |
+
|
| 382 |
+
private fun finish(scope: String) {
|
| 383 |
+
val all = outcomes.isNotEmpty() && outcomes.all { it }
|
| 384 |
+
Log.i(TAG, "RESULT ok=$all scope=$scope steps=${outcomes.size} failed=${outcomes.count { !it }}")
|
| 385 |
+
assertTrue("a device check step failed (logcat -s $TAG)", all)
|
| 386 |
+
}
|
| 387 |
+
|
| 388 |
+
private fun quote(value: String): String =
|
| 389 |
+
if (value.isNotEmpty() && value.none { it.isWhitespace() || it == '"' }) value
|
| 390 |
+
else "\"" + value.replace("\\", "\\\\").replace("\"", "\\\"").replace("\n", "\\n") + "\""
|
| 391 |
+
|
| 392 |
+
private fun fmt(value: Double, digits: Int = 2) = String.format(Locale.US, "%.${digits}f", value)
|
| 393 |
+
|
| 394 |
+
private companion object {
|
| 395 |
+
const val TAG = "asr-check"
|
| 396 |
+
const val LOAD_TIMEOUT_S = 300L
|
| 397 |
+
const val TIMEOUT_S = 120L
|
| 398 |
+
const val MIC_TEST_MS = 9000L
|
| 399 |
+
const val MIC_MAX_CER = 0.15
|
| 400 |
+
|
| 401 |
+
/** Bundles in the order the check loads them. */
|
| 402 |
+
val ORDER =
|
| 403 |
+
listOf(ModelProfile.QWEN3_ASR_1_7B, ModelProfile.CONFUCIUS4_R2T2, ModelProfile.FUN_ASR_NANO_2512)
|
| 404 |
+
|
| 405 |
+
/**
|
| 406 |
+
* The expected sentences: Qwen3-ASR-1.7B's answers on the Mac CPU (litert-lm-api 0.17.1), which the S26 app runs
|
| 407 |
+
* of 2026-10-03 matched. Fun-ASR-Nano-2512 and Confucius4-R2T2 give the same sentences on the Mac CPU.
|
| 408 |
+
*/
|
| 409 |
+
val EXPECTED =
|
| 410 |
+
mapOf(
|
| 411 |
+
"fleurs_cmn_1698" to "宇宙中的一切都由物质构成,而所有的物质都由被称为原子的微小颗粒组成。",
|
| 412 |
+
"fleurs_en_1698" to
|
| 413 |
+
"Everything in the universe is made of matter. All matter is made of tiny particles called atoms.",
|
| 414 |
+
"fleurs_ja_1698" to "宇宙に存在するあらゆるものは物質でできています。あらゆる物質は原子と呼ばれる微小な粒子でできています。",
|
| 415 |
+
)
|
| 416 |
+
}
|
| 417 |
+
}
|
android/app/src/main/AndroidManifest.xml
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
|
| 3 |
+
|
| 4 |
+
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
| 5 |
+
|
| 6 |
+
<application
|
| 7 |
+
android:allowBackup="false"
|
| 8 |
+
android:label="@string/app_name"
|
| 9 |
+
android:largeHeap="true"
|
| 10 |
+
android:theme="@style/Theme.AsrLitertlm">
|
| 11 |
+
|
| 12 |
+
<!-- Only for the language model on the GPU (launch extra backend=gpu); the CPU path does not need it. -->
|
| 13 |
+
<uses-native-library
|
| 14 |
+
android:name="libOpenCL.so"
|
| 15 |
+
android:required="false" />
|
| 16 |
+
|
| 17 |
+
<activity
|
| 18 |
+
android:name=".MainActivity"
|
| 19 |
+
android:exported="true"
|
| 20 |
+
android:launchMode="singleTask"
|
| 21 |
+
android:screenOrientation="portrait">
|
| 22 |
+
<intent-filter>
|
| 23 |
+
<action android:name="android.intent.action.MAIN" />
|
| 24 |
+
<category android:name="android.intent.category.LAUNCHER" />
|
| 25 |
+
</intent-filter>
|
| 26 |
+
</activity>
|
| 27 |
+
</application>
|
| 28 |
+
</manifest>
|
android/app/src/main/kotlin/com/asrlitertlm/AsrEngine.kt
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import android.os.SystemClock
|
| 4 |
+
import com.google.ai.edge.litertlm.Backend
|
| 5 |
+
import com.google.ai.edge.litertlm.Content
|
| 6 |
+
import com.google.ai.edge.litertlm.Contents
|
| 7 |
+
import com.google.ai.edge.litertlm.ConversationConfig
|
| 8 |
+
import com.google.ai.edge.litertlm.Engine
|
| 9 |
+
import com.google.ai.edge.litertlm.EngineConfig
|
| 10 |
+
import com.google.ai.edge.litertlm.Message
|
| 11 |
+
import com.google.ai.edge.litertlm.SamplerConfig
|
| 12 |
+
import java.io.File
|
| 13 |
+
|
| 14 |
+
/** Which processor runs the language model. The audio encoder always runs on the CPU (the GPU delegate rejects it). */
|
| 15 |
+
enum class LmBackend {
|
| 16 |
+
CPU,
|
| 17 |
+
GPU,
|
| 18 |
+
}
|
| 19 |
+
|
| 20 |
+
/** A failure the app shows as a sentence instead of crashing. */
|
| 21 |
+
class AsrException(message: String, cause: Throwable? = null) : Exception(message, cause)
|
| 22 |
+
|
| 23 |
+
/** What one load cost. "Engine load" = Engine.initialize() plus the first conversation, which creates the encoder. */
|
| 24 |
+
data class LoadInfo(
|
| 25 |
+
val profile: ModelProfile,
|
| 26 |
+
val bundle: File,
|
| 27 |
+
val bundleBytes: Long,
|
| 28 |
+
val lmBackend: LmBackend,
|
| 29 |
+
val initializeMs: Double,
|
| 30 |
+
val firstConversationMs: Double,
|
| 31 |
+
) {
|
| 32 |
+
val loadMs: Double
|
| 33 |
+
get() = initializeMs + firstConversationMs
|
| 34 |
+
}
|
| 35 |
+
|
| 36 |
+
/** One transcription. [sendMs] is the wall time of sendMessage(), from sending the clip to the full answer. */
|
| 37 |
+
data class Transcript(
|
| 38 |
+
val profile: ModelProfile,
|
| 39 |
+
val lmBackend: LmBackend,
|
| 40 |
+
val raw: String,
|
| 41 |
+
val answer: Answer,
|
| 42 |
+
/** Length of the audio the app received. */
|
| 43 |
+
val audioSeconds: Double,
|
| 44 |
+
/** Length of the audio sent to the model (at most [AsrEngine.MAX_AUDIO_SECONDS]). */
|
| 45 |
+
val sentSeconds: Double,
|
| 46 |
+
val sendMs: Double,
|
| 47 |
+
val createConversationMs: Double,
|
| 48 |
+
) {
|
| 49 |
+
/** Real-time factor: transcription time over the length of the audio the model heard. */
|
| 50 |
+
val rtf: Double
|
| 51 |
+
get() = sendMs / 1000.0 / sentSeconds
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
/**
|
| 55 |
+
* One loaded `.litertlm` bundle and the calls every model in [ModelProfile.ALL] shares: an Engine with the language
|
| 56 |
+
* model on the CPU (or the GPU) and the audio encoder on the CPU, a new Conversation per clip, one user message that
|
| 57 |
+
* holds only the audio file, greedy sampling.
|
| 58 |
+
*
|
| 59 |
+
* Not thread-safe: the app calls it from one worker thread ([AsrSession]).
|
| 60 |
+
*/
|
| 61 |
+
class AsrEngine(private val cacheDir: File) : AutoCloseable {
|
| 62 |
+
private var engine: Engine? = null
|
| 63 |
+
|
| 64 |
+
/** The bundle in memory, or null. */
|
| 65 |
+
var loaded: LoadInfo? = null
|
| 66 |
+
private set
|
| 67 |
+
|
| 68 |
+
/**
|
| 69 |
+
* Loads [profile] from [bundle], closing the engine loaded before. A missing file or a runtime failure becomes an
|
| 70 |
+
* [AsrException] with a sentence for the screen; a missing file leaves the loaded engine as it was.
|
| 71 |
+
*/
|
| 72 |
+
fun load(profile: ModelProfile, bundle: File, lmBackend: LmBackend = LmBackend.CPU): LoadInfo {
|
| 73 |
+
if (!bundle.isFile) throw AsrException(missingBundleMessage(profile, bundle.parentFile))
|
| 74 |
+
// One engine at a time: two 2.7 GB bundles in memory at once would cross what the phone gives one app.
|
| 75 |
+
release()
|
| 76 |
+
val lm = if (lmBackend == LmBackend.GPU) Backend.GPU() else Backend.CPU(threadCount = CPU_THREADS)
|
| 77 |
+
val config =
|
| 78 |
+
EngineConfig(
|
| 79 |
+
modelPath = bundle.path,
|
| 80 |
+
backend = lm,
|
| 81 |
+
audioBackend = Backend.CPU(threadCount = CPU_THREADS),
|
| 82 |
+
cacheDir = cacheDir.path,
|
| 83 |
+
)
|
| 84 |
+
val newEngine = Engine(config)
|
| 85 |
+
val start = SystemClock.elapsedRealtimeNanos()
|
| 86 |
+
val initialized: Long
|
| 87 |
+
try {
|
| 88 |
+
newEngine.initialize()
|
| 89 |
+
initialized = SystemClock.elapsedRealtimeNanos()
|
| 90 |
+
// The runtime creates the audio encoder with the first conversation; do it here so it counts as loading.
|
| 91 |
+
newEngine.createConversation(conversationConfig(profile)).close()
|
| 92 |
+
} catch (t: Throwable) {
|
| 93 |
+
if (newEngine.isInitialized()) runCatching { newEngine.close() }
|
| 94 |
+
throw AsrException("Could not load ${profile.fileName} (${lmBackend.name}): ${t.message ?: t}", t)
|
| 95 |
+
}
|
| 96 |
+
val ready = SystemClock.elapsedRealtimeNanos()
|
| 97 |
+
engine = newEngine
|
| 98 |
+
return LoadInfo(
|
| 99 |
+
profile = profile,
|
| 100 |
+
bundle = bundle,
|
| 101 |
+
bundleBytes = bundle.length(),
|
| 102 |
+
lmBackend = lmBackend,
|
| 103 |
+
initializeMs = (initialized - start) / 1e6,
|
| 104 |
+
firstConversationMs = (ready - initialized) / 1e6,
|
| 105 |
+
)
|
| 106 |
+
.also { loaded = it }
|
| 107 |
+
}
|
| 108 |
+
|
| 109 |
+
/**
|
| 110 |
+
* Transcribes a 16 kHz mono 16-bit wav. Audio longer than [MAX_AUDIO_SECONDS] is cut there first ([scratch] holds
|
| 111 |
+
* the cut copy): each model reads one window of about 30 s per message.
|
| 112 |
+
*/
|
| 113 |
+
fun transcribe(wav: File, scratch: File): Transcript {
|
| 114 |
+
val currentEngine = engine ?: throw AsrException("No model is loaded.")
|
| 115 |
+
val info = loaded ?: throw AsrException("No model is loaded.")
|
| 116 |
+
val audioSeconds = Wav.seconds(wav)
|
| 117 |
+
val sent = Wav.trimmed(wav, MAX_AUDIO_SECONDS, scratch)
|
| 118 |
+
val sentSeconds = if (sent === wav) audioSeconds else Wav.seconds(sent)
|
| 119 |
+
val created: Long
|
| 120 |
+
val sendStart: Long
|
| 121 |
+
val sendEnd: Long
|
| 122 |
+
val raw: String
|
| 123 |
+
val start = SystemClock.elapsedRealtimeNanos()
|
| 124 |
+
try {
|
| 125 |
+
currentEngine.createConversation(conversationConfig(info.profile)).use { conversation ->
|
| 126 |
+
created = SystemClock.elapsedRealtimeNanos()
|
| 127 |
+
val message = Message.user(Contents.of(Content.AudioFile(sent.absolutePath)))
|
| 128 |
+
sendStart = SystemClock.elapsedRealtimeNanos()
|
| 129 |
+
raw = conversation.sendMessage(message).toString()
|
| 130 |
+
sendEnd = SystemClock.elapsedRealtimeNanos()
|
| 131 |
+
}
|
| 132 |
+
} catch (t: Throwable) {
|
| 133 |
+
throw AsrException("${info.profile.displayName} could not transcribe this clip: ${t.message ?: t}", t)
|
| 134 |
+
}
|
| 135 |
+
return Transcript(
|
| 136 |
+
profile = info.profile,
|
| 137 |
+
lmBackend = info.lmBackend,
|
| 138 |
+
raw = raw,
|
| 139 |
+
answer = info.profile.parse(raw),
|
| 140 |
+
audioSeconds = audioSeconds,
|
| 141 |
+
sentSeconds = sentSeconds,
|
| 142 |
+
sendMs = (sendEnd - sendStart) / 1e6,
|
| 143 |
+
createConversationMs = (created - start) / 1e6,
|
| 144 |
+
)
|
| 145 |
+
}
|
| 146 |
+
|
| 147 |
+
/**
|
| 148 |
+
* Closes the engine and returns true, or returns false when none is loaded. Safe to call twice: the runtime's own
|
| 149 |
+
* Engine.close() throws on a second call, so the reference is dropped before closing.
|
| 150 |
+
*/
|
| 151 |
+
fun release(): Boolean {
|
| 152 |
+
val current = engine ?: return false
|
| 153 |
+
engine = null
|
| 154 |
+
loaded = null
|
| 155 |
+
current.close()
|
| 156 |
+
return true
|
| 157 |
+
}
|
| 158 |
+
|
| 159 |
+
override fun close() {
|
| 160 |
+
release()
|
| 161 |
+
}
|
| 162 |
+
|
| 163 |
+
companion object {
|
| 164 |
+
/** CPU threads for the language model and the audio encoder (the model cards' Galaxy S26 rows). */
|
| 165 |
+
const val CPU_THREADS = 4
|
| 166 |
+
|
| 167 |
+
/** The longest audio sent in one message; longer audio is cut here. */
|
| 168 |
+
const val MAX_AUDIO_SECONDS = 30.0
|
| 169 |
+
|
| 170 |
+
fun conversationConfig(profile: ModelProfile) =
|
| 171 |
+
ConversationConfig(
|
| 172 |
+
samplerConfig = SamplerConfig(topK = 1, topP = 1.0, temperature = 0.0),
|
| 173 |
+
maxOutputToken = profile.maxOutputTokens,
|
| 174 |
+
)
|
| 175 |
+
|
| 176 |
+
/** The sentence the picker and a failed load show for a bundle that is not on the phone. */
|
| 177 |
+
fun missingBundleMessage(profile: ModelProfile, directory: File?): String =
|
| 178 |
+
"${profile.fileName} is not on this phone. Download it from huggingface.co/${profile.hubRepo} and push it " +
|
| 179 |
+
"to ${directory?.path ?: "the app's files directory"}/ with adb (see README)."
|
| 180 |
+
}
|
| 181 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/AsrSession.kt
ADDED
|
@@ -0,0 +1,248 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import android.content.Context
|
| 4 |
+
import android.os.Handler
|
| 5 |
+
import android.os.Looper
|
| 6 |
+
import android.util.Log
|
| 7 |
+
import java.io.File
|
| 8 |
+
import java.io.IOException
|
| 9 |
+
import java.util.Locale
|
| 10 |
+
import java.util.concurrent.Callable
|
| 11 |
+
import java.util.concurrent.CopyOnWriteArrayList
|
| 12 |
+
import java.util.concurrent.ExecutorService
|
| 13 |
+
import java.util.concurrent.Executors
|
| 14 |
+
import java.util.concurrent.Future
|
| 15 |
+
|
| 16 |
+
/** One bundled clip: FLEURS test sentence 1698 read in one language (google/fleurs, CC BY 4.0). */
|
| 17 |
+
data class Clip(val id: String, val label: String, val rawRes: Int) {
|
| 18 |
+
companion object {
|
| 19 |
+
val ALL =
|
| 20 |
+
listOf(
|
| 21 |
+
Clip("fleurs_cmn_1698", "zh", R.raw.fleurs_cmn_1698),
|
| 22 |
+
Clip("fleurs_en_1698", "en", R.raw.fleurs_en_1698),
|
| 23 |
+
Clip("fleurs_ja_1698", "ja", R.raw.fleurs_ja_1698),
|
| 24 |
+
)
|
| 25 |
+
}
|
| 26 |
+
}
|
| 27 |
+
|
| 28 |
+
/** The end of one transcription request. */
|
| 29 |
+
sealed interface TranscribeOutcome {
|
| 30 |
+
data class Text(val transcript: Transcript) : TranscribeOutcome
|
| 31 |
+
|
| 32 |
+
/** The audio was close to digital silence, so the model was not called. */
|
| 33 |
+
data class NoSpeech(val levels: Wav.Levels, val seconds: Double) : TranscribeOutcome
|
| 34 |
+
}
|
| 35 |
+
|
| 36 |
+
/** What the session did, delivered to listeners on the main thread. */
|
| 37 |
+
sealed interface AsrEvent {
|
| 38 |
+
data class Loading(val profile: ModelProfile, val lmBackend: LmBackend) : AsrEvent
|
| 39 |
+
|
| 40 |
+
data class Loaded(val info: LoadInfo) : AsrEvent
|
| 41 |
+
|
| 42 |
+
data class LoadFailed(val profile: ModelProfile, val message: String) : AsrEvent
|
| 43 |
+
|
| 44 |
+
data class Transcribing(val source: String) : AsrEvent
|
| 45 |
+
|
| 46 |
+
data class Transcribed(val source: String, val outcome: TranscribeOutcome) : AsrEvent
|
| 47 |
+
|
| 48 |
+
data class TranscribeFailed(val source: String, val message: String) : AsrEvent
|
| 49 |
+
|
| 50 |
+
data class Released(val profile: ModelProfile) : AsrEvent
|
| 51 |
+
}
|
| 52 |
+
|
| 53 |
+
/**
|
| 54 |
+
* The app's one engine, shared by the screen and the device check. Every model call runs on one worker thread (an
|
| 55 |
+
* Engine is not called from two threads), and one bundle is in memory at a time. The methods return futures for
|
| 56 |
+
* callers that wait (the device check) and publish [AsrEvent]s for the screen.
|
| 57 |
+
*/
|
| 58 |
+
object AsrSession {
|
| 59 |
+
private const val TAG = "AsrLitertlm"
|
| 60 |
+
|
| 61 |
+
/** Recordings quieter than this (RMS) are reported as no speech instead of being sent to the model. */
|
| 62 |
+
const val NO_SPEECH_RMS_DBFS = -60.0
|
| 63 |
+
|
| 64 |
+
private val worker: ExecutorService = Executors.newSingleThreadExecutor { Thread(it, "asr-worker") }
|
| 65 |
+
private val main = Handler(Looper.getMainLooper())
|
| 66 |
+
private val listeners = CopyOnWriteArrayList<(AsrEvent) -> Unit>()
|
| 67 |
+
|
| 68 |
+
@Volatile private var engine: AsrEngine? = null
|
| 69 |
+
private lateinit var modelDirectory: File
|
| 70 |
+
private lateinit var cacheDirectory: File
|
| 71 |
+
private lateinit var appContext: Context
|
| 72 |
+
|
| 73 |
+
/** The bundle in memory, or null. */
|
| 74 |
+
@Volatile
|
| 75 |
+
var current: LoadInfo? = null
|
| 76 |
+
private set
|
| 77 |
+
|
| 78 |
+
/** The last load request, so that a caller can wait for a load the screen started. */
|
| 79 |
+
@Volatile
|
| 80 |
+
var lastLoad: Future<LoadInfo>? = null
|
| 81 |
+
private set
|
| 82 |
+
|
| 83 |
+
/** Where the app looks for the bundles: `/sdcard/Android/data/com.asrlitertlm/files`. */
|
| 84 |
+
val modelDir: File
|
| 85 |
+
get() = modelDirectory
|
| 86 |
+
|
| 87 |
+
fun init(context: Context) {
|
| 88 |
+
synchronized(this) {
|
| 89 |
+
if (engine != null) return
|
| 90 |
+
appContext = context.applicationContext
|
| 91 |
+
modelDirectory = appContext.getExternalFilesDir(null) ?: appContext.filesDir
|
| 92 |
+
cacheDirectory = appContext.cacheDir
|
| 93 |
+
engine = AsrEngine(cacheDirectory)
|
| 94 |
+
}
|
| 95 |
+
}
|
| 96 |
+
|
| 97 |
+
fun bundleFile(profile: ModelProfile): File = File(modelDirectory, profile.fileName)
|
| 98 |
+
|
| 99 |
+
fun isPresent(profile: ModelProfile): Boolean = bundleFile(profile).isFile
|
| 100 |
+
|
| 101 |
+
/** The clip as a wav file the runtime can open (copied out of the APK once). */
|
| 102 |
+
fun clipFile(clip: Clip): File {
|
| 103 |
+
val file = File(cacheDirectory, "${clip.id}.wav")
|
| 104 |
+
if (!file.isFile) {
|
| 105 |
+
val partial = File(cacheDirectory, "${clip.id}.wav.part")
|
| 106 |
+
appContext.resources.openRawResource(clip.rawRes).use { input ->
|
| 107 |
+
partial.outputStream().use { input.copyTo(it) }
|
| 108 |
+
}
|
| 109 |
+
check(partial.renameTo(file)) { "could not write $file" }
|
| 110 |
+
}
|
| 111 |
+
return file
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
fun addListener(listener: (AsrEvent) -> Unit) {
|
| 115 |
+
listeners.add(listener)
|
| 116 |
+
}
|
| 117 |
+
|
| 118 |
+
fun removeListener(listener: (AsrEvent) -> Unit) {
|
| 119 |
+
listeners.remove(listener)
|
| 120 |
+
}
|
| 121 |
+
|
| 122 |
+
/** Loads [profile] (closing the bundle loaded before). [bundle] defaults to the file in [modelDir]. */
|
| 123 |
+
fun load(
|
| 124 |
+
profile: ModelProfile,
|
| 125 |
+
lmBackend: LmBackend = LmBackend.CPU,
|
| 126 |
+
bundle: File = bundleFile(profile),
|
| 127 |
+
): Future<LoadInfo> =
|
| 128 |
+
worker
|
| 129 |
+
.submit(
|
| 130 |
+
Callable {
|
| 131 |
+
publish(AsrEvent.Loading(profile, lmBackend))
|
| 132 |
+
val asr = checkNotNull(engine) { "AsrSession.init() was not called" }
|
| 133 |
+
try {
|
| 134 |
+
asr.load(profile, bundle, lmBackend).also { info ->
|
| 135 |
+
current = info
|
| 136 |
+
Log.i(
|
| 137 |
+
TAG,
|
| 138 |
+
String.format(
|
| 139 |
+
Locale.US,
|
| 140 |
+
"LOAD model=%s backend=%s load_ms=%.1f initialize_ms=%.1f first_conversation_ms=%.1f bytes=%d",
|
| 141 |
+
profile.id,
|
| 142 |
+
lmBackend,
|
| 143 |
+
info.loadMs,
|
| 144 |
+
info.initializeMs,
|
| 145 |
+
info.firstConversationMs,
|
| 146 |
+
info.bundleBytes,
|
| 147 |
+
),
|
| 148 |
+
)
|
| 149 |
+
publish(AsrEvent.Loaded(info))
|
| 150 |
+
}
|
| 151 |
+
} catch (e: AsrException) {
|
| 152 |
+
current = asr.loaded
|
| 153 |
+
Log.e(TAG, "LOAD_FAILED model=${profile.id} backend=$lmBackend: ${e.message}", e.cause)
|
| 154 |
+
publish(AsrEvent.LoadFailed(profile, e.message ?: e.toString()))
|
| 155 |
+
throw e
|
| 156 |
+
}
|
| 157 |
+
}
|
| 158 |
+
)
|
| 159 |
+
.also { lastLoad = it }
|
| 160 |
+
|
| 161 |
+
/**
|
| 162 |
+
* Transcribes a 16 kHz mono 16-bit wav with the loaded bundle; [source] names the input in events and logs
|
| 163 |
+
* ("clip:<id>", "mic"). Near-silent audio returns [TranscribeOutcome.NoSpeech] without calling the model.
|
| 164 |
+
*/
|
| 165 |
+
fun transcribe(wav: File, source: String): Future<TranscribeOutcome> =
|
| 166 |
+
worker.submit(
|
| 167 |
+
Callable {
|
| 168 |
+
publish(AsrEvent.Transcribing(source))
|
| 169 |
+
val asr = checkNotNull(engine) { "AsrSession.init() was not called" }
|
| 170 |
+
try {
|
| 171 |
+
val pcm =
|
| 172 |
+
try {
|
| 173 |
+
Wav.pcm16(wav)
|
| 174 |
+
} catch (e: IOException) {
|
| 175 |
+
throw AsrException("Could not read ${wav.name}: ${e.message}", e)
|
| 176 |
+
} catch (e: RuntimeException) { // not a wav, no data chunk, not 16-bit
|
| 177 |
+
throw AsrException("Could not read ${wav.name}: ${e.message}", e)
|
| 178 |
+
}
|
| 179 |
+
val levels = Wav.levels(pcm.data)
|
| 180 |
+
val outcome =
|
| 181 |
+
if (levels.rmsDbfs < NO_SPEECH_RMS_DBFS) {
|
| 182 |
+
TranscribeOutcome.NoSpeech(levels, pcm.seconds)
|
| 183 |
+
} else {
|
| 184 |
+
TranscribeOutcome.Text(asr.transcribe(wav, File(cacheDirectory, "sent.wav")))
|
| 185 |
+
}
|
| 186 |
+
log(source, outcome)
|
| 187 |
+
publish(AsrEvent.Transcribed(source, outcome))
|
| 188 |
+
outcome
|
| 189 |
+
} catch (e: AsrException) {
|
| 190 |
+
Log.e(TAG, "TRANSCRIBE_FAILED source=$source: ${e.message}", e.cause)
|
| 191 |
+
publish(AsrEvent.TranscribeFailed(source, e.message ?: e.toString()))
|
| 192 |
+
throw e
|
| 193 |
+
}
|
| 194 |
+
}
|
| 195 |
+
)
|
| 196 |
+
|
| 197 |
+
/** Closes the loaded bundle; the future holds false when none was loaded (a second call is harmless). */
|
| 198 |
+
fun release(): Future<Boolean> =
|
| 199 |
+
worker.submit(
|
| 200 |
+
Callable {
|
| 201 |
+
val profile = current?.profile
|
| 202 |
+
val closed = checkNotNull(engine) { "AsrSession.init() was not called" }.release()
|
| 203 |
+
current = null
|
| 204 |
+
Log.i(TAG, "RELEASE model=${profile?.id} closed=$closed")
|
| 205 |
+
if (closed && profile != null) publish(AsrEvent.Released(profile))
|
| 206 |
+
closed
|
| 207 |
+
}
|
| 208 |
+
)
|
| 209 |
+
|
| 210 |
+
private fun log(source: String, outcome: TranscribeOutcome) {
|
| 211 |
+
when (outcome) {
|
| 212 |
+
is TranscribeOutcome.Text -> {
|
| 213 |
+
val transcript = outcome.transcript
|
| 214 |
+
Log.i(
|
| 215 |
+
TAG,
|
| 216 |
+
String.format(
|
| 217 |
+
Locale.US,
|
| 218 |
+
"TRANSCRIBED source=%s model=%s backend=%s audio_s=%.2f sent_s=%.2f send_ms=%.1f rtf=%.3f raw=%s",
|
| 219 |
+
source,
|
| 220 |
+
transcript.profile.id,
|
| 221 |
+
transcript.lmBackend,
|
| 222 |
+
transcript.audioSeconds,
|
| 223 |
+
transcript.sentSeconds,
|
| 224 |
+
transcript.sendMs,
|
| 225 |
+
transcript.rtf,
|
| 226 |
+
transcript.raw.replace('\n', ' '),
|
| 227 |
+
),
|
| 228 |
+
)
|
| 229 |
+
}
|
| 230 |
+
is TranscribeOutcome.NoSpeech ->
|
| 231 |
+
Log.i(
|
| 232 |
+
TAG,
|
| 233 |
+
String.format(
|
| 234 |
+
Locale.US,
|
| 235 |
+
"NO_SPEECH source=%s seconds=%.2f rms_dbfs=%.1f peak_dbfs=%.1f",
|
| 236 |
+
source,
|
| 237 |
+
outcome.seconds,
|
| 238 |
+
outcome.levels.rmsDbfs,
|
| 239 |
+
outcome.levels.peakDbfs,
|
| 240 |
+
),
|
| 241 |
+
)
|
| 242 |
+
}
|
| 243 |
+
}
|
| 244 |
+
|
| 245 |
+
private fun publish(event: AsrEvent) {
|
| 246 |
+
main.post { listeners.forEach { it(event) } }
|
| 247 |
+
}
|
| 248 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/ClipPlayer.kt
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import android.media.AudioAttributes
|
| 4 |
+
import android.media.AudioFormat
|
| 5 |
+
import android.media.AudioTrack
|
| 6 |
+
import android.os.Handler
|
| 7 |
+
import android.os.Looper
|
| 8 |
+
import java.io.File
|
| 9 |
+
|
| 10 |
+
/**
|
| 11 |
+
* Plays one 16-bit wav through the phone's speaker. [onDone] runs on the main thread once the last sample has reached
|
| 12 |
+
* the mixer plus the output latency (about 0.2 s on the Galaxy S26), or with a sentence when playback failed.
|
| 13 |
+
*/
|
| 14 |
+
class ClipPlayer {
|
| 15 |
+
private val main = Handler(Looper.getMainLooper())
|
| 16 |
+
private var track: AudioTrack? = null
|
| 17 |
+
private var finish: Runnable? = null
|
| 18 |
+
|
| 19 |
+
/** Starts [wav]; [onProgress] gets the played share (0..1) every 50 ms. Stops a clip that is still playing. */
|
| 20 |
+
fun play(wav: File, onProgress: (Float) -> Unit, onDone: (failure: String?) -> Unit) {
|
| 21 |
+
stop()
|
| 22 |
+
val newTrack: AudioTrack
|
| 23 |
+
val frames: Int
|
| 24 |
+
val rate: Int
|
| 25 |
+
try {
|
| 26 |
+
val pcm = Wav.pcm16(wav)
|
| 27 |
+
frames = pcm.frames
|
| 28 |
+
rate = pcm.sampleRate
|
| 29 |
+
newTrack =
|
| 30 |
+
AudioTrack.Builder()
|
| 31 |
+
.setAudioAttributes(
|
| 32 |
+
AudioAttributes.Builder()
|
| 33 |
+
.setUsage(AudioAttributes.USAGE_MEDIA)
|
| 34 |
+
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
|
| 35 |
+
.build()
|
| 36 |
+
)
|
| 37 |
+
.setAudioFormat(
|
| 38 |
+
AudioFormat.Builder()
|
| 39 |
+
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
| 40 |
+
.setSampleRate(pcm.sampleRate)
|
| 41 |
+
.setChannelMask(
|
| 42 |
+
if (pcm.channels == 1) AudioFormat.CHANNEL_OUT_MONO else AudioFormat.CHANNEL_OUT_STEREO
|
| 43 |
+
)
|
| 44 |
+
.build()
|
| 45 |
+
)
|
| 46 |
+
.setTransferMode(AudioTrack.MODE_STATIC)
|
| 47 |
+
.setBufferSizeInBytes(pcm.data.size)
|
| 48 |
+
.build()
|
| 49 |
+
check(newTrack.write(pcm.data, 0, pcm.data.size) == pcm.data.size) { "AudioTrack took a partial buffer" }
|
| 50 |
+
} catch (e: RuntimeException) {
|
| 51 |
+
onDone("Could not play ${wav.name}: ${e.message ?: e}")
|
| 52 |
+
return
|
| 53 |
+
}
|
| 54 |
+
val done = Runnable {
|
| 55 |
+
if (track === newTrack) {
|
| 56 |
+
stop()
|
| 57 |
+
onDone(null)
|
| 58 |
+
}
|
| 59 |
+
}
|
| 60 |
+
finish = done
|
| 61 |
+
track = newTrack
|
| 62 |
+
newTrack.positionNotificationPeriod = rate / PROGRESS_PER_SECOND
|
| 63 |
+
newTrack.notificationMarkerPosition = maxOf(1, frames - rate / MARKER_LEAD_PER_SECOND)
|
| 64 |
+
newTrack.setPlaybackPositionUpdateListener(
|
| 65 |
+
object : AudioTrack.OnPlaybackPositionUpdateListener {
|
| 66 |
+
override fun onMarkerReached(t: AudioTrack) {
|
| 67 |
+
main.removeCallbacks(done)
|
| 68 |
+
main.postDelayed(done, OUTPUT_LATENCY_MS)
|
| 69 |
+
}
|
| 70 |
+
|
| 71 |
+
override fun onPeriodicNotification(t: AudioTrack) {
|
| 72 |
+
if (track === t) onProgress((t.playbackHeadPosition.toFloat() / frames).coerceIn(0f, 1f))
|
| 73 |
+
}
|
| 74 |
+
},
|
| 75 |
+
main,
|
| 76 |
+
)
|
| 77 |
+
newTrack.play()
|
| 78 |
+
// A marker that never fires (a muted or rerouted output) still ends the clip.
|
| 79 |
+
main.postDelayed(done, frames * 1000L / rate + FALLBACK_SLACK_MS)
|
| 80 |
+
}
|
| 81 |
+
|
| 82 |
+
/** Stops and frees the current clip without calling its onDone. */
|
| 83 |
+
fun stop() {
|
| 84 |
+
finish?.let { main.removeCallbacks(it) }
|
| 85 |
+
finish = null
|
| 86 |
+
track?.let {
|
| 87 |
+
runCatching { it.stop() }
|
| 88 |
+
it.release()
|
| 89 |
+
}
|
| 90 |
+
track = null
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
private companion object {
|
| 94 |
+
const val PROGRESS_PER_SECOND = 20
|
| 95 |
+
const val MARKER_LEAD_PER_SECOND = 50 // the marker sits 20 ms before the last frame
|
| 96 |
+
const val OUTPUT_LATENCY_MS = 250L
|
| 97 |
+
const val FALLBACK_SLACK_MS = 3_000L
|
| 98 |
+
}
|
| 99 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/MainActivity.kt
ADDED
|
@@ -0,0 +1,712 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import android.Manifest
|
| 4 |
+
import android.annotation.SuppressLint
|
| 5 |
+
import android.app.Activity
|
| 6 |
+
import android.content.Intent
|
| 7 |
+
import android.content.pm.PackageManager
|
| 8 |
+
import android.graphics.Typeface
|
| 9 |
+
import android.graphics.drawable.GradientDrawable
|
| 10 |
+
import android.os.Build
|
| 11 |
+
import android.os.Bundle
|
| 12 |
+
import android.os.Handler
|
| 13 |
+
import android.os.Looper
|
| 14 |
+
import android.text.SpannableStringBuilder
|
| 15 |
+
import android.text.Spanned
|
| 16 |
+
import android.text.style.ForegroundColorSpan
|
| 17 |
+
import android.text.style.StyleSpan
|
| 18 |
+
import android.util.Log
|
| 19 |
+
import android.util.TypedValue
|
| 20 |
+
import android.view.Gravity
|
| 21 |
+
import android.view.MotionEvent
|
| 22 |
+
import android.view.View
|
| 23 |
+
import android.view.ViewGroup
|
| 24 |
+
import android.view.WindowInsets
|
| 25 |
+
import android.view.WindowManager
|
| 26 |
+
import android.widget.LinearLayout
|
| 27 |
+
import android.widget.ScrollView
|
| 28 |
+
import android.widget.TextView
|
| 29 |
+
import java.io.File
|
| 30 |
+
import java.util.Locale
|
| 31 |
+
import java.util.concurrent.CompletableFuture
|
| 32 |
+
import java.util.concurrent.Future
|
| 33 |
+
|
| 34 |
+
/** How a microphone test ended (see [MainActivity.runMicTest]). */
|
| 35 |
+
sealed interface MicTestResult {
|
| 36 |
+
data class Finished(val outcome: TranscribeOutcome) : MicTestResult
|
| 37 |
+
|
| 38 |
+
data class Failed(val message: String) : MicTestResult
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
/**
|
| 42 |
+
* Speech to text on the phone with one of three LiteRT-LM bundles: pick a model, hold the microphone button and speak
|
| 43 |
+
* (up to 30 s), or play one of three bundled clips. The bundle's language model and audio encoder run on the CPU
|
| 44 |
+
* (4 threads); every number on screen is measured by this app: "engine load" = Engine.initialize() plus the first
|
| 45 |
+
* conversation, "transcribe" = the wall time of sendMessage(), RTF = that time over the clip length.
|
| 46 |
+
*
|
| 47 |
+
* The bundles are read from /sdcard/Android/data/com.asrlitertlm/files/<file name> (adb push, see README). Launch
|
| 48 |
+
* extras, for recordings and the device check (all optional):
|
| 49 |
+
* ```
|
| 50 |
+
* --es model <id> qwen3-asr-1.7b | fun-asr-nano-2512 | confucius4-r2t2 (default: the last one used)
|
| 51 |
+
* --es backend gpu language model on the GPU (audio encoder stays on the CPU)
|
| 52 |
+
* --ez autoplay true after READY: wait delay_ms, then clips zh -> en -> ja (each played, then transcribed)
|
| 53 |
+
* --ei delay_ms 3000 with gap_ms between one transcript and the next clip
|
| 54 |
+
* --ei gap_ms 2500
|
| 55 |
+
* --ei mic_test_ms 9000 after READY and delay_ms: record that long without touching the screen, then transcribe
|
| 56 |
+
* --ez mic_test_play true and play clip en through the speaker meanwhile (needs RECORD_AUDIO granted already)
|
| 57 |
+
* ```
|
| 58 |
+
* Logs go to the tag AsrLitertlm: one `STATE <name>` line per screen state, `LOAD` and `TRANSCRIBED` lines with the
|
| 59 |
+
* measured times.
|
| 60 |
+
*/
|
| 61 |
+
class MainActivity : Activity() {
|
| 62 |
+
private val main = Handler(Looper.getMainLooper())
|
| 63 |
+
private val player = ClipPlayer()
|
| 64 |
+
private lateinit var recorder: MicRecorder
|
| 65 |
+
private val onEvent: (AsrEvent) -> Unit = { handle(it) }
|
| 66 |
+
|
| 67 |
+
private var busy = false
|
| 68 |
+
private var lmBackend = LmBackend.CPU
|
| 69 |
+
private var selected: ModelProfile? = null
|
| 70 |
+
private var micCallback: ((MicTestResult) -> Unit)? = null
|
| 71 |
+
private var extrasPending = false
|
| 72 |
+
private var autoplayIndex = -1
|
| 73 |
+
private var pendingNotice: String? = null
|
| 74 |
+
|
| 75 |
+
private lateinit var pill: TextView
|
| 76 |
+
private lateinit var engineLine: TextView
|
| 77 |
+
private lateinit var pickerRows: Map<ModelProfile, TextView>
|
| 78 |
+
private lateinit var sourceLine: TextView
|
| 79 |
+
private lateinit var bar: PlayBar
|
| 80 |
+
private lateinit var body: TextView
|
| 81 |
+
private lateinit var languageLine: TextView
|
| 82 |
+
private lateinit var timeLine: TextView
|
| 83 |
+
private lateinit var deviceLine: TextView
|
| 84 |
+
private lateinit var clipButtons: List<TextView>
|
| 85 |
+
private lateinit var micButton: TextView
|
| 86 |
+
|
| 87 |
+
private val deviceName: String
|
| 88 |
+
get() = if (Build.MODEL.startsWith("SM-S942")) "Galaxy S26" else Build.MODEL
|
| 89 |
+
|
| 90 |
+
override fun onCreate(savedInstanceState: Bundle?) {
|
| 91 |
+
super.onCreate(savedInstanceState)
|
| 92 |
+
// Show over the lock screen and keep the display on: a hidden activity's process moves to the background CPU
|
| 93 |
+
// set (little cores), where the same model runs about 10x slower.
|
| 94 |
+
setShowWhenLocked(true)
|
| 95 |
+
setTurnScreenOn(true)
|
| 96 |
+
window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
|
| 97 |
+
AsrSession.init(this)
|
| 98 |
+
recorder = MicRecorder(this)
|
| 99 |
+
setContentView(buildUi())
|
| 100 |
+
AsrSession.addListener(onEvent)
|
| 101 |
+
startFrom(intent)
|
| 102 |
+
}
|
| 103 |
+
|
| 104 |
+
override fun onNewIntent(intent: Intent) {
|
| 105 |
+
super.onNewIntent(intent)
|
| 106 |
+
setIntent(intent)
|
| 107 |
+
if (!busy) startFrom(intent)
|
| 108 |
+
}
|
| 109 |
+
|
| 110 |
+
override fun onResume() {
|
| 111 |
+
super.onResume()
|
| 112 |
+
refreshPicker()
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
override fun onDestroy() {
|
| 116 |
+
AsrSession.removeListener(onEvent)
|
| 117 |
+
recorder.stop()
|
| 118 |
+
player.stop()
|
| 119 |
+
main.removeCallbacksAndMessages(null)
|
| 120 |
+
// The engine holds gigabytes: free it when the screen goes away for good.
|
| 121 |
+
if (isFinishing) AsrSession.release()
|
| 122 |
+
super.onDestroy()
|
| 123 |
+
}
|
| 124 |
+
|
| 125 |
+
// ---- model --------------------------------------------------------------------------------------------------
|
| 126 |
+
|
| 127 |
+
/** Picks the model from the launch extras, the last one used, or the first on the phone, and loads it. */
|
| 128 |
+
private fun startFrom(launch: Intent) {
|
| 129 |
+
lmBackend = if (launch.getStringExtra(EXTRA_BACKEND) == "gpu") LmBackend.GPU else LmBackend.CPU
|
| 130 |
+
deviceLine.text = statusLine()
|
| 131 |
+
extrasPending = launch.getBooleanExtra(EXTRA_AUTOPLAY, false) || launch.getIntExtra(EXTRA_MIC_TEST_MS, 0) > 0
|
| 132 |
+
val requested = ModelProfile.byId(launch.getStringExtra(EXTRA_MODEL))
|
| 133 |
+
val saved = ModelProfile.byId(getPreferences(MODE_PRIVATE).getString(PREF_MODEL, null))
|
| 134 |
+
val choice =
|
| 135 |
+
listOfNotNull(requested, saved).firstOrNull { AsrSession.isPresent(it) }
|
| 136 |
+
?: ModelProfile.ALL.firstOrNull { AsrSession.isPresent(it) }
|
| 137 |
+
when {
|
| 138 |
+
choice == null -> showNoModel()
|
| 139 |
+
requested != null && choice != requested -> {
|
| 140 |
+
// Shown once the model that is on the phone is ready (the load clears the screen).
|
| 141 |
+
pendingNotice = AsrEngine.missingBundleMessage(requested, AsrSession.modelDir)
|
| 142 |
+
selectModel(choice)
|
| 143 |
+
}
|
| 144 |
+
else -> selectModel(choice)
|
| 145 |
+
}
|
| 146 |
+
}
|
| 147 |
+
|
| 148 |
+
/**
|
| 149 |
+
* Loads [profile] (the picker's tap). Returns the load, or null when the bundle is not on the phone (the screen
|
| 150 |
+
* then says which file to push where). A model that is already loaded is not loaded again.
|
| 151 |
+
*/
|
| 152 |
+
fun selectModel(profile: ModelProfile, backend: LmBackend = lmBackend): Future<LoadInfo>? {
|
| 153 |
+
if (!AsrSession.isPresent(profile)) {
|
| 154 |
+
refreshPicker()
|
| 155 |
+
showText(AsrEngine.missingBundleMessage(profile, AsrSession.modelDir))
|
| 156 |
+
return null
|
| 157 |
+
}
|
| 158 |
+
selected = profile
|
| 159 |
+
lmBackend = backend
|
| 160 |
+
getPreferences(MODE_PRIVATE).edit().putString(PREF_MODEL, profile.id).apply()
|
| 161 |
+
refreshPicker()
|
| 162 |
+
val current = AsrSession.current
|
| 163 |
+
if (current != null && current.profile == profile && current.lmBackend == backend) {
|
| 164 |
+
showReady(current)
|
| 165 |
+
return CompletableFuture.completedFuture(current)
|
| 166 |
+
}
|
| 167 |
+
setBusy(true)
|
| 168 |
+
return AsrSession.load(profile, backend)
|
| 169 |
+
}
|
| 170 |
+
|
| 171 |
+
private fun handle(event: AsrEvent) {
|
| 172 |
+
when (event) {
|
| 173 |
+
is AsrEvent.Loading -> {
|
| 174 |
+
setPill("LOADING", C_IDLE)
|
| 175 |
+
engineLine.text =
|
| 176 |
+
String.format(
|
| 177 |
+
Locale.US,
|
| 178 |
+
"loading %s (%.2f GB) · LM %s",
|
| 179 |
+
event.profile.displayName,
|
| 180 |
+
AsrSession.bundleFile(event.profile).length() / 1e9,
|
| 181 |
+
backendLabel(event.lmBackend),
|
| 182 |
+
)
|
| 183 |
+
clearResult()
|
| 184 |
+
}
|
| 185 |
+
is AsrEvent.Loaded -> {
|
| 186 |
+
setBusy(false)
|
| 187 |
+
showReady(event.info)
|
| 188 |
+
}
|
| 189 |
+
is AsrEvent.LoadFailed -> {
|
| 190 |
+
setBusy(false)
|
| 191 |
+
setPill("ERROR", C_LIVE)
|
| 192 |
+
// A missing file leaves the model loaded before in place.
|
| 193 |
+
engineLine.text = AsrSession.current?.let { "${it.profile.displayName} still loaded" } ?: "no model loaded"
|
| 194 |
+
showText(event.message)
|
| 195 |
+
}
|
| 196 |
+
is AsrEvent.Transcribing -> setPill("TRANSCRIBING", C_WORK)
|
| 197 |
+
is AsrEvent.Transcribed -> {
|
| 198 |
+
show(event.outcome)
|
| 199 |
+
setBusy(false)
|
| 200 |
+
if (event.source == SOURCE_MIC) finishMicTest(MicTestResult.Finished(event.outcome))
|
| 201 |
+
continueAutoplay(event.source)
|
| 202 |
+
}
|
| 203 |
+
is AsrEvent.TranscribeFailed -> {
|
| 204 |
+
setBusy(false)
|
| 205 |
+
setPill("ERROR", C_LIVE)
|
| 206 |
+
showText(event.message)
|
| 207 |
+
if (event.source == SOURCE_MIC) finishMicTest(MicTestResult.Failed(event.message))
|
| 208 |
+
autoplayIndex = -1
|
| 209 |
+
}
|
| 210 |
+
is AsrEvent.Released -> {
|
| 211 |
+
setPill("RELEASED", C_IDLE)
|
| 212 |
+
engineLine.text = "${event.profile.displayName} closed"
|
| 213 |
+
setBusy(busy)
|
| 214 |
+
}
|
| 215 |
+
}
|
| 216 |
+
}
|
| 217 |
+
|
| 218 |
+
private fun showReady(info: LoadInfo) {
|
| 219 |
+
selected = info.profile
|
| 220 |
+
setPill("READY", C_IDLE)
|
| 221 |
+
engineLine.text =
|
| 222 |
+
String.format(
|
| 223 |
+
Locale.US,
|
| 224 |
+
"engine load %.2f s · LM %s",
|
| 225 |
+
info.loadMs / 1000,
|
| 226 |
+
backendLabel(info.lmBackend),
|
| 227 |
+
)
|
| 228 |
+
deviceLine.text = statusLine()
|
| 229 |
+
refreshPicker()
|
| 230 |
+
pendingNotice?.let {
|
| 231 |
+
pendingNotice = null
|
| 232 |
+
showText(it)
|
| 233 |
+
}
|
| 234 |
+
if (extrasPending) {
|
| 235 |
+
extrasPending = false
|
| 236 |
+
runLaunchExtras()
|
| 237 |
+
}
|
| 238 |
+
}
|
| 239 |
+
|
| 240 |
+
private fun showNoModel() {
|
| 241 |
+
setPill("NO MODEL", C_LIVE)
|
| 242 |
+
engineLine.text = "no model on this phone"
|
| 243 |
+
val files = ModelProfile.ALL.joinToString("\n") { "· ${it.fileName} (${it.hubRepo})" }
|
| 244 |
+
showText("Push at least one bundle to ${AsrSession.modelDir.path}/ with adb, then reopen the app:\n$files")
|
| 245 |
+
refreshPicker()
|
| 246 |
+
}
|
| 247 |
+
|
| 248 |
+
// ---- clips --------------------------------------------------------------------------------------------------
|
| 249 |
+
|
| 250 |
+
/** A bundled clip: play it through the speaker, then transcribe it. */
|
| 251 |
+
private fun playClip(index: Int) {
|
| 252 |
+
if (busy || AsrSession.current == null) return
|
| 253 |
+
val clip = Clip.ALL[index]
|
| 254 |
+
val file = AsrSession.clipFile(clip)
|
| 255 |
+
setBusy(true)
|
| 256 |
+
clearResult()
|
| 257 |
+
setPill("● PLAYING", C_LIVE)
|
| 258 |
+
sourceLine.text = String.format(Locale.US, "clip %s · %.1f s", clip.label, Wav.seconds(file))
|
| 259 |
+
bar.set(0f, C_LIVE)
|
| 260 |
+
bar.visibility = View.VISIBLE
|
| 261 |
+
player.play(
|
| 262 |
+
file,
|
| 263 |
+
onProgress = { bar.fraction = it },
|
| 264 |
+
onDone = { failure ->
|
| 265 |
+
bar.set(1f, C_IDLE)
|
| 266 |
+
if (failure != null) {
|
| 267 |
+
setBusy(false)
|
| 268 |
+
setPill("ERROR", C_LIVE)
|
| 269 |
+
showText(failure)
|
| 270 |
+
autoplayIndex = -1
|
| 271 |
+
} else {
|
| 272 |
+
AsrSession.transcribe(file, SOURCE_CLIP + clip.id)
|
| 273 |
+
}
|
| 274 |
+
},
|
| 275 |
+
)
|
| 276 |
+
}
|
| 277 |
+
|
| 278 |
+
private fun runLaunchExtras() {
|
| 279 |
+
val delay = intent.getIntExtra(EXTRA_DELAY_MS, 3000).toLong()
|
| 280 |
+
val micTestMs = intent.getIntExtra(EXTRA_MIC_TEST_MS, 0)
|
| 281 |
+
if (intent.getBooleanExtra(EXTRA_AUTOPLAY, false)) {
|
| 282 |
+
Log.i(TAG, "AUTOPLAY_START delay_ms=$delay")
|
| 283 |
+
autoplayIndex = 0
|
| 284 |
+
main.postDelayed({ playClip(0) }, delay)
|
| 285 |
+
} else if (micTestMs > 0) {
|
| 286 |
+
val play = intent.getBooleanExtra(EXTRA_MIC_TEST_PLAY, false)
|
| 287 |
+
main.postDelayed({ runMicTest(micTestMs.toLong(), play) }, delay)
|
| 288 |
+
}
|
| 289 |
+
}
|
| 290 |
+
|
| 291 |
+
private fun continueAutoplay(source: String) {
|
| 292 |
+
if (autoplayIndex < 0 || !source.startsWith(SOURCE_CLIP)) return
|
| 293 |
+
autoplayIndex++
|
| 294 |
+
if (autoplayIndex >= Clip.ALL.size) {
|
| 295 |
+
autoplayIndex = -1
|
| 296 |
+
Log.i(TAG, "AUTOPLAY_DONE")
|
| 297 |
+
return
|
| 298 |
+
}
|
| 299 |
+
val next = autoplayIndex
|
| 300 |
+
main.postDelayed({ playClip(next) }, intent.getIntExtra(EXTRA_GAP_MS, 2500).toLong())
|
| 301 |
+
}
|
| 302 |
+
|
| 303 |
+
// ---- microphone ---------------------------------------------------------------------------------------------
|
| 304 |
+
|
| 305 |
+
/** Hold to talk: recording starts on press and ends on release (or at 30 s). */
|
| 306 |
+
@SuppressLint("ClickableViewAccessibility") // the button also takes a click: performClick() on release
|
| 307 |
+
private fun onMicTouch(view: View, event: MotionEvent): Boolean {
|
| 308 |
+
when (event.actionMasked) {
|
| 309 |
+
MotionEvent.ACTION_DOWN -> {
|
| 310 |
+
if (busy || AsrSession.current == null) return true
|
| 311 |
+
if (!recorder.hasPermission()) {
|
| 312 |
+
requestPermissions(arrayOf(Manifest.permission.RECORD_AUDIO), REQUEST_MIC)
|
| 313 |
+
return true
|
| 314 |
+
}
|
| 315 |
+
startRecording(playClip = false)
|
| 316 |
+
}
|
| 317 |
+
MotionEvent.ACTION_UP -> {
|
| 318 |
+
view.performClick()
|
| 319 |
+
recorder.stop()
|
| 320 |
+
}
|
| 321 |
+
MotionEvent.ACTION_CANCEL -> recorder.stop()
|
| 322 |
+
}
|
| 323 |
+
return true
|
| 324 |
+
}
|
| 325 |
+
|
| 326 |
+
override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<out String>, grantResults: IntArray) {
|
| 327 |
+
super.onRequestPermissionsResult(requestCode, permissions, grantResults)
|
| 328 |
+
if (requestCode != REQUEST_MIC) return
|
| 329 |
+
if (grantResults.firstOrNull() == PackageManager.PERMISSION_GRANTED) {
|
| 330 |
+
showText("Microphone allowed. Hold the button and speak.")
|
| 331 |
+
} else {
|
| 332 |
+
setPill("NO MIC", C_LIVE)
|
| 333 |
+
showText(MicRecorder.PERMISSION_MESSAGE)
|
| 334 |
+
}
|
| 335 |
+
}
|
| 336 |
+
|
| 337 |
+
/**
|
| 338 |
+
* The microphone path without touching the screen: record for [durationMs] (playing clip en through the speaker
|
| 339 |
+
* when [playClip]), then transcribe like the button does. [onFinished] gets the outcome or the failure sentence;
|
| 340 |
+
* without the permission the sentence comes back at once and nothing is recorded.
|
| 341 |
+
*/
|
| 342 |
+
fun runMicTest(durationMs: Long, playClip: Boolean, onFinished: ((MicTestResult) -> Unit)? = null) {
|
| 343 |
+
Log.i(TAG, "MIC_TEST ms=$durationMs play=$playClip")
|
| 344 |
+
micCallback = onFinished
|
| 345 |
+
if (!recorder.hasPermission()) {
|
| 346 |
+
setPill("NO MIC", C_LIVE)
|
| 347 |
+
showText(MicRecorder.PERMISSION_MESSAGE)
|
| 348 |
+
finishMicTest(MicTestResult.Failed(MicRecorder.PERMISSION_MESSAGE))
|
| 349 |
+
return
|
| 350 |
+
}
|
| 351 |
+
if (busy || AsrSession.current == null) {
|
| 352 |
+
finishMicTest(MicTestResult.Failed("The app is busy or no model is loaded."))
|
| 353 |
+
return
|
| 354 |
+
}
|
| 355 |
+
if (!startRecording(playClip)) return
|
| 356 |
+
main.postDelayed({ recorder.stop() }, durationMs)
|
| 357 |
+
}
|
| 358 |
+
|
| 359 |
+
private fun startRecording(playClip: Boolean): Boolean {
|
| 360 |
+
val wav = File(cacheDir, "mic.wav")
|
| 361 |
+
val failure =
|
| 362 |
+
recorder.start(
|
| 363 |
+
wav,
|
| 364 |
+
onProgress = { seconds -> main.post { showRecording(seconds) } },
|
| 365 |
+
onDone = { result -> main.post { onRecorded(result) } },
|
| 366 |
+
)
|
| 367 |
+
if (failure != null) {
|
| 368 |
+
setPill("ERROR", C_LIVE)
|
| 369 |
+
showText(failure)
|
| 370 |
+
finishMicTest(MicTestResult.Failed(failure))
|
| 371 |
+
return false
|
| 372 |
+
}
|
| 373 |
+
setBusy(true)
|
| 374 |
+
micButton.isEnabled = true
|
| 375 |
+
micButton.alpha = 1f
|
| 376 |
+
clearResult()
|
| 377 |
+
setPill("● LISTENING", C_LIVE)
|
| 378 |
+
showRecording(0.0)
|
| 379 |
+
bar.set(0f, C_LIVE)
|
| 380 |
+
bar.visibility = View.VISIBLE
|
| 381 |
+
if (playClip) {
|
| 382 |
+
val clip = AsrSession.clipFile(Clip.ALL[1])
|
| 383 |
+
main.postDelayed({ player.play(clip, onProgress = {}, onDone = {}) }, MIC_TEST_PLAY_DELAY_MS)
|
| 384 |
+
}
|
| 385 |
+
return true
|
| 386 |
+
}
|
| 387 |
+
|
| 388 |
+
private fun showRecording(seconds: Double) {
|
| 389 |
+
sourceLine.text = String.format(Locale.US, "mic · %.1f s of %d s", seconds, MicRecorder.MAX_SECONDS)
|
| 390 |
+
bar.fraction = (seconds / MicRecorder.MAX_SECONDS).toFloat()
|
| 391 |
+
micButton.text = String.format(Locale.US, "Release to transcribe · %.1f s", seconds)
|
| 392 |
+
}
|
| 393 |
+
|
| 394 |
+
private fun onRecorded(result: MicResult) {
|
| 395 |
+
micButton.text = MIC_LABEL
|
| 396 |
+
bar.set(1f, C_IDLE)
|
| 397 |
+
when (result) {
|
| 398 |
+
is MicResult.Failed -> {
|
| 399 |
+
setBusy(false)
|
| 400 |
+
setPill("ERROR", C_LIVE)
|
| 401 |
+
showText(result.message)
|
| 402 |
+
finishMicTest(MicTestResult.Failed(result.message))
|
| 403 |
+
}
|
| 404 |
+
is MicResult.Recorded -> {
|
| 405 |
+
Log.i(
|
| 406 |
+
TAG,
|
| 407 |
+
String.format(
|
| 408 |
+
Locale.US,
|
| 409 |
+
"MIC_RECORDED seconds=%.2f peak_dbfs=%.1f rms_dbfs=%.1f at_limit=%b",
|
| 410 |
+
result.seconds,
|
| 411 |
+
result.levels.peakDbfs,
|
| 412 |
+
result.levels.rmsDbfs,
|
| 413 |
+
result.stoppedAtLimit,
|
| 414 |
+
),
|
| 415 |
+
)
|
| 416 |
+
sourceLine.text =
|
| 417 |
+
String.format(Locale.US, "mic · %.1f s", result.seconds) +
|
| 418 |
+
if (result.stoppedAtLimit) " · stopped at the ${MicRecorder.MAX_SECONDS} s limit" else ""
|
| 419 |
+
setBusy(true)
|
| 420 |
+
AsrSession.transcribe(result.wav, SOURCE_MIC)
|
| 421 |
+
}
|
| 422 |
+
}
|
| 423 |
+
}
|
| 424 |
+
|
| 425 |
+
private fun finishMicTest(result: MicTestResult) {
|
| 426 |
+
val callback = micCallback ?: return
|
| 427 |
+
micCallback = null
|
| 428 |
+
callback(result)
|
| 429 |
+
}
|
| 430 |
+
|
| 431 |
+
// ---- screen -------------------------------------------------------------------------------------------------
|
| 432 |
+
|
| 433 |
+
private fun show(outcome: TranscribeOutcome) {
|
| 434 |
+
when (outcome) {
|
| 435 |
+
is TranscribeOutcome.NoSpeech -> {
|
| 436 |
+
setPill("NO SPEECH", C_IDLE)
|
| 437 |
+
body.text = "No speech heard."
|
| 438 |
+
languageLine.text =
|
| 439 |
+
String.format(
|
| 440 |
+
Locale.US,
|
| 441 |
+
"The recording peaked at %.0f dBFS (RMS %.0f dBFS); the model was not called.",
|
| 442 |
+
outcome.levels.peakDbfs,
|
| 443 |
+
outcome.levels.rmsDbfs,
|
| 444 |
+
)
|
| 445 |
+
timeLine.text = ""
|
| 446 |
+
}
|
| 447 |
+
is TranscribeOutcome.Text -> {
|
| 448 |
+
val transcript = outcome.transcript
|
| 449 |
+
val answer = transcript.answer
|
| 450 |
+
setPill("DONE", C_DONE)
|
| 451 |
+
body.textLocale = localeFor(answer)
|
| 452 |
+
body.text = answer.text.ifEmpty { "(no speech in the answer)" }
|
| 453 |
+
languageLine.text =
|
| 454 |
+
when {
|
| 455 |
+
transcript.profile.answerFormat == AnswerFormat.PLAIN -> ""
|
| 456 |
+
answer.language.isEmpty() -> "detected language: none"
|
| 457 |
+
else -> "detected language: ${answer.language}"
|
| 458 |
+
}
|
| 459 |
+
val numbers =
|
| 460 |
+
String.format(
|
| 461 |
+
Locale.US,
|
| 462 |
+
"transcribe %.2f s / clip %.2f s = RTF %.2f",
|
| 463 |
+
transcript.sendMs / 1000,
|
| 464 |
+
transcript.sentSeconds,
|
| 465 |
+
transcript.rtf,
|
| 466 |
+
)
|
| 467 |
+
timeLine.text =
|
| 468 |
+
SpannableStringBuilder(numbers).apply {
|
| 469 |
+
setSpan(ForegroundColorSpan(C_TEXT), 0, numbers.length, Spanned.SPAN_EXCLUSIVE_EXCLUSIVE)
|
| 470 |
+
setSpan(StyleSpan(Typeface.BOLD), 0, numbers.length, Spanned.SPAN_EXCLUSIVE_EXCLUSIVE)
|
| 471 |
+
}
|
| 472 |
+
}
|
| 473 |
+
}
|
| 474 |
+
}
|
| 475 |
+
|
| 476 |
+
private fun showText(text: String) {
|
| 477 |
+
body.textLocale = Locale.US
|
| 478 |
+
body.text = text
|
| 479 |
+
languageLine.text = ""
|
| 480 |
+
timeLine.text = ""
|
| 481 |
+
}
|
| 482 |
+
|
| 483 |
+
private fun clearResult() {
|
| 484 |
+
sourceLine.text = ""
|
| 485 |
+
body.text = ""
|
| 486 |
+
languageLine.text = ""
|
| 487 |
+
timeLine.text = ""
|
| 488 |
+
bar.visibility = View.INVISIBLE
|
| 489 |
+
}
|
| 490 |
+
|
| 491 |
+
/** The glyph forms of Han characters follow the language the model named, or the script of the text. */
|
| 492 |
+
private fun localeFor(answer: Answer): Locale =
|
| 493 |
+
when {
|
| 494 |
+
answer.language == "Japanese" || answer.text.any { it in ''..'ヿ' } -> Locale.JAPANESE
|
| 495 |
+
answer.language == "Cantonese" -> Locale.TRADITIONAL_CHINESE
|
| 496 |
+
answer.language == "Chinese" ||
|
| 497 |
+
answer.text.any { Character.UnicodeScript.of(it.code) == Character.UnicodeScript.HAN } ->
|
| 498 |
+
Locale.SIMPLIFIED_CHINESE
|
| 499 |
+
else -> Locale.US
|
| 500 |
+
}
|
| 501 |
+
|
| 502 |
+
private fun refreshPicker() {
|
| 503 |
+
if (!::pickerRows.isInitialized) return
|
| 504 |
+
val loaded = AsrSession.current?.profile
|
| 505 |
+
for ((profile, row) in pickerRows) {
|
| 506 |
+
val present = AsrSession.isPresent(profile)
|
| 507 |
+
val bytes = if (present) AsrSession.bundleFile(profile).length() else profile.hubBytes
|
| 508 |
+
val mark = if (profile == selected) "● " else "○ "
|
| 509 |
+
val state =
|
| 510 |
+
when {
|
| 511 |
+
!present -> "missing"
|
| 512 |
+
profile == loaded -> "loaded"
|
| 513 |
+
else -> "on phone"
|
| 514 |
+
}
|
| 515 |
+
val head = mark + profile.displayName
|
| 516 |
+
val text = String.format(Locale.US, "%s %.2f GB · %s", head, bytes / 1e9, state)
|
| 517 |
+
row.text =
|
| 518 |
+
SpannableStringBuilder(text).apply {
|
| 519 |
+
setSpan(StyleSpan(Typeface.BOLD), 0, head.length, Spanned.SPAN_EXCLUSIVE_EXCLUSIVE)
|
| 520 |
+
}
|
| 521 |
+
row.setTextColor(if (present) C_TEXT else C_SUB)
|
| 522 |
+
row.background = rounded(if (profile == selected) C_SELECTED else C_BUTTON, C_BUTTON_EDGE)
|
| 523 |
+
row.isEnabled = !busy
|
| 524 |
+
}
|
| 525 |
+
}
|
| 526 |
+
|
| 527 |
+
private fun setBusy(on: Boolean) {
|
| 528 |
+
busy = on
|
| 529 |
+
val ready = !on && AsrSession.current != null
|
| 530 |
+
for (button in clipButtons + micButton) {
|
| 531 |
+
button.isEnabled = ready
|
| 532 |
+
button.alpha = if (ready) 1f else 0.4f
|
| 533 |
+
}
|
| 534 |
+
refreshPicker()
|
| 535 |
+
}
|
| 536 |
+
|
| 537 |
+
private fun setPill(label: String, color: Int) {
|
| 538 |
+
pill.text = label
|
| 539 |
+
pill.background = GradientDrawable().apply {
|
| 540 |
+
cornerRadius = dp(999f)
|
| 541 |
+
setColor(color)
|
| 542 |
+
}
|
| 543 |
+
Log.i(TAG, "STATE ${label.removePrefix("● ").replace(' ', '_')}")
|
| 544 |
+
}
|
| 545 |
+
|
| 546 |
+
private fun statusLine(): String =
|
| 547 |
+
String.format(
|
| 548 |
+
Locale.US,
|
| 549 |
+
"%s · LiteRT-LM %s · LM %s, audio CPU",
|
| 550 |
+
deviceName,
|
| 551 |
+
BuildConfig.LITERTLM_VERSION,
|
| 552 |
+
if (lmBackend == LmBackend.GPU) "GPU" else "CPU",
|
| 553 |
+
)
|
| 554 |
+
|
| 555 |
+
private fun backendLabel(backend: LmBackend): String =
|
| 556 |
+
if (backend == LmBackend.GPU) "GPU" else "CPU ${AsrEngine.CPU_THREADS} threads"
|
| 557 |
+
|
| 558 |
+
private fun buildUi(): View {
|
| 559 |
+
val side = dp(20f).toInt()
|
| 560 |
+
val root =
|
| 561 |
+
LinearLayout(this).apply {
|
| 562 |
+
orientation = LinearLayout.VERTICAL
|
| 563 |
+
setBackgroundColor(C_BG)
|
| 564 |
+
setOnApplyWindowInsetsListener { view, insets ->
|
| 565 |
+
val bars = insets.getInsets(WindowInsets.Type.systemBars() or WindowInsets.Type.displayCutout())
|
| 566 |
+
view.setPadding(side + bars.left, dp(14f).toInt() + bars.top, side + bars.right, dp(14f).toInt() + bars.bottom)
|
| 567 |
+
insets
|
| 568 |
+
}
|
| 569 |
+
}
|
| 570 |
+
val title = label(15f, C_TEXT, bold = true).apply { text = "Speech to text on the phone · LiteRT-LM" }
|
| 571 |
+
val picker = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL }
|
| 572 |
+
pickerRows =
|
| 573 |
+
ModelProfile.ALL.associateWith { profile ->
|
| 574 |
+
label(14f, C_TEXT).apply {
|
| 575 |
+
setPadding(dp(14f).toInt(), dp(10f).toInt(), dp(14f).toInt(), dp(10f).toInt())
|
| 576 |
+
maxLines = 1
|
| 577 |
+
setOnClickListener { if (!busy) selectModel(profile) }
|
| 578 |
+
picker.addView(this, layoutParams(top = 8f))
|
| 579 |
+
}
|
| 580 |
+
}
|
| 581 |
+
pill =
|
| 582 |
+
label(13f, C_TEXT, bold = true).apply {
|
| 583 |
+
letterSpacing = 0.08f
|
| 584 |
+
gravity = Gravity.CENTER_VERTICAL
|
| 585 |
+
setPadding(dp(12f).toInt(), 0, dp(12f).toInt(), 0)
|
| 586 |
+
}
|
| 587 |
+
engineLine = label(14f, C_SUB)
|
| 588 |
+
sourceLine = label(17f, C_SUB)
|
| 589 |
+
bar = PlayBar(this, C_BUTTON_EDGE).apply { visibility = View.INVISIBLE }
|
| 590 |
+
body = label(24f, C_TEXT).apply { setLineSpacing(0f, 1.1f) }
|
| 591 |
+
languageLine = label(17f, C_TEXT)
|
| 592 |
+
timeLine = label(15f, C_SUB)
|
| 593 |
+
deviceLine = label(13f, C_SUB)
|
| 594 |
+
val attribution = label(13f, C_SUB).apply { text = ATTRIBUTION }
|
| 595 |
+
|
| 596 |
+
val result =
|
| 597 |
+
LinearLayout(this).apply {
|
| 598 |
+
orientation = LinearLayout.VERTICAL
|
| 599 |
+
addView(sourceLine, layoutParams(top = 14f))
|
| 600 |
+
addView(
|
| 601 |
+
bar,
|
| 602 |
+
LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, dp(6f).toInt()).apply {
|
| 603 |
+
topMargin = dp(10f).toInt()
|
| 604 |
+
},
|
| 605 |
+
)
|
| 606 |
+
addView(body, layoutParams(top = 14f))
|
| 607 |
+
addView(languageLine, layoutParams(top = 14f))
|
| 608 |
+
}
|
| 609 |
+
val scroll =
|
| 610 |
+
ScrollView(this).apply {
|
| 611 |
+
isVerticalScrollBarEnabled = false
|
| 612 |
+
addView(result)
|
| 613 |
+
}
|
| 614 |
+
val clipRow = LinearLayout(this).apply { orientation = LinearLayout.HORIZONTAL }
|
| 615 |
+
clipButtons =
|
| 616 |
+
Clip.ALL.mapIndexed { index, clip ->
|
| 617 |
+
val seconds = Wav.seconds(AsrSession.clipFile(clip))
|
| 618 |
+
button(String.format(Locale.US, "%s · %.1f s", clip.label, seconds)) { playClip(index) }
|
| 619 |
+
.also {
|
| 620 |
+
clipRow.addView(
|
| 621 |
+
it,
|
| 622 |
+
LinearLayout.LayoutParams(0, dp(52f).toInt(), 1f).apply {
|
| 623 |
+
if (index > 0) marginStart = dp(10f).toInt()
|
| 624 |
+
},
|
| 625 |
+
)
|
| 626 |
+
}
|
| 627 |
+
}
|
| 628 |
+
micButton = button(MIC_LABEL) {}.apply { setOnTouchListener { view, event -> onMicTouch(view, event) } }
|
| 629 |
+
setBusy(false)
|
| 630 |
+
|
| 631 |
+
root.addView(title)
|
| 632 |
+
root.addView(picker, layoutParams(top = 4f))
|
| 633 |
+
root.addView(
|
| 634 |
+
pill,
|
| 635 |
+
LinearLayout.LayoutParams(ViewGroup.LayoutParams.WRAP_CONTENT, dp(28f).toInt()).apply {
|
| 636 |
+
topMargin = dp(14f).toInt()
|
| 637 |
+
},
|
| 638 |
+
)
|
| 639 |
+
root.addView(engineLine, layoutParams(top = 6f))
|
| 640 |
+
root.addView(scroll, LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, 0, 1f))
|
| 641 |
+
root.addView(clipRow, layoutParams(top = 10f))
|
| 642 |
+
root.addView(
|
| 643 |
+
micButton,
|
| 644 |
+
LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, dp(64f).toInt()).apply {
|
| 645 |
+
topMargin = dp(10f).toInt()
|
| 646 |
+
},
|
| 647 |
+
)
|
| 648 |
+
// Footer: the measured seconds, then device, runtime and backend, then the clips' source.
|
| 649 |
+
root.addView(timeLine, layoutParams(top = 12f))
|
| 650 |
+
root.addView(deviceLine, layoutParams(top = 4f))
|
| 651 |
+
root.addView(attribution, layoutParams(top = 2f))
|
| 652 |
+
return root
|
| 653 |
+
}
|
| 654 |
+
|
| 655 |
+
private fun layoutParams(top: Float) =
|
| 656 |
+
LinearLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT).apply {
|
| 657 |
+
topMargin = dp(top).toInt()
|
| 658 |
+
}
|
| 659 |
+
|
| 660 |
+
private fun label(sizeSp: Float, color: Int, bold: Boolean = false) =
|
| 661 |
+
TextView(this).apply {
|
| 662 |
+
setTextSize(TypedValue.COMPLEX_UNIT_SP, sizeSp)
|
| 663 |
+
setTextColor(color)
|
| 664 |
+
if (bold) typeface = Typeface.create(Typeface.DEFAULT, Typeface.BOLD)
|
| 665 |
+
}
|
| 666 |
+
|
| 667 |
+
private fun button(text: String, onClick: () -> Unit) =
|
| 668 |
+
label(16f, C_TEXT, bold = true).apply {
|
| 669 |
+
this.text = text
|
| 670 |
+
gravity = Gravity.CENTER
|
| 671 |
+
background = rounded(C_BUTTON, C_BUTTON_EDGE)
|
| 672 |
+
setOnClickListener { onClick() }
|
| 673 |
+
}
|
| 674 |
+
|
| 675 |
+
private fun rounded(fill: Int, edge: Int) =
|
| 676 |
+
GradientDrawable().apply {
|
| 677 |
+
cornerRadius = dp(14f)
|
| 678 |
+
setColor(fill)
|
| 679 |
+
setStroke(dp(1f).toInt(), edge)
|
| 680 |
+
}
|
| 681 |
+
|
| 682 |
+
private fun dp(value: Float) = TypedValue.applyDimension(TypedValue.COMPLEX_UNIT_DIP, value, resources.displayMetrics)
|
| 683 |
+
|
| 684 |
+
companion object {
|
| 685 |
+
private const val TAG = "AsrLitertlm"
|
| 686 |
+
const val EXTRA_MODEL = "model"
|
| 687 |
+
const val EXTRA_BACKEND = "backend"
|
| 688 |
+
const val EXTRA_AUTOPLAY = "autoplay"
|
| 689 |
+
const val EXTRA_DELAY_MS = "delay_ms"
|
| 690 |
+
const val EXTRA_GAP_MS = "gap_ms"
|
| 691 |
+
const val EXTRA_MIC_TEST_MS = "mic_test_ms"
|
| 692 |
+
const val EXTRA_MIC_TEST_PLAY = "mic_test_play"
|
| 693 |
+
const val SOURCE_MIC = "mic"
|
| 694 |
+
const val SOURCE_CLIP = "clip:"
|
| 695 |
+
private const val PREF_MODEL = "model"
|
| 696 |
+
private const val REQUEST_MIC = 1
|
| 697 |
+
private const val MIC_TEST_PLAY_DELAY_MS = 500L
|
| 698 |
+
private const val MIC_LABEL = "Hold to talk (up to 30 s)"
|
| 699 |
+
// The clips are FLEURS recordings with one linear gain each: CC BY 4.0 asks to say so.
|
| 700 |
+
private const val ATTRIBUTION = "clips: FLEURS #1698, CC BY 4.0, level-normalized"
|
| 701 |
+
private const val C_BG = 0xFF0E1116.toInt()
|
| 702 |
+
private const val C_IDLE = 0xFF5F6368.toInt()
|
| 703 |
+
private const val C_LIVE = 0xFFE53935.toInt()
|
| 704 |
+
private const val C_WORK = 0xFF1565C0.toInt()
|
| 705 |
+
private const val C_DONE = 0xFF2E7D32.toInt()
|
| 706 |
+
private const val C_TEXT = 0xFFE6E8EB.toInt()
|
| 707 |
+
private const val C_SUB = 0xFF8A919C.toInt()
|
| 708 |
+
private const val C_BUTTON = 0xFF1A1F27.toInt()
|
| 709 |
+
private const val C_SELECTED = 0xFF1E3A5F.toInt()
|
| 710 |
+
private const val C_BUTTON_EDGE = 0xFF2C333D.toInt()
|
| 711 |
+
}
|
| 712 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/MicRecorder.kt
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import android.Manifest
|
| 4 |
+
import android.annotation.SuppressLint
|
| 5 |
+
import android.content.Context
|
| 6 |
+
import android.content.pm.PackageManager
|
| 7 |
+
import android.media.AudioFormat
|
| 8 |
+
import android.media.AudioRecord
|
| 9 |
+
import android.media.MediaRecorder
|
| 10 |
+
import java.io.ByteArrayOutputStream
|
| 11 |
+
import java.io.File
|
| 12 |
+
import kotlin.concurrent.thread
|
| 13 |
+
|
| 14 |
+
/** The end of one recording. */
|
| 15 |
+
sealed interface MicResult {
|
| 16 |
+
/** A wav of 16 kHz mono 16-bit audio. [stoppedAtLimit] is true when the 30 s limit ended it. */
|
| 17 |
+
data class Recorded(val wav: File, val seconds: Double, val levels: Wav.Levels, val stoppedAtLimit: Boolean) :
|
| 18 |
+
MicResult
|
| 19 |
+
|
| 20 |
+
/** A sentence for the screen. */
|
| 21 |
+
data class Failed(val message: String) : MicResult
|
| 22 |
+
}
|
| 23 |
+
|
| 24 |
+
/**
|
| 25 |
+
* Records 16 kHz mono 16-bit audio (what the three models take) from the microphone, on its own thread, until
|
| 26 |
+
* [stop] is called or [MAX_SECONDS] have been recorded.
|
| 27 |
+
*/
|
| 28 |
+
class MicRecorder(private val context: Context) {
|
| 29 |
+
@Volatile private var stopRequested = false
|
| 30 |
+
|
| 31 |
+
@Volatile
|
| 32 |
+
var isRecording = false
|
| 33 |
+
private set
|
| 34 |
+
|
| 35 |
+
fun hasPermission(): Boolean =
|
| 36 |
+
context.checkSelfPermission(Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED
|
| 37 |
+
|
| 38 |
+
/**
|
| 39 |
+
* Starts a recording into [output]. Returns a failure sentence at once (and records nothing) when the microphone
|
| 40 |
+
* permission is missing or a recording is running; otherwise returns null, reports the seconds recorded so far to
|
| 41 |
+
* [onProgress] and the result to [onDone], both on the recording thread.
|
| 42 |
+
*/
|
| 43 |
+
fun start(output: File, onProgress: (Double) -> Unit, onDone: (MicResult) -> Unit): String? {
|
| 44 |
+
if (!hasPermission()) return PERMISSION_MESSAGE
|
| 45 |
+
if (isRecording) return "A recording is already running."
|
| 46 |
+
stopRequested = false
|
| 47 |
+
isRecording = true
|
| 48 |
+
thread(name = "mic-recorder") {
|
| 49 |
+
val result =
|
| 50 |
+
try {
|
| 51 |
+
record(output, onProgress)
|
| 52 |
+
} catch (e: SecurityException) {
|
| 53 |
+
MicResult.Failed(PERMISSION_MESSAGE)
|
| 54 |
+
} catch (e: RuntimeException) {
|
| 55 |
+
MicResult.Failed("Recording failed: ${e.message ?: e}")
|
| 56 |
+
}
|
| 57 |
+
isRecording = false
|
| 58 |
+
onDone(result)
|
| 59 |
+
}
|
| 60 |
+
return null
|
| 61 |
+
}
|
| 62 |
+
|
| 63 |
+
/** Ends the recording; [start]'s onDone follows. */
|
| 64 |
+
fun stop() {
|
| 65 |
+
stopRequested = true
|
| 66 |
+
}
|
| 67 |
+
|
| 68 |
+
@SuppressLint("MissingPermission") // checked in start(); a revocation in between ends in SecurityException
|
| 69 |
+
private fun record(output: File, onProgress: (Double) -> Unit): MicResult {
|
| 70 |
+
val minBuffer =
|
| 71 |
+
AudioRecord.getMinBufferSize(SAMPLE_RATE, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT)
|
| 72 |
+
if (minBuffer <= 0) return MicResult.Failed("This phone cannot record 16 kHz mono audio.")
|
| 73 |
+
val recorder =
|
| 74 |
+
AudioRecord(
|
| 75 |
+
MediaRecorder.AudioSource.VOICE_RECOGNITION,
|
| 76 |
+
SAMPLE_RATE,
|
| 77 |
+
AudioFormat.CHANNEL_IN_MONO,
|
| 78 |
+
AudioFormat.ENCODING_PCM_16BIT,
|
| 79 |
+
maxOf(minBuffer, SAMPLE_RATE),
|
| 80 |
+
)
|
| 81 |
+
if (recorder.state != AudioRecord.STATE_INITIALIZED) {
|
| 82 |
+
recorder.release()
|
| 83 |
+
return MicResult.Failed("The microphone could not be opened. Another app may be using it.")
|
| 84 |
+
}
|
| 85 |
+
val pcm = ByteArrayOutputStream()
|
| 86 |
+
val maxBytes = SAMPLE_RATE * BYTES_PER_SAMPLE * MAX_SECONDS
|
| 87 |
+
try {
|
| 88 |
+
recorder.startRecording()
|
| 89 |
+
if (recorder.recordingState != AudioRecord.RECORDSTATE_RECORDING) {
|
| 90 |
+
return MicResult.Failed("The microphone did not start. Another app may be using it.")
|
| 91 |
+
}
|
| 92 |
+
val buffer = ByteArray(SAMPLE_RATE / 10 * BYTES_PER_SAMPLE) // 100 ms
|
| 93 |
+
while (!stopRequested && pcm.size() < maxBytes) {
|
| 94 |
+
val count = recorder.read(buffer, 0, minOf(buffer.size, maxBytes - pcm.size()))
|
| 95 |
+
if (count < 0) return MicResult.Failed("Recording failed (AudioRecord error $count).")
|
| 96 |
+
pcm.write(buffer, 0, count)
|
| 97 |
+
onProgress(pcm.size().toDouble() / (SAMPLE_RATE * BYTES_PER_SAMPLE))
|
| 98 |
+
}
|
| 99 |
+
} finally {
|
| 100 |
+
runCatching { recorder.stop() }
|
| 101 |
+
recorder.release()
|
| 102 |
+
}
|
| 103 |
+
val bytes = pcm.toByteArray()
|
| 104 |
+
if (bytes.isEmpty()) return MicResult.Failed("No audio was recorded.")
|
| 105 |
+
Wav.writeMono16(output, bytes, SAMPLE_RATE)
|
| 106 |
+
return MicResult.Recorded(
|
| 107 |
+
wav = output,
|
| 108 |
+
seconds = bytes.size.toDouble() / (SAMPLE_RATE * BYTES_PER_SAMPLE),
|
| 109 |
+
levels = Wav.levels(bytes),
|
| 110 |
+
stoppedAtLimit = bytes.size >= maxBytes,
|
| 111 |
+
)
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
companion object {
|
| 115 |
+
const val SAMPLE_RATE = 16_000
|
| 116 |
+
const val MAX_SECONDS = 30
|
| 117 |
+
private const val BYTES_PER_SAMPLE = 2
|
| 118 |
+
|
| 119 |
+
const val PERMISSION_MESSAGE =
|
| 120 |
+
"Microphone permission is off, so the app cannot record. The three clips still work. To use the " +
|
| 121 |
+
"microphone, allow it in Settings > Apps > ASR LiteRT-LM > Permissions."
|
| 122 |
+
}
|
| 123 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/ModelProfile.kt
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
/** How a model writes its answer. */
|
| 4 |
+
enum class AnswerFormat {
|
| 5 |
+
/** Qwen3-ASR and its fine-tunes: `language <name><asr_text><transcript>`. */
|
| 6 |
+
LANGUAGE_TAGGED,
|
| 7 |
+
|
| 8 |
+
/** Fun-ASR-Nano: the transcript alone. */
|
| 9 |
+
PLAIN,
|
| 10 |
+
}
|
| 11 |
+
|
| 12 |
+
/** A transcript, split into the language the model named (empty when it named none) and the text. */
|
| 13 |
+
data class Answer(val language: String, val text: String)
|
| 14 |
+
|
| 15 |
+
/**
|
| 16 |
+
* One bundle's contract with this app. All three bundles go through the same calls: one Engine, a new Conversation per
|
| 17 |
+
* clip, one user message holding only the audio (each bundle's own template adds the rest: Fun-ASR's default
|
| 18 |
+
* instruction `语音转写:`, Qwen3-ASR's audio markers), greedy sampling and [maxOutputTokens]. Only the shape of the
|
| 19 |
+
* answer differs ([answerFormat]).
|
| 20 |
+
*/
|
| 21 |
+
data class ModelProfile(
|
| 22 |
+
/** Stable key for launch extras and the device check. */
|
| 23 |
+
val id: String,
|
| 24 |
+
val displayName: String,
|
| 25 |
+
/** The file name the app looks for in its external files directory. */
|
| 26 |
+
val fileName: String,
|
| 27 |
+
val hubRepo: String,
|
| 28 |
+
/** Size of the file on the Hub, shown before the file is on the phone. */
|
| 29 |
+
val hubBytes: Long,
|
| 30 |
+
val license: String,
|
| 31 |
+
val languages: String,
|
| 32 |
+
val answerFormat: AnswerFormat,
|
| 33 |
+
val maxOutputTokens: Int = 512,
|
| 34 |
+
) {
|
| 35 |
+
/** Splits a raw answer into language and text. */
|
| 36 |
+
fun parse(raw: String): Answer =
|
| 37 |
+
when (answerFormat) {
|
| 38 |
+
AnswerFormat.PLAIN -> Answer(language = "", text = raw.trim())
|
| 39 |
+
AnswerFormat.LANGUAGE_TAGGED -> parseTagged(raw)
|
| 40 |
+
}
|
| 41 |
+
|
| 42 |
+
companion object {
|
| 43 |
+
private const val ASR_TEXT = "<asr_text>"
|
| 44 |
+
private const val LANGUAGE_PREFIX = "language "
|
| 45 |
+
|
| 46 |
+
val QWEN3_ASR_1_7B =
|
| 47 |
+
ModelProfile(
|
| 48 |
+
id = "qwen3-asr-1.7b",
|
| 49 |
+
displayName = "Qwen3-ASR-1.7B",
|
| 50 |
+
fileName = "Qwen3-ASR-1.7B.litertlm",
|
| 51 |
+
hubRepo = "litert-community/Qwen3-ASR-1.7B",
|
| 52 |
+
hubBytes = 2_693_064_192L,
|
| 53 |
+
license = "Apache-2.0",
|
| 54 |
+
languages = "30 languages",
|
| 55 |
+
answerFormat = AnswerFormat.LANGUAGE_TAGGED,
|
| 56 |
+
)
|
| 57 |
+
|
| 58 |
+
val FUN_ASR_NANO_2512 =
|
| 59 |
+
ModelProfile(
|
| 60 |
+
id = "fun-asr-nano-2512",
|
| 61 |
+
displayName = "Fun-ASR-Nano-2512",
|
| 62 |
+
fileName = "Fun-ASR-Nano-2512.litertlm",
|
| 63 |
+
hubRepo = "litert-community/Fun-ASR-Nano-2512",
|
| 64 |
+
hubBytes = 1_255_894_736L,
|
| 65 |
+
license = "Apache-2.0",
|
| 66 |
+
languages = "zh / en / ja",
|
| 67 |
+
answerFormat = AnswerFormat.PLAIN,
|
| 68 |
+
)
|
| 69 |
+
|
| 70 |
+
val CONFUCIUS4_R2T2 =
|
| 71 |
+
ModelProfile(
|
| 72 |
+
id = "confucius4-r2t2",
|
| 73 |
+
displayName = "Confucius4-R2T2",
|
| 74 |
+
fileName = "Confucius4-R2T2.litertlm",
|
| 75 |
+
hubRepo = "mlboydaisuke/Confucius4-R2T2-LiteRT",
|
| 76 |
+
hubBytes = 2_693_064_192L,
|
| 77 |
+
license = "NetEase Youdao Model Use License",
|
| 78 |
+
languages = "zh / en (Qwen3-ASR fine-tune)",
|
| 79 |
+
answerFormat = AnswerFormat.LANGUAGE_TAGGED,
|
| 80 |
+
)
|
| 81 |
+
|
| 82 |
+
/** The picker's rows, in screen order. */
|
| 83 |
+
val ALL = listOf(QWEN3_ASR_1_7B, FUN_ASR_NANO_2512, CONFUCIUS4_R2T2)
|
| 84 |
+
|
| 85 |
+
fun byId(id: String?): ModelProfile? = ALL.firstOrNull { it.id == id }
|
| 86 |
+
|
| 87 |
+
/**
|
| 88 |
+
* `language <name><asr_text><text>` -> (name, text), as qwen-asr's parse_asr_output does without a forced
|
| 89 |
+
* language; "language None" gives an empty name, and an answer without the tag is all text.
|
| 90 |
+
*/
|
| 91 |
+
private fun parseTagged(raw: String): Answer {
|
| 92 |
+
val answer = raw.trim()
|
| 93 |
+
val tag = answer.indexOf(ASR_TEXT)
|
| 94 |
+
if (tag < 0) return Answer(language = "", text = answer)
|
| 95 |
+
val meta = answer.substring(0, tag)
|
| 96 |
+
val name =
|
| 97 |
+
meta
|
| 98 |
+
.lineSequence()
|
| 99 |
+
.map { it.trim() }
|
| 100 |
+
.firstOrNull { it.lowercase().startsWith(LANGUAGE_PREFIX) }
|
| 101 |
+
?.substring(LANGUAGE_PREFIX.length)
|
| 102 |
+
?.trim()
|
| 103 |
+
.orEmpty()
|
| 104 |
+
val language = if (name.equals("None", ignoreCase = true)) "" else name
|
| 105 |
+
return Answer(language = language, text = answer.substring(tag + ASR_TEXT.length).trim())
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/PlayBar.kt
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import android.content.Context
|
| 4 |
+
import android.graphics.Canvas
|
| 5 |
+
import android.graphics.Paint
|
| 6 |
+
import android.view.View
|
| 7 |
+
|
| 8 |
+
/** A thin rounded bar: the share of a clip played, or of the 30 s recording limit used. */
|
| 9 |
+
class PlayBar(context: Context, trackColor: Int) : View(context) {
|
| 10 |
+
private val trackPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply { color = trackColor }
|
| 11 |
+
private val fillPaint = Paint(Paint.ANTI_ALIAS_FLAG)
|
| 12 |
+
|
| 13 |
+
var fraction = 0f
|
| 14 |
+
set(value) {
|
| 15 |
+
if (value != field) {
|
| 16 |
+
field = value
|
| 17 |
+
invalidate()
|
| 18 |
+
}
|
| 19 |
+
}
|
| 20 |
+
|
| 21 |
+
fun set(fraction: Float, fillColor: Int) {
|
| 22 |
+
fillPaint.color = fillColor
|
| 23 |
+
this.fraction = fraction
|
| 24 |
+
invalidate()
|
| 25 |
+
}
|
| 26 |
+
|
| 27 |
+
override fun onDraw(canvas: Canvas) {
|
| 28 |
+
val height = height.toFloat()
|
| 29 |
+
val radius = height / 2
|
| 30 |
+
canvas.drawRoundRect(0f, 0f, width.toFloat(), height, radius, radius, trackPaint)
|
| 31 |
+
if (fraction > 0f) {
|
| 32 |
+
canvas.drawRoundRect(0f, 0f, maxOf(height, width * fraction), height, radius, radius, fillPaint)
|
| 33 |
+
}
|
| 34 |
+
}
|
| 35 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/TextMatch.kt
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import java.text.Normalizer
|
| 4 |
+
|
| 5 |
+
/**
|
| 6 |
+
* Transcript comparison for the device check, the way the model cards score: NFKC, lowercase, every punctuation,
|
| 7 |
+
* separator and whitespace character removed, then the character error rate against the expected text.
|
| 8 |
+
*/
|
| 9 |
+
object TextMatch {
|
| 10 |
+
fun normalize(text: String): String {
|
| 11 |
+
val folded = Normalizer.normalize(text, Normalizer.Form.NFKC).lowercase()
|
| 12 |
+
val out = StringBuilder()
|
| 13 |
+
var i = 0
|
| 14 |
+
while (i < folded.length) {
|
| 15 |
+
val codePoint = folded.codePointAt(i)
|
| 16 |
+
if (!isDropped(codePoint)) out.appendCodePoint(codePoint)
|
| 17 |
+
i += Character.charCount(codePoint)
|
| 18 |
+
}
|
| 19 |
+
return out.toString()
|
| 20 |
+
}
|
| 21 |
+
|
| 22 |
+
/** Edit distance between the normalized texts over the normalized [expected] length (code points). */
|
| 23 |
+
fun cer(expected: String, actual: String): Double {
|
| 24 |
+
val reference = normalize(expected).codePoints().toArray()
|
| 25 |
+
val hypothesis = normalize(actual).codePoints().toArray()
|
| 26 |
+
if (reference.isEmpty()) return if (hypothesis.isEmpty()) 0.0 else 1.0
|
| 27 |
+
return editDistance(reference, hypothesis).toDouble() / reference.size
|
| 28 |
+
}
|
| 29 |
+
|
| 30 |
+
/** True when the two texts are equal after [normalize]. */
|
| 31 |
+
fun matches(expected: String, actual: String): Boolean = normalize(expected) == normalize(actual)
|
| 32 |
+
|
| 33 |
+
private fun isDropped(codePoint: Int): Boolean =
|
| 34 |
+
Character.isWhitespace(codePoint) ||
|
| 35 |
+
when (Character.getType(codePoint).toByte()) {
|
| 36 |
+
Character.CONNECTOR_PUNCTUATION,
|
| 37 |
+
Character.DASH_PUNCTUATION,
|
| 38 |
+
Character.START_PUNCTUATION,
|
| 39 |
+
Character.END_PUNCTUATION,
|
| 40 |
+
Character.INITIAL_QUOTE_PUNCTUATION,
|
| 41 |
+
Character.FINAL_QUOTE_PUNCTUATION,
|
| 42 |
+
Character.OTHER_PUNCTUATION,
|
| 43 |
+
Character.SPACE_SEPARATOR,
|
| 44 |
+
Character.LINE_SEPARATOR,
|
| 45 |
+
Character.PARAGRAPH_SEPARATOR -> true
|
| 46 |
+
else -> false
|
| 47 |
+
}
|
| 48 |
+
|
| 49 |
+
private fun editDistance(a: IntArray, b: IntArray): Int {
|
| 50 |
+
var previous = IntArray(b.size + 1) { it }
|
| 51 |
+
var current = IntArray(b.size + 1)
|
| 52 |
+
for (i in 1..a.size) {
|
| 53 |
+
current[0] = i
|
| 54 |
+
for (j in 1..b.size) {
|
| 55 |
+
val substitution = previous[j - 1] + if (a[i - 1] == b[j - 1]) 0 else 1
|
| 56 |
+
current[j] = minOf(substitution, previous[j] + 1, current[j - 1] + 1)
|
| 57 |
+
}
|
| 58 |
+
val swap = previous
|
| 59 |
+
previous = current
|
| 60 |
+
current = swap
|
| 61 |
+
}
|
| 62 |
+
return previous[b.size]
|
| 63 |
+
}
|
| 64 |
+
}
|
android/app/src/main/kotlin/com/asrlitertlm/Wav.kt
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import java.io.File
|
| 4 |
+
import java.io.RandomAccessFile
|
| 5 |
+
import java.nio.ByteBuffer
|
| 6 |
+
import java.nio.ByteOrder
|
| 7 |
+
import kotlin.math.abs
|
| 8 |
+
import kotlin.math.log10
|
| 9 |
+
import kotlin.math.max
|
| 10 |
+
|
| 11 |
+
/**
|
| 12 |
+
* Minimal RIFF/WAVE helpers: the length and the samples of a wav (walks the chunks; the bundled clips carry a LIST
|
| 13 |
+
* chunk), a 16-bit mono writer for the microphone, a trimmed copy for clips longer than one model window, and the
|
| 14 |
+
* level of a recording.
|
| 15 |
+
*/
|
| 16 |
+
object Wav {
|
| 17 |
+
/** 16-bit PCM samples of a wav, interleaved when [channels] > 1. */
|
| 18 |
+
class Pcm(val data: ByteArray, val sampleRate: Int, val channels: Int) {
|
| 19 |
+
/** Frames (samples per channel). */
|
| 20 |
+
val frames: Int
|
| 21 |
+
get() = data.size / (2 * channels)
|
| 22 |
+
|
| 23 |
+
val seconds: Double
|
| 24 |
+
get() = frames.toDouble() / sampleRate
|
| 25 |
+
}
|
| 26 |
+
|
| 27 |
+
/** Peak and RMS of a recording in dBFS (-120 for digital silence). */
|
| 28 |
+
data class Levels(val peakDbfs: Double, val rmsDbfs: Double)
|
| 29 |
+
|
| 30 |
+
private class Info(
|
| 31 |
+
val sampleRate: Int,
|
| 32 |
+
val channels: Int,
|
| 33 |
+
val bits: Int,
|
| 34 |
+
val byteRate: Int,
|
| 35 |
+
val dataSize: Long,
|
| 36 |
+
val data: ByteArray?,
|
| 37 |
+
)
|
| 38 |
+
|
| 39 |
+
private fun read(file: File, withData: Boolean): Info {
|
| 40 |
+
RandomAccessFile(file, "r").use { f ->
|
| 41 |
+
val head = ByteArray(12).also { f.readFully(it) }
|
| 42 |
+
require(String(head, 0, 4) == "RIFF" && String(head, 8, 4) == "WAVE") { "not a wav: $file" }
|
| 43 |
+
var fmt: ByteBuffer? = null
|
| 44 |
+
while (f.filePointer + 8 <= f.length()) {
|
| 45 |
+
val header = ByteArray(8).also { f.readFully(it) }
|
| 46 |
+
val id = String(header, 0, 4)
|
| 47 |
+
val size = ByteBuffer.wrap(header, 4, 4).order(ByteOrder.LITTLE_ENDIAN).int.toLong() and 0xffffffffL
|
| 48 |
+
when (id) {
|
| 49 |
+
"fmt " -> {
|
| 50 |
+
fmt = ByteBuffer.wrap(ByteArray(size.toInt()).also { f.readFully(it) }).order(ByteOrder.LITTLE_ENDIAN)
|
| 51 |
+
if (size % 2 == 1L) f.skipBytes(1)
|
| 52 |
+
}
|
| 53 |
+
"data" -> {
|
| 54 |
+
val format = requireNotNull(fmt) { "data before fmt in $file" }
|
| 55 |
+
// A recorder that never patched the header reports 0 or 0xFFFFFFFF: take what the file holds.
|
| 56 |
+
val available = f.length() - f.filePointer
|
| 57 |
+
val length = if (size == 0L || size > available) available else size
|
| 58 |
+
val data = if (withData) ByteArray(length.toInt()).also { f.readFully(it) } else null
|
| 59 |
+
return Info(
|
| 60 |
+
sampleRate = format.getInt(4),
|
| 61 |
+
channels = format.getShort(2).toInt(),
|
| 62 |
+
bits = format.getShort(14).toInt(),
|
| 63 |
+
byteRate = format.getInt(8),
|
| 64 |
+
dataSize = length,
|
| 65 |
+
data = data,
|
| 66 |
+
)
|
| 67 |
+
}
|
| 68 |
+
else -> f.seek(f.filePointer + size + (size % 2))
|
| 69 |
+
}
|
| 70 |
+
}
|
| 71 |
+
}
|
| 72 |
+
error("no data chunk in $file")
|
| 73 |
+
}
|
| 74 |
+
|
| 75 |
+
/** The length of a wav in seconds, from its header. */
|
| 76 |
+
fun seconds(file: File): Double = read(file, withData = false).let { it.dataSize.toDouble() / it.byteRate }
|
| 77 |
+
|
| 78 |
+
/** The samples of a 16-bit PCM wav. */
|
| 79 |
+
fun pcm16(file: File): Pcm =
|
| 80 |
+
read(file, withData = true).let {
|
| 81 |
+
require(it.bits == 16) { "not 16-bit PCM: $file" }
|
| 82 |
+
Pcm(it.data!!, it.sampleRate, it.channels)
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
/** Writes 16-bit mono PCM as a canonical 44-byte-header wav. */
|
| 86 |
+
fun writeMono16(file: File, pcm: ByteArray, sampleRate: Int) {
|
| 87 |
+
val header =
|
| 88 |
+
ByteBuffer.allocate(44).order(ByteOrder.LITTLE_ENDIAN).apply {
|
| 89 |
+
put("RIFF".toByteArray())
|
| 90 |
+
putInt(36 + pcm.size)
|
| 91 |
+
put("WAVE".toByteArray())
|
| 92 |
+
put("fmt ".toByteArray())
|
| 93 |
+
putInt(16)
|
| 94 |
+
putShort(1)
|
| 95 |
+
putShort(1)
|
| 96 |
+
putInt(sampleRate)
|
| 97 |
+
putInt(sampleRate * 2)
|
| 98 |
+
putShort(2)
|
| 99 |
+
putShort(16)
|
| 100 |
+
put("data".toByteArray())
|
| 101 |
+
putInt(pcm.size)
|
| 102 |
+
}
|
| 103 |
+
file.outputStream().use {
|
| 104 |
+
it.write(header.array())
|
| 105 |
+
it.write(pcm)
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
|
| 109 |
+
/**
|
| 110 |
+
* Returns [source] when it is at most [maxSeconds] long; otherwise writes its first [maxSeconds] (mono, 16-bit)
|
| 111 |
+
* to [target] and returns that. The models read one window of about 30 s per message.
|
| 112 |
+
*/
|
| 113 |
+
fun trimmed(source: File, maxSeconds: Double, target: File): File {
|
| 114 |
+
if (seconds(source) <= maxSeconds) return source
|
| 115 |
+
val pcm = pcm16(source)
|
| 116 |
+
require(pcm.channels == 1) { "only mono clips are trimmed: $source" }
|
| 117 |
+
val keepBytes = (maxSeconds * pcm.sampleRate).toInt() * 2
|
| 118 |
+
writeMono16(target, pcm.data.copyOf(keepBytes), pcm.sampleRate)
|
| 119 |
+
return target
|
| 120 |
+
}
|
| 121 |
+
|
| 122 |
+
/** Peak and RMS of 16-bit little-endian PCM. */
|
| 123 |
+
fun levels(pcm: ByteArray): Levels {
|
| 124 |
+
val samples = ByteBuffer.wrap(pcm).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer()
|
| 125 |
+
var peak = 0
|
| 126 |
+
var sumOfSquares = 0.0
|
| 127 |
+
for (k in 0 until samples.limit()) {
|
| 128 |
+
val value = samples.get(k).toInt()
|
| 129 |
+
peak = max(peak, abs(value))
|
| 130 |
+
sumOfSquares += value.toDouble() * value
|
| 131 |
+
}
|
| 132 |
+
val count = max(samples.limit(), 1)
|
| 133 |
+
return Levels(peakDbfs = toDbfs(peak.toDouble()), rmsDbfs = toDbfs(Math.sqrt(sumOfSquares / count)))
|
| 134 |
+
}
|
| 135 |
+
|
| 136 |
+
private fun toDbfs(amplitude: Double): Double =
|
| 137 |
+
if (amplitude <= 0.0) SILENCE_DBFS else max(SILENCE_DBFS, 20 * log10(amplitude / 32768.0))
|
| 138 |
+
|
| 139 |
+
private const val SILENCE_DBFS = -120.0
|
| 140 |
+
}
|
android/app/src/main/res/raw/fleurs_cmn_1698.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f72a74b2b83aee568941fe6bd174309a5a54ac9d2dc0b698f6587a8b29aeffd2
|
| 3 |
+
size 311084
|
android/app/src/main/res/raw/fleurs_en_1698.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a696132b9f79f7936d470a6dc6fdb19ce009b4701165afe9097fa970923dff76
|
| 3 |
+
size 205484
|
android/app/src/main/res/raw/fleurs_ja_1698.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1235eb34a774a5b52e2032e0a344765cbc7decadad943d8ae77872c482599ba7
|
| 3 |
+
size 353324
|
android/app/src/main/res/values/strings.xml
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<resources>
|
| 3 |
+
<string name="app_name">ASR LiteRT-LM</string>
|
| 4 |
+
</resources>
|
android/app/src/main/res/values/themes.xml
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<?xml version="1.0" encoding="utf-8"?>
|
| 2 |
+
<resources>
|
| 3 |
+
<style name="Theme.AsrLitertlm" parent="@android:style/Theme.DeviceDefault.NoActionBar">
|
| 4 |
+
<item name="android:windowBackground">@color/bg</item>
|
| 5 |
+
<item name="android:colorBackground">@color/bg</item>
|
| 6 |
+
<item name="android:statusBarColor">@color/bg</item>
|
| 7 |
+
<item name="android:navigationBarColor">@color/bg</item>
|
| 8 |
+
<item name="android:windowLightStatusBar">false</item>
|
| 9 |
+
<!-- A plain dark launch screen: no default launcher icon on the system splash. -->
|
| 10 |
+
<item name="android:windowSplashScreenBackground">@color/bg</item>
|
| 11 |
+
<item name="android:windowSplashScreenAnimatedIcon">@android:color/transparent</item>
|
| 12 |
+
</style>
|
| 13 |
+
|
| 14 |
+
<color name="bg">#0E1116</color>
|
| 15 |
+
</resources>
|
android/app/src/test/kotlin/com/asrlitertlm/AsrLogicTest.kt
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package com.asrlitertlm
|
| 2 |
+
|
| 3 |
+
import java.io.File
|
| 4 |
+
import java.nio.ByteBuffer
|
| 5 |
+
import java.nio.ByteOrder
|
| 6 |
+
import kotlin.math.sin
|
| 7 |
+
import org.junit.Assert.assertEquals
|
| 8 |
+
import org.junit.Assert.assertFalse
|
| 9 |
+
import org.junit.Assert.assertSame
|
| 10 |
+
import org.junit.Assert.assertTrue
|
| 11 |
+
import org.junit.Rule
|
| 12 |
+
import org.junit.Test
|
| 13 |
+
import org.junit.rules.TemporaryFolder
|
| 14 |
+
|
| 15 |
+
/** The parts of the app that run without a phone: answer parsing, transcript matching and the wav helpers. */
|
| 16 |
+
class AsrLogicTest {
|
| 17 |
+
@get:Rule val folder = TemporaryFolder()
|
| 18 |
+
|
| 19 |
+
@Test
|
| 20 |
+
fun taggedAnswerSplitsLanguageAndText() {
|
| 21 |
+
val answer =
|
| 22 |
+
ModelProfile.QWEN3_ASR_1_7B.parse(
|
| 23 |
+
"language English<asr_text>Everything in the universe is made of matter."
|
| 24 |
+
)
|
| 25 |
+
assertEquals(Answer("English", "Everything in the universe is made of matter."), answer)
|
| 26 |
+
}
|
| 27 |
+
|
| 28 |
+
@Test
|
| 29 |
+
fun taggedAnswerWithLanguageNoneHasNoLanguage() {
|
| 30 |
+
assertEquals(Answer("", "hello"), ModelProfile.CONFUCIUS4_R2T2.parse("language None<asr_text>hello"))
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
@Test
|
| 34 |
+
fun answerWithoutTagIsAllText() {
|
| 35 |
+
assertEquals(Answer("", "宇宙中的一切"), ModelProfile.QWEN3_ASR_1_7B.parse(" 宇宙中的一切 "))
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
@Test
|
| 39 |
+
fun plainAnswerKeepsTheTagLikeText() {
|
| 40 |
+
// Fun-ASR writes the transcript alone; nothing in it is read as a language tag.
|
| 41 |
+
assertEquals(
|
| 42 |
+
Answer("", "language English<asr_text>x"),
|
| 43 |
+
ModelProfile.FUN_ASR_NANO_2512.parse("language English<asr_text>x"),
|
| 44 |
+
)
|
| 45 |
+
}
|
| 46 |
+
|
| 47 |
+
@Test
|
| 48 |
+
fun normalizationDropsPunctuationSpaceAndCase() {
|
| 49 |
+
assertTrue(
|
| 50 |
+
TextMatch.matches(
|
| 51 |
+
"Everything in the universe is made of matter. All matter is made of tiny particles called atoms.",
|
| 52 |
+
"everything in the Universe is made of matter, all matter is made of tiny particles called atoms",
|
| 53 |
+
)
|
| 54 |
+
)
|
| 55 |
+
assertTrue(TextMatch.matches("宇宙中的一切都由物质构成,而所有的物质", "宇宙中的一切都由物质构成 而所有的物质。"))
|
| 56 |
+
assertTrue(TextMatch.matches("ABC123", "abc123")) // NFKC folds full-width forms
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
@Test
|
| 60 |
+
fun cerCountsCharacterEdits() {
|
| 61 |
+
// 微小粒子 -> 微小颗粒: two substitutions over 32 characters (punctuation removed).
|
| 62 |
+
val reference = "宇宙中的一切都由物质构成,而所有的物质都由被称为原子的微小粒子组成。"
|
| 63 |
+
val hypothesis = "宇宙中的一切都由物质构成,而所有的物质都由被称为原子的微小颗粒组成。"
|
| 64 |
+
assertEquals(2.0 / 32.0, TextMatch.cer(reference, hypothesis), 1e-9)
|
| 65 |
+
assertEquals(0.0, TextMatch.cer(reference, reference), 0.0)
|
| 66 |
+
assertEquals(1.0, TextMatch.cer(reference, ""), 0.0)
|
| 67 |
+
}
|
| 68 |
+
|
| 69 |
+
@Test
|
| 70 |
+
fun shortWavIsSentAsIs() {
|
| 71 |
+
val wav = tone(seconds = 2.0)
|
| 72 |
+
assertSame(wav, Wav.trimmed(wav, 30.0, folder.newFile("cut.wav")))
|
| 73 |
+
}
|
| 74 |
+
|
| 75 |
+
@Test
|
| 76 |
+
fun longWavIsCutAtThirtySeconds() {
|
| 77 |
+
val wav = tone(seconds = 33.6)
|
| 78 |
+
assertEquals(33.6, Wav.seconds(wav), 1e-6)
|
| 79 |
+
val cut = Wav.trimmed(wav, 30.0, folder.newFile("cut.wav"))
|
| 80 |
+
assertEquals(30.0, Wav.seconds(cut), 1e-6)
|
| 81 |
+
val original = Wav.pcm16(wav).data
|
| 82 |
+
val kept = Wav.pcm16(cut).data
|
| 83 |
+
assertTrue(kept.contentEquals(original.copyOf(kept.size)))
|
| 84 |
+
}
|
| 85 |
+
|
| 86 |
+
@Test
|
| 87 |
+
fun silenceAndToneLevels() {
|
| 88 |
+
val silence = Wav.levels(ByteArray(32_000))
|
| 89 |
+
assertEquals(-120.0, silence.rmsDbfs, 0.0)
|
| 90 |
+
assertTrue(silence.rmsDbfs < AsrSession.NO_SPEECH_RMS_DBFS)
|
| 91 |
+
val tone = Wav.levels(Wav.pcm16(tone(seconds = 1.0)).data)
|
| 92 |
+
// A sine at half scale: peak about -6 dBFS, RMS about -9 dBFS.
|
| 93 |
+
assertEquals(-6.0, tone.peakDbfs, 0.1)
|
| 94 |
+
assertEquals(-9.0, tone.rmsDbfs, 0.1)
|
| 95 |
+
assertFalse(tone.rmsDbfs < AsrSession.NO_SPEECH_RMS_DBFS)
|
| 96 |
+
}
|
| 97 |
+
|
| 98 |
+
private fun tone(seconds: Double, rate: Int = 16_000): File {
|
| 99 |
+
val samples = (seconds * rate).toInt()
|
| 100 |
+
val pcm = ByteBuffer.allocate(samples * 2).order(ByteOrder.LITTLE_ENDIAN)
|
| 101 |
+
for (i in 0 until samples) pcm.putShort((16384 * sin(2 * Math.PI * 440 * i / rate)).toInt().toShort())
|
| 102 |
+
return folder.newFile().also { Wav.writeMono16(it, pcm.array(), rate) }
|
| 103 |
+
}
|
| 104 |
+
}
|
android/build.gradle.kts
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// AGP 9 has Kotlin built in; do not apply org.jetbrains.kotlin.android.
|
| 2 |
+
// litertlm-android 0.17.1 ships Kotlin 2.4 metadata. The Kotlin Gradle plugin that AGP 9.3.1 brings (2.2.10) reads
|
| 3 |
+
// metadata up to 2.3 only, so the build classpath gets KGP 2.4.0, which the built-in Kotlin then uses.
|
| 4 |
+
buildscript {
|
| 5 |
+
repositories { mavenCentral() }
|
| 6 |
+
dependencies { classpath("org.jetbrains.kotlin:kotlin-gradle-plugin:2.4.0") }
|
| 7 |
+
}
|
| 8 |
+
|
| 9 |
+
plugins { id("com.android.application") version "9.3.1" apply false }
|
android/gradle.properties
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
org.gradle.jvmargs=-Xmx4g -Dfile.encoding=UTF-8
|
| 2 |
+
android.useAndroidX=true
|
| 3 |
+
kotlin.code.style=official
|
| 4 |
+
# In-process Kotlin compilation: the daemon writes under ~/Library, which sandboxed builds cannot.
|
| 5 |
+
kotlin.compiler.execution.strategy=in-process
|
| 6 |
+
# The LiteRT-LM runtime the three model cards' Kotlin examples were run with; shown on screen as the runtime version.
|
| 7 |
+
litertlmVersion=0.17.1
|
android/gradle/wrapper/gradle-wrapper.jar
ADDED
|
Binary file (47.5 kB). View file
|
|
|
android/gradle/wrapper/gradle-wrapper.properties
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
distributionBase=GRADLE_USER_HOME
|
| 2 |
+
distributionPath=wrapper/dists
|
| 3 |
+
distributionUrl=https\://services.gradle.org/distributions/gradle-9.7.0-bin.zip
|
| 4 |
+
networkTimeout=10000
|
| 5 |
+
validateDistributionUrl=true
|
| 6 |
+
zipStoreBase=GRADLE_USER_HOME
|
| 7 |
+
zipStorePath=wrapper/dists
|
android/gradlew
ADDED
|
@@ -0,0 +1,248 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/sh
|
| 2 |
+
|
| 3 |
+
#
|
| 4 |
+
# Copyright © 2015 the original authors.
|
| 5 |
+
#
|
| 6 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 7 |
+
# you may not use this file except in compliance with the License.
|
| 8 |
+
# You may obtain a copy of the License at
|
| 9 |
+
#
|
| 10 |
+
# https://www.apache.org/licenses/LICENSE-2.0
|
| 11 |
+
#
|
| 12 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 13 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 14 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 15 |
+
# See the License for the specific language governing permissions and
|
| 16 |
+
# limitations under the License.
|
| 17 |
+
#
|
| 18 |
+
# SPDX-License-Identifier: Apache-2.0
|
| 19 |
+
#
|
| 20 |
+
|
| 21 |
+
##############################################################################
|
| 22 |
+
#
|
| 23 |
+
# gradlew start up script for POSIX generated by Gradle.
|
| 24 |
+
#
|
| 25 |
+
# Important for running:
|
| 26 |
+
#
|
| 27 |
+
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
|
| 28 |
+
# noncompliant, but you have some other compliant shell such as ksh or
|
| 29 |
+
# bash, then to run this script, type that shell name before the whole
|
| 30 |
+
# command line, like:
|
| 31 |
+
#
|
| 32 |
+
# ksh gradlew
|
| 33 |
+
#
|
| 34 |
+
# Busybox and similar reduced shells will NOT work, because this script
|
| 35 |
+
# requires all of these POSIX shell features:
|
| 36 |
+
# * functions;
|
| 37 |
+
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
|
| 38 |
+
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
|
| 39 |
+
# * compound commands having a testable exit status, especially «case»;
|
| 40 |
+
# * various built-in commands including «command», «set», and «ulimit».
|
| 41 |
+
#
|
| 42 |
+
# Important for patching:
|
| 43 |
+
#
|
| 44 |
+
# (2) This script targets any POSIX shell, so it avoids extensions provided
|
| 45 |
+
# by Bash, Ksh, etc; in particular arrays are avoided.
|
| 46 |
+
#
|
| 47 |
+
# The "traditional" practice of packing multiple parameters into a
|
| 48 |
+
# space-separated string is a well documented source of bugs and security
|
| 49 |
+
# problems, so this is (mostly) avoided, by progressively accumulating
|
| 50 |
+
# options in "$@", and eventually passing that to Java.
|
| 51 |
+
#
|
| 52 |
+
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
|
| 53 |
+
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
|
| 54 |
+
# see the in-line comments for details.
|
| 55 |
+
#
|
| 56 |
+
# There are tweaks for specific operating systems such as AIX, CygWin,
|
| 57 |
+
# Darwin, MinGW, and NonStop.
|
| 58 |
+
#
|
| 59 |
+
# (3) This script is generated from the Groovy template
|
| 60 |
+
# https://github.com/gradle/gradle/blob/3d91ce3b8caaf77ad09f381f43615b715b53f72c/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
|
| 61 |
+
# within the Gradle project.
|
| 62 |
+
#
|
| 63 |
+
# You can find Gradle at https://github.com/gradle/gradle/.
|
| 64 |
+
#
|
| 65 |
+
##############################################################################
|
| 66 |
+
|
| 67 |
+
# Attempt to set APP_HOME
|
| 68 |
+
|
| 69 |
+
# Resolve links: $0 may be a link
|
| 70 |
+
app_path=$0
|
| 71 |
+
|
| 72 |
+
# Need this for daisy-chained symlinks.
|
| 73 |
+
while
|
| 74 |
+
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
|
| 75 |
+
[ -h "$app_path" ]
|
| 76 |
+
do
|
| 77 |
+
ls=$( ls -ld "$app_path" )
|
| 78 |
+
link=${ls#*' -> '}
|
| 79 |
+
case $link in #(
|
| 80 |
+
/*) app_path=$link ;; #(
|
| 81 |
+
*) app_path=$APP_HOME$link ;;
|
| 82 |
+
esac
|
| 83 |
+
done
|
| 84 |
+
|
| 85 |
+
# This is normally unused
|
| 86 |
+
# shellcheck disable=SC2034
|
| 87 |
+
APP_BASE_NAME=${0##*/}
|
| 88 |
+
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
|
| 89 |
+
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
|
| 90 |
+
|
| 91 |
+
# Use the maximum available, or set MAX_FD != -1 to use that value.
|
| 92 |
+
MAX_FD=maximum
|
| 93 |
+
|
| 94 |
+
warn () {
|
| 95 |
+
echo "$*"
|
| 96 |
+
} >&2
|
| 97 |
+
|
| 98 |
+
die () {
|
| 99 |
+
echo
|
| 100 |
+
echo "$*"
|
| 101 |
+
echo
|
| 102 |
+
exit 1
|
| 103 |
+
} >&2
|
| 104 |
+
|
| 105 |
+
# OS specific support (must be 'true' or 'false').
|
| 106 |
+
cygwin=false
|
| 107 |
+
msys=false
|
| 108 |
+
darwin=false
|
| 109 |
+
nonstop=false
|
| 110 |
+
case "$( uname )" in #(
|
| 111 |
+
CYGWIN* ) cygwin=true ;; #(
|
| 112 |
+
Darwin* ) darwin=true ;; #(
|
| 113 |
+
MSYS* | MINGW* ) msys=true ;; #(
|
| 114 |
+
NONSTOP* ) nonstop=true ;;
|
| 115 |
+
esac
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
# Determine the Java command to use to start the JVM.
|
| 120 |
+
if [ -n "$JAVA_HOME" ] ; then
|
| 121 |
+
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
|
| 122 |
+
# IBM's JDK on AIX uses strange locations for the executables
|
| 123 |
+
JAVACMD=$JAVA_HOME/jre/sh/java
|
| 124 |
+
else
|
| 125 |
+
JAVACMD=$JAVA_HOME/bin/java
|
| 126 |
+
fi
|
| 127 |
+
if [ ! -x "$JAVACMD" ] ; then
|
| 128 |
+
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
|
| 129 |
+
|
| 130 |
+
Please set the JAVA_HOME variable in your environment to match the
|
| 131 |
+
location of your Java installation."
|
| 132 |
+
fi
|
| 133 |
+
else
|
| 134 |
+
JAVACMD=java
|
| 135 |
+
if ! command -v java >/dev/null 2>&1
|
| 136 |
+
then
|
| 137 |
+
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
|
| 138 |
+
|
| 139 |
+
Please set the JAVA_HOME variable in your environment to match the
|
| 140 |
+
location of your Java installation."
|
| 141 |
+
fi
|
| 142 |
+
fi
|
| 143 |
+
|
| 144 |
+
# Increase the maximum file descriptors if we can.
|
| 145 |
+
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
|
| 146 |
+
case $MAX_FD in #(
|
| 147 |
+
max*)
|
| 148 |
+
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
|
| 149 |
+
# shellcheck disable=SC2039,SC3045
|
| 150 |
+
MAX_FD=$( ulimit -H -n ) ||
|
| 151 |
+
warn "Could not query maximum file descriptor limit"
|
| 152 |
+
esac
|
| 153 |
+
case $MAX_FD in #(
|
| 154 |
+
'' | soft) :;; #(
|
| 155 |
+
*)
|
| 156 |
+
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
|
| 157 |
+
# shellcheck disable=SC2039,SC3045
|
| 158 |
+
ulimit -n "$MAX_FD" ||
|
| 159 |
+
warn "Could not set maximum file descriptor limit to $MAX_FD"
|
| 160 |
+
esac
|
| 161 |
+
fi
|
| 162 |
+
|
| 163 |
+
# Collect all arguments for the java command, stacking in reverse order:
|
| 164 |
+
# * args from the command line
|
| 165 |
+
# * the main class name
|
| 166 |
+
# * -classpath
|
| 167 |
+
# * -D...appname settings
|
| 168 |
+
# * --module-path (only if needed)
|
| 169 |
+
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
|
| 170 |
+
|
| 171 |
+
# For Cygwin or MSYS, switch paths to Windows format before running java
|
| 172 |
+
if "$cygwin" || "$msys" ; then
|
| 173 |
+
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
|
| 174 |
+
|
| 175 |
+
JAVACMD=$( cygpath --unix "$JAVACMD" )
|
| 176 |
+
|
| 177 |
+
# Now convert the arguments - kludge to limit ourselves to /bin/sh
|
| 178 |
+
for arg do
|
| 179 |
+
if
|
| 180 |
+
case $arg in #(
|
| 181 |
+
-*) false ;; # don't mess with options #(
|
| 182 |
+
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
|
| 183 |
+
[ -e "$t" ] ;; #(
|
| 184 |
+
*) false ;;
|
| 185 |
+
esac
|
| 186 |
+
then
|
| 187 |
+
arg=$( cygpath --path --ignore --mixed "$arg" )
|
| 188 |
+
fi
|
| 189 |
+
# Roll the args list around exactly as many times as the number of
|
| 190 |
+
# args, so each arg winds up back in the position where it started, but
|
| 191 |
+
# possibly modified.
|
| 192 |
+
#
|
| 193 |
+
# NB: a `for` loop captures its iteration list before it begins, so
|
| 194 |
+
# changing the positional parameters here affects neither the number of
|
| 195 |
+
# iterations, nor the values presented in `arg`.
|
| 196 |
+
shift # remove old arg
|
| 197 |
+
set -- "$@" "$arg" # push replacement arg
|
| 198 |
+
done
|
| 199 |
+
fi
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
|
| 203 |
+
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
|
| 204 |
+
|
| 205 |
+
# Collect all arguments for the java command:
|
| 206 |
+
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
|
| 207 |
+
# and any embedded shellness will be escaped.
|
| 208 |
+
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
|
| 209 |
+
# treated as '${Hostname}' itself on the command line.
|
| 210 |
+
|
| 211 |
+
set -- \
|
| 212 |
+
"-Dorg.gradle.appname=$APP_BASE_NAME" \
|
| 213 |
+
-jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \
|
| 214 |
+
"$@"
|
| 215 |
+
|
| 216 |
+
# Stop when "xargs" is not available.
|
| 217 |
+
if ! command -v xargs >/dev/null 2>&1
|
| 218 |
+
then
|
| 219 |
+
die "xargs is not available"
|
| 220 |
+
fi
|
| 221 |
+
|
| 222 |
+
# Use "xargs" to parse quoted args.
|
| 223 |
+
#
|
| 224 |
+
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
|
| 225 |
+
#
|
| 226 |
+
# In Bash we could simply go:
|
| 227 |
+
#
|
| 228 |
+
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
|
| 229 |
+
# set -- "${ARGS[@]}" "$@"
|
| 230 |
+
#
|
| 231 |
+
# but POSIX shell has neither arrays nor command substitution, so instead we
|
| 232 |
+
# post-process each arg (as a line of input to sed) to backslash-escape any
|
| 233 |
+
# character that might be a shell metacharacter, then use eval to reverse
|
| 234 |
+
# that process (while maintaining the separation between arguments), and wrap
|
| 235 |
+
# the whole thing up as a single "set" statement.
|
| 236 |
+
#
|
| 237 |
+
# This will of course break if any of these variables contains a newline or
|
| 238 |
+
# an unmatched quote.
|
| 239 |
+
#
|
| 240 |
+
|
| 241 |
+
eval "set -- $(
|
| 242 |
+
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
|
| 243 |
+
xargs -n1 |
|
| 244 |
+
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
|
| 245 |
+
tr '\n' ' '
|
| 246 |
+
)" '"$@"'
|
| 247 |
+
|
| 248 |
+
exec "$JAVACMD" "$@"
|
android/settings.gradle.kts
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pluginManagement {
|
| 2 |
+
repositories {
|
| 3 |
+
google()
|
| 4 |
+
mavenCentral()
|
| 5 |
+
gradlePluginPortal()
|
| 6 |
+
}
|
| 7 |
+
}
|
| 8 |
+
|
| 9 |
+
dependencyResolutionManagement {
|
| 10 |
+
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
|
| 11 |
+
// The LiteRT-LM AARs live on Google Maven.
|
| 12 |
+
repositories {
|
| 13 |
+
google()
|
| 14 |
+
mavenCentral()
|
| 15 |
+
}
|
| 16 |
+
}
|
| 17 |
+
|
| 18 |
+
rootProject.name = "asr-litertlm"
|
| 19 |
+
|
| 20 |
+
include(":app")
|