File size: 13,480 Bytes
bbdc3c1 c51367e b6d72cc bbdc3c1 0dfe32a b8f1336 0dfe32a b8f1336 0dfe32a 6b53dac d28f709 6b53dac d28f709 6b53dac 0dfe32a d28f709 0dfe32a 6b53dac 9a604d7 b8f1336 8ac3d3f bbdc3c1 028091a bf3e4b8 028091a bbdc3c1 d28f709 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 | ---
license: apache-2.0
base_model: openai/whisper-base
language:
- ms
- en
pipeline_tag: audio-classification
library_name: transformers
tags:
- end-of-turn-detection
- turn-detection
- semantic-vad
- endpointing
- voice-agent
- livekit
- onnx
- whisper
- telephony
---
# Semantic VAD — Whisper-base end-of-turn detector (audio only)
The whisper-base sibling of
[`Scicom-intl/semantic-vad-eot-whisper-tiny`](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-tiny):
same recipe, same input contract, same data, a 2.5× larger encoder. Given the last 8 seconds of a caller's
16 kHz audio it returns `p(end of turn)` — finished speaking vs paused mid-sentence — with no transcript.
**20 M parameters · int8 ONNX 24 MB · roughly twice the compute of the tiny model** (int8 69 ms vs 34 ms per
prediction measured back to back on the same busy CPU; the tiny model runs ≈30 ms idle on one thread). It
ranks turns better than the tiny model (AUC 0.88 vs 0.84 in the pipeline, 0.85 / 0.87 / 0.98 vs
0.80 / 0.81 / 0.97 offline at 0 / 0.2 / 0.6 s into a pause) — worth it when CPU is not the constraint or when
you want a stricter threshold; otherwise use the tiny model, which we recommend for production.
**Serving cost** (int8 ONNX, measured 2026-09-08): 64 ms per prediction on one CPU thread standalone, 87 / 104 ms p50 / p90 inside a LiveKit agent process that also runs Silero and six sessions (≈ 1.3 % of a core per call at the ≈ 0.15 predictions per second a call needs); ≈ 950 predictions/s on a 164-core node as a dedicated process. The agent does this work itself; latency does not depend on the audio length sent (fixed 8 s window).
## Results
### Compared with other open detectors (eot-bench, private telephony)
Same [eot-bench](https://github.com/livekit/eot-bench) harness for every model (100 ms causal grid over every pause
≥ 0.1 s, `threshold × action_delay × timeout` policy sweep, scalar metrics scored 0.2 s into each pause), random
private telephony test turns, both language tags pooled. Third-party models run through eot-bench's own adapters with
the language gate widened to Malay; the text detector on transcripts from our Whisper STT (segment timestamps
interpolated to words, ~30 % of these short turns have no transcript); ultraVAD without the assistant context it was
designed for (this set has none); **LiveKit's cloud Turn Detector v1** streamed the 300-turn set once with the data
owner's approval (292 of 300 turns scored, 8 failed the gateway handshake; LiveKit Cloud caps this project at ~5
streaming turns per minute, so the run was paced).
**The 300 benchmark turns (300 eot / 191 hold spans) — every detector, LiveKit cloud v1 included:**
| model | cutoff @ 300 ms | cutoff @ 600 ms | latency @ 5 % cutoff | latency @ 10 % cutoff | AUC |
|---|---:|---:|---:|---:|---:|
| **Scicom Semantic VAD** (enterprise model, GPU-served, private) | **38.6 %** | **20.0 %** | 1 444 ms | **1 034 ms** | **0.87** |
| Semantic-VAD whisper-small v6 | 48.6 % | 22.1 % | **1 381 ms** | 1 099 ms | 0.86 |
| Semantic-VAD whisper-base v6 | 45.0 % | 23.6 % | 1 592 ms | 1 260 ms | 0.84 |
| Semantic-VAD whisper-tiny v6 | 57.9 % | 28.6 % | 1 685 ms | 1 332 ms | 0.78 |
| LiveKit turn-detector v1-mini (audio-only, local) | 55.7 % | 28.6 % | 1 633 ms | 1 292 ms | 0.76 |
| **LiveKit Turn Detector v1 (cloud, audio)** | 66.7 % | 27.8 % | 1 568 ms | 1 348 ms | 0.70 |
| ultraVAD (no text context) | 65.7 % | 32.1 % | 1 880 ms | 1 384 ms | 0.65 |
| smart-turn v3.2 | 73.6 % | 31.4 % | 1 863 ms | 1 367 ms | 0.64 |
| smart-turn v2 | 77.1 % | 32.1 % | 2 000 ms | 1 420 ms | 0.64 |
| LiveKit text turn-detector v0.4.1-intl (on STT transcripts) | – | – | 1 952 ms | 1 769 ms | 0.45 |
| VAD baseline (silence timer) | 77.9 % | 32.1 % | 1 900 ms | 1 510 ms | – |
LiveKit's cloud v1 lands between ultraVAD and the silence timer on this Malay-heavy telephony audio (no Malay, no
telephony in its training); its local v1-mini does better. Scicom Semantic VAD is the enterprise member of this family, served from a GPU with dynamic batching, and is not public.
**1 000 random test turns (1 010 eot / 569 hold spans) — the larger sample; the cloud detector was not run here:**
| model | cutoff @ 300 ms | cutoff @ 600 ms | latency @ 5 % cutoff | latency @ 10 % cutoff | AUC |
|---|---:|---:|---:|---:|---:|
| **Scicom Semantic VAD** (enterprise model, GPU-served, private) | **43.8 %** | **24.5 %** | **1 663 ms** | **1 215 ms** | **0.87** |
| **Semantic-VAD whisper-small v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-small)) | 45.2 % | **24.5 %** | 1 839 ms | 1 226 ms | 0.86 |
| **Semantic-VAD whisper-base v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base)) | 47.3 % | 25.4 % | 1 812 ms | 1 280 ms | 0.85 |
| **Semantic-VAD whisper-tiny v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-tiny)) | 52.2 % | 30.8 % | 2 042 ms | 1 503 ms | 0.81 |
| LiveKit turn-detector v1-mini (audio-only, `livekit-local-inference`) | 63.6 % | 35.7 % | 2 156 ms | 1 720 ms | 0.74 |
| ultraVAD (`fixie-ai/ultraVAD`, 0.7 B, no text context) | 71.6 % | 39.6 % | 2 212 ms | 1 784 ms | 0.65 |
| smart-turn v3.2 (`pipecat-ai/smart-turn-v3`) | 73.7 % | 36.6 % | 2 296 ms | 1 860 ms | 0.65 |
| LiveKit text turn-detector v0.4.1-intl (on STT transcripts) | – | – | 2 381 ms | 1 894 ms | 0.45 |
| smart-turn v2 (`pipecat-ai/smart-turn-v2`, 95 M wav2vec2) | 74.1 % | 39.6 % | 2 500 ms | 2 000 ms | 0.62 |
| VAD baseline (silence timer) | 78.1 % | 43.6 % | 2 250 ms | 1 770 ms | – |
### In the pipeline and at fixed cut points
**In a real LiveKit Agents 1.8 pipeline** (Silero VAD → turn detector → endpointing, no STT, 300 recorded
telephony turns, LiveKit defaults: VAD silence 0.55 s, `min_delay` 0.5 s, `max_delay` 3.0 s):
| turn detector | latency p50 / p90 | turns cut off | finished turns on the fast path | AUC (eot vs hold) |
|---|---:|---:|---:|---:|
| VAD only | 0.63 / 0.71 s | 14.3 % | – | – |
| smart-turn-v3, threshold 0.5 | 0.65 / 3.04 s | 10.0 % | 82 % | 0.74 |
| tiny variant, threshold 0.5 | 0.64 / 0.74 s | 10.0 % | 95 % | 0.84 |
| **this model, threshold 0.3** | **0.64 / 0.74 s** | **9.7 %** | **96 %** | **0.88** |
| this model, threshold 0.5 | 0.65 / 2.93 s | 9.3 % | 90 % | 0.88 |
The int8 export's scores sit a little lower than the tiny model's (recall at 0.5 is 0.87 vs 0.92 at the pause
start), so **0.3 is this model's equivalent of the tiny model's 0.5**; at 0.5 it is stricter — one more cut-off
avoided, but 10 % of finished turns wait for `max_delay`. One cut-off turn in 300 separates it from the tiny
model at matched fast-path share, which is within noise.
**Offline, at fixed cut points relative to the start of each pause** (AUC, same 300 turns, every pause):
| cut relative to pause start | −0.4 s | −0.2 s | 0.0 s | +0.2 s | +0.6 s |
|---|---:|---:|---:|---:|---:|
| smart-turn-v3 | 0.60 | 0.62 | 0.63 | 0.65 | 0.69 |
| tiny variant (int8) | 0.72 | 0.78 | 0.80 | 0.81 | 0.97 |
| **this model (int8)** | **0.77** | **0.82** | **0.85** | **0.87** | **0.98** |
Score smoothness along a pause matches the tiny model (local std 0.044 over 200 ms, threshold flips 1.7 %
per 20 ms step; smart-turn-v3 0.124 / 9.8 %).
**Under LiveKit's [eot-bench](https://github.com/livekit/eot-bench) harness** (100 ms causal grid over
every pause ≥ 0.1 s, `threshold × action_delay × timeout` policy sweep; same adapter for all audio models,
scored 0.2 s into each pause for the scalar metrics):
| set | model | cutoff @ 300 ms budget | cutoff @ 600 ms | latency @ 5 % cutoff | latency @ 10 % cutoff | AUC |
|---|---|---:|---:|---:|---:|---:|
| telephony test, 1 000 turns, English (510 eot / 260 hold spans) | **this model** | **47.3 %** | **25.0 %** | **1 722 ms** | **1 261 ms** | **0.84** |
| | tiny variant | 50.8 % | 30.0 % | 2 039 ms | 1 529 ms | 0.80 |
| | smart-turn-v3 | 69.6 % | 35.4 % | 2 269 ms | 1 756 ms | 0.66 |
| | VAD baseline | 77.3 % | 41.9 % | 2 020 ms | 1 610 ms | – |
| telephony test, 1 000 turns, Malay (485 / 169) | **this model** | **49.1 %** | **25.4 %** | **1 903 ms** | 1 494 ms | **0.86** |
| | tiny variant | 55.0 % | 32.5 % | 2 019 ms | **1 423 ms** | 0.81 |
| | smart-turn-v3 | 78.1 % | 39.1 % | 2 635 ms | 2 116 ms | 0.63 |
| | VAD baseline | 79.3 % | 46.2 % | 2 540 ms | 2 060 ms | – |
| telephony, the 300 benchmark turns, English (188 / 105) | **this model** | **45.7 %** | **21.9 %** | **1 530 ms** | **1 151 ms** | **0.83** |
| | tiny variant | 58.1 % | 27.6 % | 1 636 ms | 1 198 ms | 0.77 |
| | smart-turn-v3 | 74.3 % | 30.5 % | 1 649 ms | 1 164 ms | 0.64 |
| | VAD baseline | 77.1 % | 31.4 % | 1 800 ms | 1 510 ms | – |
| telephony, the 300 benchmark turns, Malay (112 / 35) | **this model** | **48.6 %** | **25.7 %** | **1 761 ms** | **1 238 ms** | **0.85** |
| | tiny variant | 57.1 % | 31.4 % | 1 843 ms | 1 482 ms | 0.78 |
| | smart-turn-v3 | 68.6 % | 34.3 % | 2 357 ms | 1 603 ms | 0.63 |
| | VAD baseline | 80.0 % | 34.3 % | 2 410 ms | 1 830 ms | – |
Where the tiny model only ties the VAD timer on latency at a 5 % cutoff budget, this one is ahead of it on
every operating point of every subset, and ahead of the tiny model everywhere except latency at 10 % on the
Malay 1 000-turn set. The harness asks within the first 100–300 ms of every pause, before an audio model has
silence evidence; the extra encoder capacity buys the most exactly there (AUC 0.85 vs 0.80 at the pause
start). In the LiveKit pipeline, which asks after the VAD's 0.4–0.55 s of silence, the two are one cut-off
turn apart.
## Files
| file | what |
|---|---|
| `onnx/model.int8.onnx` | MatMul-only dynamic int8, 24 MB |
| `onnx/model.fp32.onnx` | fp32 export, 81 MB; max abs Δp vs PyTorch 1e-6 |
| `onnx/export_report.json` | sizes, parity vs PyTorch, latency at export time |
| `encoder/` | fine-tuned Whisper-base encoder, HF format (`config.json`, `model.safetensors`, bf16) |
| `eot_head.pt` | `{"state_dict": LayerNorm→Linear(512,256)→GELU→Linear(256,1), "pooling": "last5"}` |
| `eot_window.json` / `preprocessor_config.json` | the input contract: 8 s window, 80 mel bins, 16 kHz, no mel normalisation, mean of the last 5 encoder frames |
| `training_summary.json` | best step, validation AUC history |
Input: `input_features` `[batch, 80, 800]` float32 — Whisper log-mel of the **last 8 s of audio, left-padded
with zeros when shorter**, `do_normalize=False`. Output: `probability` `[batch, 1]`, already through the
sigmoid.
## Usage
Identical to the tiny model — substitute the repo id. In short (ONNX, no torch):
```python
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor
REPO, SR, WINDOW = "Scicom-intl/semantic-vad-eot-whisper-base", 16000, 8 * 16000
opts = ort.SessionOptions(); opts.intra_op_num_threads = 1
sess = ort.InferenceSession(hf_hub_download(REPO, "onnx/model.int8.onnx"), opts, providers=["CPUExecutionProvider"])
fe = WhisperFeatureExtractor(feature_size=80, sampling_rate=SR, chunk_length=8)
def p_end_of_turn(pcm: np.ndarray) -> float:
"""pcm: float32 in [-1, 1] at 16 kHz, the caller's audio up to *now* (any length)."""
pcm = np.asarray(pcm, dtype=np.float32)
if pcm.size and np.abs(pcm).max() > 1.5: # int16-scale samples -> unit float
pcm = pcm / 32768.0
pcm = pcm[-WINDOW:] if len(pcm) >= WINDOW else np.pad(pcm, (WINDOW - len(pcm), 0))
feats = fe([pcm], sampling_rate=SR, return_tensors="np", padding="max_length", max_length=WINDOW,
truncation=True, do_normalize=False)["input_features"].astype(np.float32)
return float(sess.run(None, {"input_features": feats})[0].reshape(-1)[0])
```
For LiveKit Agents use it as the backend of [STT-API](https://github.com/Scicom-AI-Enterprise-Organization/STT-API)'s
`SemanticVAD` through a three-line `predict(pcm) -> p(eot)` backend around the ONNX snippet, exactly as on the tiny
model's card.
Use `p ≥ 0.3` as "the turn is over" for the operating point in the table above. The PyTorch loading
snippet (a `WhisperEncoder` subclass that accepts the 8 s window + the 3-layer head) is on the tiny model's
card and works unchanged with this repo id (`d_model` 512).
## License
Apache-2.0 (the Whisper encoder it fine-tunes is Apache-2.0).
|