---
license: apache-2.0
base_model: openai/whisper-base
language:
- ms
- en
pipeline_tag: audio-classification
library_name: transformers
tags:
- end-of-turn-detection
- turn-detection
- semantic-vad
- endpointing
- voice-agent
- livekit
- onnx
- whisper
- telephony
---

# Semantic VAD — Whisper-base end-of-turn detector (audio only)

The whisper-base sibling of
[`Scicom-intl/semantic-vad-eot-whisper-tiny`](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-tiny):
same recipe, same input contract, same data, a 2.5× larger encoder. Given the last 8 seconds of a caller's
16 kHz audio it returns `p(end of turn)` — finished speaking vs paused mid-sentence — with no transcript.

**20 M parameters · int8 ONNX 24 MB · roughly twice the compute of the tiny model** (int8 69 ms vs 34 ms per
prediction measured back to back on the same busy CPU; the tiny model runs ≈30 ms idle on one thread). It
ranks turns better than the tiny model (AUC 0.88 vs 0.84 in the pipeline, 0.85 / 0.87 / 0.98 vs
0.80 / 0.81 / 0.97 offline at 0 / 0.2 / 0.6 s into a pause) — worth it when CPU is not the constraint or when
you want a stricter threshold; otherwise use the tiny model, which we recommend for production.

**Serving cost** (int8 ONNX, measured 2026-09-08): 64 ms per prediction on one CPU thread standalone, 87 / 104 ms p50 / p90 inside a LiveKit agent process that also runs Silero and six sessions (≈ 1.3 % of a core per call at the ≈ 0.15 predictions per second a call needs); ≈ 950 predictions/s on a 164-core node as a dedicated process. The agent does this work itself; latency does not depend on the audio length sent (fixed 8 s window).

## Results

### Compared with other open detectors (eot-bench, private telephony)

Same [eot-bench](https://github.com/livekit/eot-bench) harness for every model (100 ms causal grid over every pause
≥ 0.1 s, `threshold × action_delay × timeout` policy sweep, scalar metrics scored 0.2 s into each pause), random
private telephony test turns, both language tags pooled. Third-party models run through eot-bench's own adapters with
the language gate widened to Malay; the text detector on transcripts from our Whisper STT (segment timestamps
interpolated to words, ~30 % of these short turns have no transcript); ultraVAD without the assistant context it was
designed for (this set has none); **LiveKit's cloud Turn Detector v1** streamed the 300-turn set once with the data
owner's approval (292 of 300 turns scored, 8 failed the gateway handshake; LiveKit Cloud caps this project at ~5
streaming turns per minute, so the run was paced).

**The 300 benchmark turns (300 eot / 191 hold spans) — every detector, LiveKit cloud v1 included:**

| model | cutoff @ 300 ms | cutoff @ 600 ms | latency @ 5 % cutoff | latency @ 10 % cutoff | AUC |
|---|---:|---:|---:|---:|---:|
| **Scicom Semantic VAD** (enterprise model, GPU-served, private) | **38.6 %** | **20.0 %** | 1 444 ms | **1 034 ms** | **0.87** |
| Semantic-VAD whisper-small v6 | 48.6 % | 22.1 % | **1 381 ms** | 1 099 ms | 0.86 |
| Semantic-VAD whisper-base v6 | 45.0 % | 23.6 % | 1 592 ms | 1 260 ms | 0.84 |
| Semantic-VAD whisper-tiny v6 | 57.9 % | 28.6 % | 1 685 ms | 1 332 ms | 0.78 |
| LiveKit turn-detector v1-mini (audio-only, local) | 55.7 % | 28.6 % | 1 633 ms | 1 292 ms | 0.76 |
| **LiveKit Turn Detector v1 (cloud, audio)** | 66.7 % | 27.8 % | 1 568 ms | 1 348 ms | 0.70 |
| ultraVAD (no text context) | 65.7 % | 32.1 % | 1 880 ms | 1 384 ms | 0.65 |
| smart-turn v3.2 | 73.6 % | 31.4 % | 1 863 ms | 1 367 ms | 0.64 |
| smart-turn v2 | 77.1 % | 32.1 % | 2 000 ms | 1 420 ms | 0.64 |
| LiveKit text turn-detector v0.4.1-intl (on STT transcripts) | – | – | 1 952 ms | 1 769 ms | 0.45 |
| VAD baseline (silence timer) | 77.9 % | 32.1 % | 1 900 ms | 1 510 ms | – |

LiveKit's cloud v1 lands between ultraVAD and the silence timer on this Malay-heavy telephony audio (no Malay, no
telephony in its training); its local v1-mini does better. Scicom Semantic VAD is the enterprise member of this family, served from a GPU with dynamic batching, and is not public.

![Pareto frontier, 300 turns](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base/resolve/main/assets/300/pareto.png)

![Best false-cutoff rate at a 300 / 600 ms latency budget, 300 turns](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base/resolve/main/assets/300/cutoff_at_budget.png)

![Best mean latency at a 5 / 10 % false-cutoff budget, 300 turns](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base/resolve/main/assets/300/latency_at_budget.png)

**1 000 random test turns (1 010 eot / 569 hold spans) — the larger sample; the cloud detector was not run here:**

| model | cutoff @ 300 ms | cutoff @ 600 ms | latency @ 5 % cutoff | latency @ 10 % cutoff | AUC |
|---|---:|---:|---:|---:|---:|
| **Scicom Semantic VAD** (enterprise model, GPU-served, private) | **43.8 %** | **24.5 %** | **1 663 ms** | **1 215 ms** | **0.87** |
| **Semantic-VAD whisper-small v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-small)) | 45.2 % | **24.5 %** | 1 839 ms | 1 226 ms | 0.86 |
| **Semantic-VAD whisper-base v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base)) | 47.3 % | 25.4 % | 1 812 ms | 1 280 ms | 0.85 |
| **Semantic-VAD whisper-tiny v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-tiny)) | 52.2 % | 30.8 % | 2 042 ms | 1 503 ms | 0.81 |
| LiveKit turn-detector v1-mini (audio-only, `livekit-local-inference`) | 63.6 % | 35.7 % | 2 156 ms | 1 720 ms | 0.74 |
| ultraVAD (`fixie-ai/ultraVAD`, 0.7 B, no text context) | 71.6 % | 39.6 % | 2 212 ms | 1 784 ms | 0.65 |
| smart-turn v3.2 (`pipecat-ai/smart-turn-v3`) | 73.7 % | 36.6 % | 2 296 ms | 1 860 ms | 0.65 |
| LiveKit text turn-detector v0.4.1-intl (on STT transcripts) | – | – | 2 381 ms | 1 894 ms | 0.45 |
| smart-turn v2 (`pipecat-ai/smart-turn-v2`, 95 M wav2vec2) | 74.1 % | 39.6 % | 2 500 ms | 2 000 ms | 0.62 |
| VAD baseline (silence timer) | 78.1 % | 43.6 % | 2 250 ms | 1 770 ms | – |

![Pareto frontier, 1 000 turns](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base/resolve/main/assets/pareto.png)

![Best false-cutoff rate at a 300 / 600 ms latency budget, 1 000 turns](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base/resolve/main/assets/cutoff_at_budget.png)

![Best mean latency at a 5 / 10 % false-cutoff budget, 1 000 turns](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base/resolve/main/assets/latency_at_budget.png)

![Operating points across sets and language tags](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base/resolve/main/assets/heatmap.png)

### In the pipeline and at fixed cut points

**In a real LiveKit Agents 1.8 pipeline** (Silero VAD → turn detector → endpointing, no STT, 300 recorded
telephony turns, LiveKit defaults: VAD silence 0.55 s, `min_delay` 0.5 s, `max_delay` 3.0 s):

| turn detector | latency p50 / p90 | turns cut off | finished turns on the fast path | AUC (eot vs hold) |
|---|---:|---:|---:|---:|
| VAD only | 0.63 / 0.71 s | 14.3 % | – | – |
| smart-turn-v3, threshold 0.5 | 0.65 / 3.04 s | 10.0 % | 82 % | 0.74 |
| tiny variant, threshold 0.5 | 0.64 / 0.74 s | 10.0 % | 95 % | 0.84 |
| **this model, threshold 0.3** | **0.64 / 0.74 s** | **9.7 %** | **96 %** | **0.88** |
| this model, threshold 0.5 | 0.65 / 2.93 s | 9.3 % | 90 % | 0.88 |

The int8 export's scores sit a little lower than the tiny model's (recall at 0.5 is 0.87 vs 0.92 at the pause
start), so **0.3 is this model's equivalent of the tiny model's 0.5**; at 0.5 it is stricter — one more cut-off
avoided, but 10 % of finished turns wait for `max_delay`. One cut-off turn in 300 separates it from the tiny
model at matched fast-path share, which is within noise.

**Offline, at fixed cut points relative to the start of each pause** (AUC, same 300 turns, every pause):

| cut relative to pause start | −0.4 s | −0.2 s | 0.0 s | +0.2 s | +0.6 s |
|---|---:|---:|---:|---:|---:|
| smart-turn-v3 | 0.60 | 0.62 | 0.63 | 0.65 | 0.69 |
| tiny variant (int8) | 0.72 | 0.78 | 0.80 | 0.81 | 0.97 |
| **this model (int8)** | **0.77** | **0.82** | **0.85** | **0.87** | **0.98** |

Score smoothness along a pause matches the tiny model (local std 0.044 over 200 ms, threshold flips 1.7 %
per 20 ms step; smart-turn-v3 0.124 / 9.8 %).

**Under LiveKit's [eot-bench](https://github.com/livekit/eot-bench) harness** (100 ms causal grid over
every pause ≥ 0.1 s, `threshold × action_delay × timeout` policy sweep; same adapter for all audio models,
scored 0.2 s into each pause for the scalar metrics):

| set | model | cutoff @ 300 ms budget | cutoff @ 600 ms | latency @ 5 % cutoff | latency @ 10 % cutoff | AUC |
|---|---|---:|---:|---:|---:|---:|
| telephony test, 1 000 turns, English (510 eot / 260 hold spans) | **this model** | **47.3 %** | **25.0 %** | **1 722 ms** | **1 261 ms** | **0.84** |
| | tiny variant | 50.8 % | 30.0 % | 2 039 ms | 1 529 ms | 0.80 |
| | smart-turn-v3 | 69.6 % | 35.4 % | 2 269 ms | 1 756 ms | 0.66 |
| | VAD baseline | 77.3 % | 41.9 % | 2 020 ms | 1 610 ms | – |
| telephony test, 1 000 turns, Malay (485 / 169) | **this model** | **49.1 %** | **25.4 %** | **1 903 ms** | 1 494 ms | **0.86** |
| | tiny variant | 55.0 % | 32.5 % | 2 019 ms | **1 423 ms** | 0.81 |
| | smart-turn-v3 | 78.1 % | 39.1 % | 2 635 ms | 2 116 ms | 0.63 |
| | VAD baseline | 79.3 % | 46.2 % | 2 540 ms | 2 060 ms | – |
| telephony, the 300 benchmark turns, English (188 / 105) | **this model** | **45.7 %** | **21.9 %** | **1 530 ms** | **1 151 ms** | **0.83** |
| | tiny variant | 58.1 % | 27.6 % | 1 636 ms | 1 198 ms | 0.77 |
| | smart-turn-v3 | 74.3 % | 30.5 % | 1 649 ms | 1 164 ms | 0.64 |
| | VAD baseline | 77.1 % | 31.4 % | 1 800 ms | 1 510 ms | – |
| telephony, the 300 benchmark turns, Malay (112 / 35) | **this model** | **48.6 %** | **25.7 %** | **1 761 ms** | **1 238 ms** | **0.85** |
| | tiny variant | 57.1 % | 31.4 % | 1 843 ms | 1 482 ms | 0.78 |
| | smart-turn-v3 | 68.6 % | 34.3 % | 2 357 ms | 1 603 ms | 0.63 |
| | VAD baseline | 80.0 % | 34.3 % | 2 410 ms | 1 830 ms | – |

Where the tiny model only ties the VAD timer on latency at a 5 % cutoff budget, this one is ahead of it on
every operating point of every subset, and ahead of the tiny model everywhere except latency at 10 % on the
Malay 1 000-turn set. The harness asks within the first 100–300 ms of every pause, before an audio model has
silence evidence; the extra encoder capacity buys the most exactly there (AUC 0.85 vs 0.80 at the pause
start). In the LiveKit pipeline, which asks after the VAD's 0.4–0.55 s of silence, the two are one cut-off
turn apart.

## Files

| file | what |
|---|---|
| `onnx/model.int8.onnx` | MatMul-only dynamic int8, 24 MB |
| `onnx/model.fp32.onnx` | fp32 export, 81 MB; max abs Δp vs PyTorch 1e-6 |
| `onnx/export_report.json` | sizes, parity vs PyTorch, latency at export time |
| `encoder/` | fine-tuned Whisper-base encoder, HF format (`config.json`, `model.safetensors`, bf16) |
| `eot_head.pt` | `{"state_dict": LayerNorm→Linear(512,256)→GELU→Linear(256,1), "pooling": "last5"}` |
| `eot_window.json` / `preprocessor_config.json` | the input contract: 8 s window, 80 mel bins, 16 kHz, no mel normalisation, mean of the last 5 encoder frames |
| `training_summary.json` | best step, validation AUC history |

Input: `input_features` `[batch, 80, 800]` float32 — Whisper log-mel of the **last 8 s of audio, left-padded
with zeros when shorter**, `do_normalize=False`. Output: `probability` `[batch, 1]`, already through the
sigmoid.

## Usage

Identical to the tiny model — substitute the repo id. In short (ONNX, no torch):

```python
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor

REPO, SR, WINDOW = "Scicom-intl/semantic-vad-eot-whisper-base", 16000, 8 * 16000
opts = ort.SessionOptions(); opts.intra_op_num_threads = 1
sess = ort.InferenceSession(hf_hub_download(REPO, "onnx/model.int8.onnx"), opts, providers=["CPUExecutionProvider"])
fe = WhisperFeatureExtractor(feature_size=80, sampling_rate=SR, chunk_length=8)

def p_end_of_turn(pcm: np.ndarray) -> float:
    """pcm: float32 in [-1, 1] at 16 kHz, the caller's audio up to *now* (any length)."""
    pcm = np.asarray(pcm, dtype=np.float32)
    if pcm.size and np.abs(pcm).max() > 1.5:   # int16-scale samples -> unit float
        pcm = pcm / 32768.0
    pcm = pcm[-WINDOW:] if len(pcm) >= WINDOW else np.pad(pcm, (WINDOW - len(pcm), 0))
    feats = fe([pcm], sampling_rate=SR, return_tensors="np", padding="max_length", max_length=WINDOW,
               truncation=True, do_normalize=False)["input_features"].astype(np.float32)
    return float(sess.run(None, {"input_features": feats})[0].reshape(-1)[0])
```

For LiveKit Agents use it as the backend of [STT-API](https://github.com/Scicom-AI-Enterprise-Organization/STT-API)'s
`SemanticVAD` through a three-line `predict(pcm) -> p(eot)` backend around the ONNX snippet, exactly as on the tiny
model's card.

Use `p ≥ 0.3` as "the turn is over" for the operating point in the table above. The PyTorch loading
snippet (a `WhisperEncoder` subclass that accepts the 8 s window + the 3-layer head) is on the tiny model's
card and works unchanged with this repo id (`d_model` 512).

## License

Apache-2.0 (the Whisper encoder it fine-tunes is Apache-2.0).