Model card: comparison with other open detectors under eot-bench + charts

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/cutoff_at_budget.png filter=lfs diff=lfs merge=lfs -text
+assets/heatmap.png filter=lfs diff=lfs merge=lfs -text
+assets/latency_at_budget.png filter=lfs diff=lfs merge=lfs -text
+assets/pareto.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -92,6 +92,34 @@ silence evidence; the extra encoder capacity buys the most exactly there (AUC 0.
 start). In the LiveKit pipeline, which asks after the VAD's 0.4–0.55 s of silence, the two are one cut-off
 turn apart.
 
+## Compared with other open detectors (eot-bench, 1 000 telephony turns)
+
+Same harness, same 1 000 random `Scicom-intl/semantic-vad-eot-emgs` test turns (both language tags pooled),
+every open audio-native end-of-turn detector we could run locally; cloud services were not run because the
+call audio may not leave our infrastructure. Third-party models run through eot-bench's own adapters with the
+language gate widened to Malay; the text detector on transcripts from our Whisper STT (segment timestamps
+interpolated to words, ~30 % of these short turns have no transcript); ultraVAD without the assistant context
+it was designed for (this set has none).
+
+| model | cutoff @ 300 ms | cutoff @ 600 ms | latency @ 5 % cutoff | latency @ 10 % cutoff | AUC |
+|---|---:|---:|---:|---:|---:|
+| **Semantic-VAD whisper-base v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-base)) | **47.3 %** | **25.4 %** | **1 812 ms** | **1 280 ms** | **0.85** |
+| **Semantic-VAD whisper-tiny v6** ([repo](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-tiny)) | 52.2 % | 30.8 % | 2 042 ms | 1 503 ms | 0.81 |
+| LiveKit turn-detector v1-mini (audio-only, `livekit-local-inference`) | 63.6 % | 35.7 % | 2 156 ms | 1 720 ms | 0.74 |
+| ultraVAD (`fixie-ai/ultraVAD`, 0.7 B, no text context) | 71.6 % | 39.6 % | 2 212 ms | 1 784 ms | 0.65 |
+| smart-turn v3.2 (`pipecat-ai/smart-turn-v3`) | 73.7 % | 36.6 % | 2 296 ms | 1 860 ms | 0.65 |
+| LiveKit text turn-detector v0.4.1-intl (on STT transcripts) | – | – | 2 381 ms | 1 894 ms | 0.45 |
+| smart-turn v2 (`pipecat-ai/smart-turn-v2`, 95 M wav2vec2) | 74.1 % | 39.6 % | 2 500 ms | 2 000 ms | 0.62 |
+| VAD baseline (silence timer) | 78.1 % | 43.6 % | 2 250 ms | 1 770 ms | – |
+
+![Pareto frontier: false cutoffs vs latency](assets/pareto.png)
+
+![Best false-cutoff rate at a 300 / 600 ms latency budget](assets/cutoff_at_budget.png)
+
+![Best mean latency at a 5 / 10 % false-cutoff budget](assets/latency_at_budget.png)
+
+![Operating points across sets and language tags](assets/heatmap.png)
+
 ## Files
 
 | file | what |
@@ -141,23 +169,6 @@ Use `p ≥ 0.3` as "the turn is over" for the operating point in the table above
 snippet (a `WhisperEncoder` subclass that accepts the 8 s window + the 3-layer head) is on the tiny model's
 card and works unchanged with this repo id (`d_model` 512).
 
-## Training
-
-Same as the tiny model: `Scicom-intl/semantic-vad-eot-emgs` (private call-centre telephony, Malay/English,
-customer + agent channels), all train + validation files, early stopping (patience 3) on a fixed 4 000-cut
-sample of the test split — the pipeline benchmark's 300 turns come from that split too, so the numbers are
-in-distribution. Six cut offsets per pause drawn uniformly in [−0.4, +1.2] s (never a fixed grid — it gets
-memorised), 8 s left-padded window, mean of the last 5 encoder frames → `EoTHead`. `openai/whisper-base`
-encoder (6 layers, d 512) fully fine-tuned in bf16, batch 128, AdamW lr 5e-5 constant after warm-up.
-Early-stopped at step 32 000 (validation AUC 0.877 on uniform cuts; tiny: 0.859 at step 8 000). ONNX via
-`torch.onnx.export` at the fixed 800-frame input, MatMul-only dynamic int8.
-
-## Limitations
-
-As for the tiny model: telephony Malay/English only (matches, does not beat, the VAD baseline on Malay read
-speech); expects the raw phone channel (noise cancellation in front of it hurt); feed unit-scale float audio;
-query it after a short VAD silence, not on every frame; a 300-turn sample stands behind the pipeline numbers.
-
 ## License
 
 Apache-2.0 (the Whisper encoder it fine-tunes is Apache-2.0). The training data is not released.