Upload PyTorch encoder + head and ONNX (fp32, int8) exports

README.md

@@ -0,0 +1,129 @@
+---
+license: apache-2.0
+base_model: openai/whisper-base
+datasets:
+- Scicom-intl/semantic-vad-eot-emgs
+language:
+- ms
+- en
+pipeline_tag: audio-classification
+library_name: transformers
+tags:
+- end-of-turn-detection
+- turn-detection
+- semantic-vad
+- endpointing
+- voice-agent
+- livekit
+- onnx
+- whisper
+- telephony
+---
+
+# Semantic VAD — Whisper-base end-of-turn detector (audio only)
+
+The whisper-base sibling of
+[`Scicom-intl/semantic-vad-eot-whisper-tiny`](https://huggingface.co/Scicom-intl/semantic-vad-eot-whisper-tiny):
+same recipe, same input contract, same data, a 2.5× larger encoder. Given the last 8 seconds of a caller's
+16 kHz audio it returns `p(end of turn)` — finished speaking vs paused mid-sentence — with no transcript.
+
+**20 M parameters · int8 ONNX 24 MB · roughly twice the compute of the tiny model** (int8 69 ms vs 34 ms per
+prediction measured back to back on the same busy CPU; the tiny model runs ≈30 ms idle on one thread). It
+ranks turns better than the tiny model (AUC 0.88 vs 0.84 in the pipeline, 0.85 / 0.87 / 0.98 vs
+0.80 / 0.81 / 0.97 offline at 0 / 0.2 / 0.6 s into a pause) — worth it when CPU is not the constraint or when
+you want a stricter threshold; otherwise use the tiny model, which we recommend for production.
+
+## Results
+
+**In a real LiveKit Agents 1.8 pipeline** (Silero VAD → turn detector → endpointing, no STT, 300 recorded
+telephony turns, LiveKit defaults: VAD silence 0.55 s, `min_delay` 0.5 s, `max_delay` 3.0 s):
+
+| turn detector | latency p50 / p90 | turns cut off | finished turns on the fast path | AUC (eot vs hold) |
+|---|---:|---:|---:|---:|
+| VAD only | 0.63 / 0.71 s | 14.3 % | – | – |
+| smart-turn-v3, threshold 0.5 | 0.65 / 3.04 s | 10.0 % | 82 % | 0.74 |
+| tiny variant, threshold 0.5 | 0.64 / 0.74 s | 10.0 % | 95 % | 0.84 |
+| **this model, threshold 0.3** | **0.64 / 0.74 s** | **9.7 %** | **96 %** | **0.88** |
+| this model, threshold 0.5 | 0.65 / 2.93 s | 9.3 % | 90 % | 0.88 |
+
+The int8 export's scores sit a little lower than the tiny model's (recall at 0.5 is 0.87 vs 0.92 at the pause
+start), so **0.3 is this model's equivalent of the tiny model's 0.5**; at 0.5 it is stricter — one more cut-off
+avoided, but 10 % of finished turns wait for `max_delay`. One cut-off turn in 300 separates it from the tiny
+model at matched fast-path share, which is within noise.
+
+**Offline, at fixed cut points relative to the start of each pause** (AUC, same 300 turns, every pause):
+
+| cut relative to pause start | −0.4 s | −0.2 s | 0.0 s | +0.2 s | +0.6 s |
+|---|---:|---:|---:|---:|---:|
+| smart-turn-v3 | 0.60 | 0.62 | 0.63 | 0.65 | 0.69 |
+| tiny variant (int8) | 0.72 | 0.78 | 0.80 | 0.81 | 0.97 |
+| **this model (int8)** | **0.77** | **0.82** | **0.85** | **0.87** | **0.98** |
+
+Score smoothness along a pause matches the tiny model (local std 0.044 over 200 ms, threshold flips 1.7 %
+per 20 ms step; smart-turn-v3 0.124 / 9.8 %).
+
+## Files
+
+| file | what |
+|---|---|
+| `onnx/model.int8.onnx` | MatMul-only dynamic int8, 24 MB |
+| `onnx/model.fp32.onnx` | fp32 export, 81 MB; max abs Δp vs PyTorch 1e-6 |
+| `onnx/export_report.json` | sizes, parity vs PyTorch, latency at export time |
+| `encoder/` | fine-tuned Whisper-base encoder, HF format (`config.json`, `model.safetensors`, bf16) |
+| `eot_head.pt` | `{"state_dict": LayerNorm→Linear(512,256)→GELU→Linear(256,1), "pooling": "last5"}` |
+| `eot_window.json` / `preprocessor_config.json` | the input contract: 8 s window, 80 mel bins, 16 kHz, no mel normalisation, mean of the last 5 encoder frames |
+| `training_summary.json` | best step, validation AUC history |
+
+Input: `input_features` `[batch, 80, 800]` float32 — Whisper log-mel of the **last 8 s of audio, left-padded
+with zeros when shorter**, `do_normalize=False`. Output: `probability` `[batch, 1]`, already through the
+sigmoid.
+
+## Usage
+
+Identical to the tiny model — substitute the repo id. In short (ONNX, no torch):
+
+```python
+import numpy as np, onnxruntime as ort
+from huggingface_hub import hf_hub_download
+from transformers import WhisperFeatureExtractor
+
+REPO, SR, WINDOW = "Scicom-intl/semantic-vad-eot-whisper-base", 16000, 8 * 16000
+opts = ort.SessionOptions(); opts.intra_op_num_threads = 1
+sess = ort.InferenceSession(hf_hub_download(REPO, "onnx/model.int8.onnx"), opts, providers=["CPUExecutionProvider"])
+fe = WhisperFeatureExtractor(feature_size=80, sampling_rate=SR, chunk_length=8)
+
+def p_end_of_turn(pcm: np.ndarray) -> float:
+    """pcm: float32 in [-1, 1] at 16 kHz, the caller's audio up to *now* (any length)."""
+    pcm = np.asarray(pcm, dtype=np.float32)
+    if pcm.size and np.abs(pcm).max() > 1.5:   # int16-scale samples -> unit float
+        pcm = pcm / 32768.0
+    pcm = pcm[-WINDOW:] if len(pcm) >= WINDOW else np.pad(pcm, (WINDOW - len(pcm), 0))
+    feats = fe([pcm], sampling_rate=SR, return_tensors="np", padding="max_length", max_length=WINDOW,
+               truncation=True, do_normalize=False)["input_features"].astype(np.float32)
+    return float(sess.run(None, {"input_features": feats})[0].reshape(-1)[0])
+```
+
+Use `p ≥ 0.3` as "the turn is over" for the operating point in the table above. The PyTorch loading
+snippet (a `WhisperEncoder` subclass that accepts the 8 s window + the 3-layer head) is on the tiny model's
+card and works unchanged with this repo id (`d_model` 512).
+
+## Training
+
+Same as the tiny model: `Scicom-intl/semantic-vad-eot-emgs` (private call-centre telephony, Malay/English,
+customer + agent channels), all train + validation files, early stopping (patience 3) on a fixed 4 000-cut
+sample of the test split — the pipeline benchmark's 300 turns come from that split too, so the numbers are
+in-distribution. Six cut offsets per pause drawn uniformly in [−0.4, +1.2] s (never a fixed grid — it gets
+memorised), 8 s left-padded window, mean of the last 5 encoder frames → `EoTHead`. `openai/whisper-base`
+encoder (6 layers, d 512) fully fine-tuned in bf16, batch 128, AdamW lr 5e-5 constant after warm-up.
+Early-stopped at step 32 000 (validation AUC 0.877 on uniform cuts; tiny: 0.859 at step 8 000). ONNX via
+`torch.onnx.export` at the fixed 800-frame input, MatMul-only dynamic int8.
+
+## Limitations
+
+As for the tiny model: telephony Malay/English only (matches, does not beat, the VAD baseline on Malay read
+speech); expects the raw phone channel (noise cancellation in front of it hurt); feed unit-scale float audio;
+query it after a short VAD silence, not on every frame; a 300-turn sample stands behind the pipeline numbers.
+
+## License
+
+Apache-2.0 (the Whisper encoder it fine-tunes is Apache-2.0). The training data is not released.

encoder/config.json

@@ -0,0 +1,152 @@
+{
+  "activation_dropout": 0.0,
+  "activation_function": "gelu",
+  "apply_spec_augment": false,
+  "architectures": [
+    "VariableLengthWhisperEncoder"
+  ],
+  "attention_dropout": 0.0,
+  "begin_suppress_tokens": [
+    220,
+    50257
+  ],
+  "bos_token_id": 50257,
+  "classifier_proj_size": 256,
+  "d_model": 512,
+  "decoder_attention_heads": 8,
+  "decoder_ffn_dim": 2048,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 6,
+  "decoder_start_token_id": 50258,
+  "dropout": 0.0,
+  "dtype": "bfloat16",
+  "encoder_attention_heads": 8,
+  "encoder_ffn_dim": 2048,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 6,
+  "eos_token_id": 50257,
+  "forced_decoder_ids": [
+    [
+      1,
+      50259
+    ],
+    [
+      2,
+      50359
+    ],
+    [
+      3,
+      50363
+    ]
+  ],
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "mask_feature_length": 10,
+  "mask_feature_min_masks": 0,
+  "mask_feature_prob": 0.0,
+  "mask_time_length": 10,
+  "mask_time_min_masks": 2,
+  "mask_time_prob": 0.05,
+  "max_source_positions": 1500,
+  "max_target_positions": 448,
+  "median_filter_width": 7,
+  "model_type": "whisper",
+  "num_mel_bins": 80,
+  "pad_token_id": 50257,
+  "scale_embedding": false,
+  "suppress_tokens": [
+    1,
+    2,
+    7,
+    8,
+    9,
+    10,
+    14,
+    25,
+    26,
+    27,
+    28,
+    29,
+    31,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    90,
+    91,
+    92,
+    93,
+    359,
+    503,
+    522,
+    542,
+    873,
+    893,
+    902,
+    918,
+    922,
+    931,
+    1350,
+    1853,
+    1982,
+    2460,
+    2627,
+    3246,
+    3253,
+    3268,
+    3536,
+    3846,
+    3961,
+    4183,
+    4667,
+    6585,
+    6647,
+    7273,
+    9061,
+    9383,
+    10428,
+    10929,
+    11938,
+    12033,
+    12331,
+    12562,
+    13793,
+    14157,
+    14635,
+    15265,
+    15618,
+    16553,
+    16604,
+    18362,
+    18956,
+    20075,
+    21675,
+    22520,
+    26130,
+    26161,
+    26435,
+    28279,
+    29464,
+    31650,
+    32302,
+    32470,
+    36865,
+    42863,
+    47425,
+    49870,
+    50254,
+    50258,
+    50358,
+    50359,
+    50360,
+    50361,
+    50362
+  ],
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.0",
+  "use_cache": true,
+  "use_weighted_layer_sum": false,
+  "vocab_size": 51865
+}

encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bc40ad0cae34e26ebd291598a2c887c20e2240ae4a2563985c3372565843611
+size 41190688

eot_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5dbe75d29b53b8f1a1025df5a6c587be457e1f8bf8f1aa64519b4ab2d959a38c
+size 533601

eot_window.json

@@ -0,0 +1,7 @@
+{
+ "window_seconds": 8.0,
+ "do_normalize": false,
+ "num_mel_bins": 80,
+ "sample_rate": 16000,
+ "pooling": "last5"
+}

onnx/eot_window.json

@@ -0,0 +1,7 @@
+{
+ "window_seconds": 8.0,
+ "do_normalize": false,
+ "num_mel_bins": 80,
+ "sample_rate": 16000,
+ "pooling": "last5"
+}

onnx/export_report.json

@@ -0,0 +1,29 @@
+{
+ "window": {
+  "window_seconds": 8.0,
+  "do_normalize": false,
+  "num_mel_bins": 80,
+  "sample_rate": 16000,
+  "pooling": "last5"
+ },
+ "params": 20160001,
+ "fp32": {
+  "bytes": 80718310,
+  "max_abs_dp_vs_torch": 1.3113021850585938e-06,
+  "latency": {
+   "p50_ms": 163.6397559195757,
+   "p90_ms": 165.72624072432518,
+   "min_ms": 146.37318346649408
+  }
+ },
+ "int8": {
+  "bytes": 23776856,
+  "max_abs_dp_vs_torch": 0.14714109897613525,
+  "mean_abs_dp_vs_torch": 0.047445375472307205,
+  "latency": {
+   "p50_ms": 68.82935343310237,
+   "p90_ms": 69.49121318757534,
+   "min_ms": 57.50598106533289
+  }
+ }
+}

onnx/model.fp32.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae327622a3d579a27aa2c51780c9349261920e12018b1642825ece1185876c7a
+size 80718310

onnx/model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a480ce7730609685f4481d95de374b932c3578aeef86f826acb4225c6dc2369d
+size 23776856

preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "chunk_length": 8,
+  "dither": 0.0,
+  "feature_extractor_type": "WhisperFeatureExtractor",
+  "feature_size": 80,
+  "hop_length": 160,
+  "n_fft": 400,
+  "n_samples": 128000,
+  "nb_max_frames": 800,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "return_attention_mask": false,
+  "sampling_rate": 16000
+}

training_summary.json

@@ -0,0 +1,103 @@
+{
+ "best_step": 32000,
+ "best_eval_auc": 0.8770214562847013,
+ "steps_trained": 38000,
+ "eval_history": [
+  {
+   "step": 2000,
+   "eval_auc": 0.8451234196267309,
+   "eval_loss": 0.05243109539151192
+  },
+  {
+   "step": 4000,
+   "eval_auc": 0.8548836878720985,
+   "eval_loss": 0.05047003552317619
+  },
+  {
+   "step": 6000,
+   "eval_auc": 0.8605236972372734,
+   "eval_loss": 0.050362538546323776
+  },
+  {
+   "step": 8000,
+   "eval_auc": 0.864192420897719,
+   "eval_loss": 0.04977281764149666
+  },
+  {
+   "step": 10000,
+   "eval_auc": 0.867650010034116,
+   "eval_loss": 0.049336887896060944
+  },
+  {
+   "step": 12000,
+   "eval_auc": 0.8667155829821394,
+   "eval_loss": 0.048195865005254745
+  },
+  {
+   "step": 14000,
+   "eval_auc": 0.8678381497090106,
+   "eval_loss": 0.0470057912170887
+  },
+  {
+   "step": 16000,
+   "eval_auc": 0.8705327279416684,
+   "eval_loss": 0.047456044703722
+  },
+  {
+   "step": 18000,
+   "eval_auc": 0.8703926684059134,
+   "eval_loss": 0.0463123619556427
+  },
+  {
+   "step": 20000,
+   "eval_auc": 0.8699223192186769,
+   "eval_loss": 0.04807543754577637
+  },
+  {
+   "step": 22000,
+   "eval_auc": 0.8728593885878654,
+   "eval_loss": 0.046615827828645706
+  },
+  {
+   "step": 24000,
+   "eval_auc": 0.8696860994046425,
+   "eval_loss": 0.0480119027197361
+  },
+  {
+   "step": 26000,
+   "eval_auc": 0.8674555990367249,
+   "eval_loss": 0.04942657798528671
+  },
+  {
+   "step": 28000,
+   "eval_auc": 0.8743436015787008,
+   "eval_loss": 0.04626317694783211
+  },
+  {
+   "step": 30000,
+   "eval_auc": 0.8736662987490802,
+   "eval_loss": 0.046077266335487366
+  },
+  {
+   "step": 32000,
+   "eval_auc": 0.8770214562847013,
+   "eval_loss": 0.04583379626274109
+  },
+  {
+   "step": 34000,
+   "eval_auc": 0.8757483778179143,
+   "eval_loss": 0.04697026312351227
+  },
+  {
+   "step": 36000,
+   "eval_auc": 0.8766556291390728,
+   "eval_loss": 0.04517030343413353
+  },
+  {
+   "step": 38000,
+   "eval_auc": 0.8752487624590273,
+   "eval_loss": 0.04783298820257187
+  }
+ ],
+ "early_stopping_patience": 3
+}