--- license: mit language: - en library_name: transformers pipeline_tag: text-classification tags: - medical - triage - emergency-medicine - esi - biomedbert - clinical-nlp - decision-support - not-for-clinical-use base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext model-index: - name: bert-esi-triage-v57 results: - task: type: text-classification name: ESI 1-5 triage classification dataset: type: mimic-iv-ed-holdout name: MIMIC-IV-ED holdout (n=7,917) metrics: - type: recall name: ESI 1 recall (calibrated) value: 0.781 - type: accuracy name: ESI exact (calibrated) value: 0.598 - task: type: text-classification name: ESI 1-5 triage classification dataset: type: mc-med-clean name: MC-MED Stanford clean (n=1,000) metrics: - type: recall name: ESI 1 recall (calibrated) value: 0.770 - type: accuracy name: ESI exact (calibrated) value: 0.586 - task: type: text-classification name: ESI 1-5 triage classification dataset: type: mietic-clean name: MIETIC narrative clean (n=200) metrics: - type: recall name: ESI 1 recall (ensembled) value: 0.967 - type: accuracy name: ESI exact (calibrated) value: 0.840 - task: type: text-classification name: ESI 1-5 triage classification dataset: type: lukina-v3 name: Lukina v3 (n=201) metrics: - type: recall name: ESI 1 recall (calibrated) value: 0.943 --- # BERT-ESI-Triage v57 — BiomedBERT multi-head triage classifier ## TL;DR `bert-esi-triage-v57` is a fine-tuned BiomedBERT classifier for the **Emergency Severity Index (ESI 1-5)** triage decision. It is the first model in this line to meet ESI 1 safety recall **≥75% on every primary eval slice** (MIMIC, MC-MED, MIETIC, Lukina), with a mean ESI 1 recall of **87.5%** across 18 eval slices. - **Model type:** BiomedBERT-base + 11-d engineered-feature fusion (MEWS / qSOFA / shock_index / etc.) → `esi_head` Linear(768+64, 5) - **Auxiliary heads (training only):** symptom, flag, pain, arrival, gestalt, disposition, resource, vitals, etc. — 20+ heads supervise the encoder; only `esi_head` is needed for inference. - **Input:** ED triage text (compact CC, telegraphic, or narrative), ≤ 512 BERT tokens. - **Output:** ESI 1-5 prediction (1 = most acute, 5 = least). - **Calibration recipe (validated production):** demographic normalizer → 6-way sub-dialect detection → per-dialect temperature scaling (Guo et al. 2017) → confidence-aware ESI 1 logit bias → optional engine ensemble `min(BERT, engine)`. - **License:** MIT (model weights). Training data is private (MIMIC-IV-ED + Stanford MC-MED + MIETIC + Lukina v3 + curated synth). ⚠️ **Not a medical device. Not for clinical use.** This is a research artifact distributed for reproducibility and benchmarking. Real deployment requires institutional validation, ED nurse oversight, and the deterministic engine safety floor (see *Recommended inference pipeline* below). ## How to use ```python import torch, torch.nn as nn from transformers import AutoTokenizer, AutoModel from huggingface_hub import hf_hub_download ENCODER = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext" class V57BertESI(nn.Module): def __init__(self): super().__init__() self.encoder = AutoModel.from_pretrained(ENCODER) self.feature_proj = nn.Sequential( nn.Linear(11, 64), nn.GELU(), nn.LayerNorm(64), ) self.esi_head = nn.Linear(768 + 64, 5) def forward(self, input_ids, attention_mask, eng_features): cls = self.encoder(input_ids=input_ids, attention_mask=attention_mask ).last_hidden_state[:, 0] return self.esi_head(torch.cat([cls, self.feature_proj(eng_features)], -1)) tok = AutoTokenizer.from_pretrained(ENCODER) model = V57BertESI() ckpt = torch.load(hf_hub_download("vadimbelsky/bert-esi-triage-v57", "model.pt"), map_location="cpu", weights_only=False) # Many auxiliary heads in the checkpoint are not needed for ESI inference. sd = model.state_dict() model.load_state_dict({k: v for k, v in ckpt.items() if k in sd and sd[k].shape == v.shape}, strict=False) model.eval() text = "67yo M ambulance. CC: Chest pain, Diaphoresis. BP 95/60 HR 120 RR 22 SpO2 94%." enc = tok(text, return_tensors="pt", truncation=True, max_length=512) # Engineered features: 11-d z-scored vector (MEWS/qSOFA/shock_index/etc.). # See the Space app.py compute_engineered_features() for the canonical impl. feats = torch.zeros(1, 11) # zero-vector = "no structured fields parsed" logits = model(enc["input_ids"], enc["attention_mask"], feats) esi = int(logits.argmax(-1)) + 1 # 1-5 print(f"Predicted ESI {esi}") ``` For the **full validated inference stack** (race/SES normalizer, 6-way dialect detection, per-dialect temperature + bias, engine ensemble safety floor), use the Hugging Face Space app.py as the reference implementation: https://huggingface.co/spaces/vadimbelsky/esi-triage-demo ## Recommended inference pipeline Single-model BERT output is **not the production recipe**. Documented v57 deployment recipe (see Space `app.py` for the canonical implementation): 1. **Demographic normalizer** — strip race/SES tokens to prevent bias leakage. Maps `(white|black|hispanic|...)` → `[demographic_redacted]` and `(homeless|low-income|medicaid|...)` → `[social_redacted]`. 2. **6-way sub-dialect detection** — `mimic_compact`, `mcmed_telegraphic`, `narrative_lukina`, `narrative_mietic`, `narrative_general`, `unknown_compact`. See `sub_dialect_detector_v2` (regex-based, no extra deps). 3. **Per-dialect temperature scaling** — Guo et al. 2017. T values: `mimic_compact=1.6369`, `mcmed_telegraphic=1.6577`, others 1.0. Argmax-preserving by construction; smooths Expected Calibration Error. 4. **Confidence-aware per-dialect ESI 1 logit bias** — bias is added to the ESI 1 logit only when the top-2 calibrated logit margin < 1.5. This protects high-confidence predictions from over-correction. Bias values: `{mimic_compact: 0.5, mcmed_telegraphic: 1.0, narrative_lukina: 1.0, narrative_mietic: 0.25, narrative_general: 0.25, unknown_compact: 1.0}`. 5. **Engine ensemble safety floor:** `final_esi = min(bert_cal, engine)`, where `engine` is a deterministic handbook v4 implementation (Step A/B/D triggers). Catches the ~28pp of MIMIC ESI 1 cardiac-arrest cases that BERT alone routes to ESI 5. Lightweight regex implementation bundled with the Space as `engine_ensemble.py`. ## Validated performance (v57 epoch 3, 18-eval suite, n=10,553) ### Primary slices | Eval | n | cal_exact | cal_ESI 1 R | ens_ESI 1 R | |---|---:|---:|---:|---:| | MIMIC-IV-ED holdout | 7,917 | 59.8% | 76.6% | **78.1%** | | MC-MED Stanford clean | 1,000 | 58.6% | 77.0% | 77.0% | | MIETIC narrative | 200 | 84.0% | 90.0% | **96.7%** | | Lukina v3 | 201 | 44.8% | 94.3% | 94.3% | ### Condition-specific slices | Eval | n | cal_ESI 1 R | ens_ESI 1 R | |---|---:|---:|---:| | Sepsis | 93 | 95.0% | **97.5%** | | Stroke | 97 | 95.0% | 95.0% | | Anaphylaxis | 90 | 97.1% | 97.1% | | Cardiac arrest | 103 | 93.3% | 96.7% | | OB emergency | 98 | 95.0% | 95.0% | | Pediatric n=200 v2 | 200 | 95.0% | 95.0% | | Judgment gap v1 | 177 | 88.9% | 94.4% | ### Subgroup slices | Eval | n | cal_exact | cal_ESI 1 R | |---|---:|---:|---:| | Geriatric n=200 | 200 | 61.5% | 78.8% | | Polypharmacy n=200 | 200 | 67.0% | 72.5% | | Frequent flyer n=200 | 135 | 61.5% | 80.0% | | Vital completeness | 231 | 56.3% | 83.3% | | Multi-CC complexity | 304 | 58.2% | 76.2% | | Concept density | 260 | 59.6% | 80.0% | **Mean ESI 1 recall across 18 slices: 87.5%.** ## Head-to-head vs v56 epoch 3 - ESI 1 recall wins: **16 of 18 slices** (1 loss = polypharmacy −1.2pp; 1 neutral) - Mean Δ ESI 1 R: **+14.6pp**, median +11.2pp, max +34.3pp (Lukina) - Mean Δ ESI exact: **−0.3pp** (essentially neutral) Top 5 ESI 1 recall improvements: 1. Lukina v3 +34.3pp (60.0% → 94.3%) — v57 Lukina synth + dialect bucket + bias 2. MC-MED +33.0pp (44.0% → 77.0%) — telegraphic dialect lift 3. MIETIC +30.0pp (60.0% → 90.0%) — narrative lift 4. OB emergency +30.0pp (65.0% → 95.0%) 5. Frequent flyer +20.0pp — counters documented downtriage bias ## Trade-off honesty Per-dialect bias optimizes for the **ESI 1 safety floor** at a real cost to ESI 2 precision on some slices: - Lukina cal_exact −10.9pp vs raw (bias pulls ESI 2s up to ESI 1) - ESI 5-consolidated cal_exact −6.1pp (structural cost of ESI 1 bias) - Pediatric cal_exact −5.0pp (but ESI 1 R +5pp) - Condition slices: heavy ESI 2 → ESI 1 leakage from +1.0 bias This is clinically defensible — missing a critical patient is worse than over-triaging from ESI 2 to ESI 1 — but the model should be deployed as a *safety-first decision support tool*, not as "87% accurate triage AI." Probabilities are smoothed by per-dialect temperature scaling for better ECE; treat them as ordering, not ground truth. ## Training data - **MIMIC-IV-ED** (Beth Israel Deaconess, Boston) — bulk + nurse-assigned ground-truth ESI labels + pyxis medication ground truth. Compact CC dialect, ~5% ESI 1 prevalence. - **MC-MED clean** (Stanford ED) — telegraphic dialect, ICD-inferred resources where pyxis is empty. - **MIETIC** — narrative paraphrase dialect, sentence-level mix. - **Lukina v3** — Russian-physician translation style; eval-only. - **Curated medgemma-grounded synth** — sparse-concept and sparse-dialect coverage; ESI labels inherited from real parent records (LLM never decides the label). Capped at ≤5% of total corpus. Total: ~400K records after BERT 512-token filtering and eval-leakage guard. ER-REASON (discharge summaries) was retained per "don't drop narratives" directive but removed from the dedicated ER-REASON eval slice (most exceed 512 tokens). ## Architecture detail - **Encoder:** `microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext` (12-layer, 768-hidden) - **Engineered-feature fusion (v53.3 Phase 4):** 11-d z-scored features → `feature_proj = Linear(11, 64) → GELU → LayerNorm(64)` → concat with [CLS] (768-d) → `esi_head: Linear(832, 5)`. - **11 engineered features:** shock_index, MEWS proxy, qSOFA-like, critical_vital_count, abnormal_vital_count, arrival_acuity_prior, pain_bucket, age_lifecycle, comorbidity_burden, cc_complexity, concept_density. - **Auxiliary heads (training only):** 20+ heads supervise the encoder on side tasks (symptom labels, flags, vitals reconstruction, etc.). All heads except `esi_head` (+ `feature_proj`) are removed at inference. ## Calibration theory Under asymmetric cost (missing ESI 1 ≫ over-triage by one tier), the Bayes-optimal decision boundary shifts away from p=0.5. Per Guo et al. 2017, the calibrated logit shift is `bias = log(C_FN / C_FP)`. The production +1.0 bias on compact-CC dialects implies a cost ratio of ~2.7×, consistent with ED clinical literature (which generally rates under-triage at 10-50× the cost of one-tier over-triage; the +1.0 value is empirically tuned to the conservative end of that range). Sub-dialect–specific calibration is required because the prior prevalence of ESI 1 varies materially across input formats (~5% in compact CC vs ~17% in narrative). ## Limitations & known gaps - ⚠️ **Research demo only.** Not for clinical use. Not a medical device. - Trained primarily on MIMIC-IV-ED (Boston ED, English only). Geographic and demographic generalization is unverified. - Pediatric data is sparse in training (~0.002% of native records; filled with curated synth). Pediatric vital interpretation is not age-bucketed in v57 (planned in v58). - Single-rater labels — no formal kappa validation across raters. - MC-MED ESI 5 recall weak (25%); Lukina ESI 5 recall weak (25%). - Lukina exact 44.8% — narrative dialect ESI 2-5 boundaries remain fuzzy after v57 calibration trades exact for ESI 1 safety. - Probabilities are NOT clinical truth — temperature scaling smooths ECE but the absolute values should not be interpreted as risk scores. - ER-REASON discharge summaries exceed BERT's 512-token window; the model sees only the truncated first ~512 tokens (where the CC sits). - Synth records are capped at ≤5% of total corpus to bound LLM-induced drift. ## Citation ```bibtex @misc{esi_triage_v57_2026, title = {ESI Triage v57 — BiomedBERT multi-head decision-support classifier with per-dialect calibration and deterministic engine ensemble safety floor}, author = {Belski, Vadim}, year = {2026}, url = {https://huggingface.co/vadimbelsky/bert-esi-triage-v57}, note = {Validated on 18-eval suite: MIMIC-IV-ED holdout, MC-MED Stanford clean, MIETIC narrative clean, Lukina v3, 7 condition-specific slices, 6 subgroup slices. v57 epoch 3. First model in this line meeting ESI 1 recall ≥75% on every primary eval slice.} } ``` ## Related artifacts - **Space (live demo):** https://huggingface.co/spaces/vadimbelsky/esi-triage-demo - **Engine ensemble:** `engine_ensemble.py` in the Space repo — self-contained handbook v4 implementation; bundled with the Space. - **Predecessor:** v49 (still referenced in some downstream pipelines; retired in favor of v57 for new deployments). - **Successor:** v58 (training in progress as of 2026-06-01; targets six corpus fixes for tightened ESI 2 / ESI 5 boundaries).