Publish v2: larger synthetic dataset, checkpoints and evaluation

README.md

@@ -4,16 +4,14 @@ language: en
 library_name: transformers.js
 pipeline_tag: text-classification
 base_model: sentence-transformers/all-MiniLM-L6-v2
-tags: [nodd, onnx, text-classification, calibrated]
+tags: [nodd, onnx, text-classification, synthetic-data]
 ---
 
-# sentiment (v1)
+# sentiment (v2)
 
 What feeling does the writer express about the thing they're talking about (a product, place, service or experience)?
 
-A tiny calibrated classifier made with [nodd](https://github.com/michaljach/nodd): a fine-tuned
-`sentence-transformers/all-MiniLM-L6-v2` exported to ONNX (int8, 24.4 MB) that runs in the browser
-with one forward pass, and knows when it is unsure so hard cases can go to a bigger model.
+A small classifier trained with [nodd](https://github.com/michaljach/nodd), exported as ONNX fp32 and int8 for browser and Node.js inference. This version uses a larger offline synthetic training dataset.
 
 | Label | Meaning |
 |---|---|
@@ -21,43 +19,56 @@ with one forward pass, and knows when it is unsure so hard cases can go to a big
 | `neutral` | Facts, questions, plans or mixed feelings with no clear lean. |
 | `negative` | Unhappy, disappointed, angry or warns others away. Includes sarcastic praise. |
 
-## Numbers
+## Evaluation
 
-| | |
-|---|---|
-| Test macro-F1 (int8 ONNX) | 0.882 |
-| Label agreement int8 vs PyTorch | 96.1% |
-| Calibration (temperature) | T = 0.539, val ECE 0.184 → 0.055 |
-| Confidence threshold | 0.858: 97.8% precision on 90.2% of inputs (val) |
-| Data (train / val / test) | 238 / 51 / 51 |
-| Seed | 42 |
+| Metric | Value |
+|---|---:|
+| Test macro F1, PyTorch | 0.8808 |
+| Test macro F1, int8 ONNX | 0.8802 |
+| int8 vs PyTorch label agreement | 96.08% |
+| int8 download | 24.31 MB |
+| Train / validation / test | 3238 / 51 / 51 |
+| Temperature | 1.549442 |
+| Confidence threshold | 0.946760 |
+| Validation precision at threshold | 100.00% |
+| Test accuracy on covered inputs, PyTorch | 96.15% |
+| Test coverage, PyTorch | 50.98% |
+| Training seed | 42 |
+
+See [baseline comparison](comparison.md), [evaluation report](report.md), and [browser parity results](browser_parity.json).
+
+## Data and limitations
 
-Below the threshold, `isConfident()` is false: escalate to a bigger model.
+Added 1,000 examples per label using deterministic offline templates, with generation seed 20260928. Labels were assigned by construction, not by an independent teacher. Template variants are correlated and do not represent independent scenarios. New rows are training-only; the original validation and test records were preserved. Exact duplicates and examples with embedding cosine similarity above 0.90 to either holdout were rejected.
+
+All evaluation inputs are synthetic; real-world accuracy is unmeasured. More data did not consistently improve held-out quality: consult the comparison before replacing an earlier version. The 97% precision target selected on validation was not met on this version's covered PyTorch test subset. Quantization can change individual probabilities substantially, so label parity does not establish confidence equivalence. Prompt-injection detection, where applicable, is not a standalone security boundary.
 
 ## Use
 
-Download the folder and serve it from your own origin (`@nodd/browser` does not load from the Hub):
+Download this version and serve it from your own origin:
 
 ```sh
-hf download nodd-repo/sentiment --local-dir public/models/sentiment/v1
-npm install @nodd/browser   # or @nodd/node on a server
+hf download nodd-repo/sentiment --revision v2 --local-dir public/models/sentiment/v2
+npm install @nodd/browser
 ```
 
 ```ts
-import { nodd } from "@nodd/browser";   // or "@nodd/node": same API, reads the folder from disk
-
-const m = await nodd.load("/models/sentiment/v1");
-const d = await m.decide("...");   // { label, probabilities, confidence, ... }
-if (!m.isConfident(d)) { /* below the calibrated threshold: escalate */ }
-m.dispose();   // stops the worker
+import { nodd } from "@nodd/browser";
+const model = await nodd.load("/models/sentiment/v2");
+const decision = await model.decide("Your input text");
+if (!model.isConfident(decision)) { /* escalate for review */ }
+model.dispose();
 ```
 
-Needs `@nodd/browser` / `@nodd/node` ≥ 0.2.0.
+For Node.js, use `@nodd/node` and the downloaded directory path. Requires nodd JavaScript packages >= 0.2.0.
+
+## Artifacts
 
-## Files
+- `onnx/model_quantized.onnx`, `onnx/model.onnx`: int8 and fp32 browser models.
+- `nodd.json`, tokenizer files and `config.json`: browser runtime configuration.
+- `encoder/`: original Transformers checkpoint, loadable with `AutoModelForSequenceClassification.from_pretrained(local_path, subfolder="encoder")`.
+- `labeled.jsonl`: exact training, validation and test records with split and provenance fields.
+- `synthetic_manifest.json`: generation metadata and baseline fingerprint.
+- `model_card.json`, reports and parity files: training, calibration and evaluation evidence.
 
-- `onnx/model_quantized.onnx`: int8 model used by default; `onnx/model.onnx`: fp32 (`dtype: "fp32"`)
-- `tokenizer.json`, `tokenizer_config.json`, `config.json`
-- `nodd.json`: labels, temperature, threshold, tokenizer and ONNX settings read by the runtime
-- `model_card.json`: full training, calibration and export metadata
-- `parity.jsonl`: test-split inputs with reference probabilities, for checking a runtime
+Earlier releases remain available through their version tags. The generation script and cross-task results are in the [source repository](https://github.com/michaljach/nodd/blob/main/scripts/train_synthetic.py).

browser_parity.json

@@ -0,0 +1,22 @@
+{
+  "model": "/models/sentiment/v2",
+  "n": 51,
+  "results": [
+    {
+      "name": "transformers.js \u00b7 wasm \u00b7 q8",
+      "device": "wasm",
+      "agreement": 1,
+      "maxProbDiff": 0.009276406422991879,
+      "pass": true
+    },
+    {
+      "name": "transformers.js \u00b7 webgpu \u00b7 q8",
+      "device": "webgpu",
+      "agreement": 1,
+      "maxProbDiff": 0.06435286888915914,
+      "pass": true
+    }
+  ],
+  "pass": true,
+  "webgpu": true
+}

comparison.md

@@ -0,0 +1,24 @@
+# Model comparison — sentiment
+
+Test split identical across versions (fingerprint a9ff6039d984, n=51).
+
+| | sentiment@v1 | sentiment@v2 |
+|---|---|---|
+| tier | encoder | encoder |
+| base | `sentence-transformers/all-MiniLM-L6-v2` | `sentence-transformers/all-MiniLM-L6-v2` |
+| **macro F1** | **0.882** | **0.881** |
+| accuracy | 88.2% | 88.2% |
+| coverage @ threshold | 82.4% (95.2% accurate) | 51.0% (96.2% accurate) |
+| escalation rate | 17.6% | 49.0% |
+| ECE after calibration | 0.066 | 0.093 |
+| download | 23.7 MB | 24.31 MB |
+| latency p95, python | 3.99 ms (cpu) | 3.57 ms (cpu) |
+| latency p95, browser (best) | — | — |
+| train time | 19 s | 120 s |
+| F1 · positive | 0.875 | 0.919 |
+| F1 · neutral | 0.914 | 0.875 |
+| F1 · negative | 0.857 | 0.848 |
+
+Prediction agreement on the test split: sentiment@v1 vs sentiment@v2: 88.2%
+
+Targets: macro F1 ≥ 0.9, download ≤ 30.0 MB.

encoder/config.json

@@ -0,0 +1,40 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "id2label": {
+    "0": "positive",
+    "1": "neutral",
+    "2": "negative"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "is_decoder": false,
+  "label2id": {
+    "negative": 2,
+    "neutral": 1,
+    "positive": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:030c2ac940cde0809b5b4826d06d50e390a70da495f859657b7efc3ed40802c3
+size 90869492

encoder/nodd.json

@@ -0,0 +1 @@
+{"labels": ["positive", "neutral", "negative"], "max_tokens": 256}

encoder/tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

model_card.json

@@ -1,14 +1,14 @@
 {
   "artifact_version": 1,
-  "model": "sentiment@v1",
+  "model": "sentiment@v2",
   "tier": "encoder",
   "labels": [
     "positive",
     "neutral",
     "negative"
   ],
-  "temperature": 0.5394950342333552,
-  "threshold": 0.8577380690700984,
+  "temperature": 1.5494422923834061,
+  "threshold": 0.9467601639708129,
   "spec": {
     "task": "sentiment",
     "description": "What feeling does the writer express about the thing they're talking about (a product, place, service or experience)?\n",
@@ -25,7 +25,7 @@
       }
     },
     "model": {
-      "tier": "auto",
+      "tier": "encoder",
       "base": null,
       "quantization": "q8"
     },
@@ -54,34 +54,35 @@
     "seed": 42
   },
   "task": "sentiment",
-  "version": "v1",
+  "version": "v2",
   "base": "sentence-transformers/all-MiniLM-L6-v2",
   "prompt_template": null,
   "calibration": {
     "method": "temperature",
-    "temperature": 0.5394950342333552,
-    "val_ece_before": 0.18350414857910052,
-    "val_ece_after": 0.05482613536715122,
-    "threshold": 0.8577380690700984,
-    "val_coverage": 0.9019607843137255,
-    "val_precision_at_threshold": 0.9782608695652174,
+    "temperature": 1.5494422923834061,
+    "val_ece_before": 0.10366493771366977,
+    "val_ece_after": 0.10166117276176741,
+    "threshold": 0.9467601639708129,
+    "val_coverage": 0.5294117647058824,
+    "val_precision_at_threshold": 1.0,
     "target_precision_reached": true
   },
   "download_mb": 23.7,
   "training": {
+    "selection": "best fit",
     "chosen": {
       "tier": "encoder",
       "base": "sentence-transformers/all-MiniLM-L6-v2",
       "est_download_mb": 23.7,
-      "val_macro_f1": 0.9209,
-      "best_epoch": 6,
+      "val_macro_f1": 0.9029,
+      "best_epoch": 2,
       "val_macro_f1_by_epoch": [
-        0.6337,
-        0.865,
-        0.9029,
+        0.8835,
         0.9029,
-        0.901,
-        0.9209
+        0.8674,
+        0.8454,
+        0.8407,
+        0.8407
       ],
       "epochs": 6,
       "lr": 0.0001,
@@ -90,36 +91,19 @@
       "device": "cpu"
     },
     "candidates": [
-      {
-        "tier": "static",
-        "base": "minishlab/potion-base-8M",
-        "est_download_mb": 8.6,
-        "C": 0.1,
-        "val_macro_f1": 0.8216,
-        "val_macro_f1_by_C": {
-          "0.003": 0.7592,
-          "0.01": 0.7806,
-          "0.03": 0.7826,
-          "0.1": 0.8216,
-          "0.3": 0.8013,
-          "1.0": 0.8013,
-          "3.0": 0.7397
-        },
-        "embedding_dtype": "int8"
-      },
       {
         "tier": "encoder",
         "base": "sentence-transformers/paraphrase-MiniLM-L3-v2",
         "est_download_mb": 18.4,
-        "val_macro_f1": 0.8628,
-        "best_epoch": 3,
+        "val_macro_f1": 0.8432,
+        "best_epoch": 6,
         "val_macro_f1_by_epoch": [
-          0.6763,
-          0.7637,
-          0.8628,
-          0.8628,
-          0.8424,
-          0.8424
+          0.763,
+          0.8402,
+          0.8229,
+          0.8205,
+          0.8226,
+          0.8432
         ],
         "epochs": 6,
         "lr": 0.0001,
@@ -131,15 +115,15 @@
         "tier": "encoder",
         "base": "sentence-transformers/all-MiniLM-L6-v2",
         "est_download_mb": 23.7,
-        "val_macro_f1": 0.9209,
-        "best_epoch": 6,
+        "val_macro_f1": 0.9029,
+        "best_epoch": 2,
         "val_macro_f1_by_epoch": [
-          0.6337,
-          0.865,
+          0.8835,
           0.9029,
-          0.9029,
-          0.901,
-          0.9209
+          0.8674,
+          0.8454,
+          0.8407,
+          0.8407
         ],
         "epochs": 6,
         "lr": 0.0001,
@@ -148,7 +132,7 @@
         "device": "cpu"
       }
     ],
-    "train_seconds": 19.45,
+    "train_seconds": 120.39,
     "sample_weight": "teacher confidence"
   },
   "teacher": {
@@ -158,12 +142,36 @@
     "min_confidence": 0.0
   },
   "data": {
-    "train": 238,
+    "train": 3238,
     "val": 51,
     "test": 51
   },
   "seed": 42,
-  "created": "2026-09-27T18:37:39+00:00",
+  "created": "2026-09-28T09:38:29+00:00",
+  "synthetic_augmentation": {
+    "baseline": "runs/sentiment/v1",
+    "baseline_sha256": "fe5d48bb3f9f751fa8ec1869257a48db3bd7c90d6ee40964646bd927c9f99c2c",
+    "seed": 20260928,
+    "added_per_label": {
+      "positive": 1000,
+      "neutral": 1000,
+      "negative": 1000
+    },
+    "before": {
+      "train": 238,
+      "test": 51,
+      "val": 51
+    },
+    "after": {
+      "train": 3238,
+      "test": 51,
+      "val": 51
+    },
+    "generator": "offline-templates-v1",
+    "holdout_max_cosine": 0.9,
+    "rejected": {},
+    "limitations": "Template variants are correlated. Labels are assigned by construction, not an independent teacher. Original synthetic validation/test are unchanged. Real-world generalization is unmeasured."
+  },
   "export": {
     "format": "nodd",
     "format_version": 2,
@@ -171,19 +179,19 @@
       "n": 51,
       "onnx_fp32": {
         "label_agreement": 1.0,
-        "max_abs_prob_diff": 3.0934404020577233e-06,
-        "macro_f1": 0.8821428571428571,
+        "max_abs_prob_diff": 1.439289778637498e-06,
+        "macro_f1": 0.8808012558012558,
         "f1_drop": 0.0
       },
       "onnx_q8": {
         "label_agreement": 0.9607843137254902,
-        "max_abs_prob_diff": 0.5095771768447779,
-        "macro_f1": 0.8823529411764706,
-        "f1_drop": -0.00021008403361344463
+        "max_abs_prob_diff": 0.41190388463864464,
+        "macro_f1": 0.8801699426699426,
+        "f1_drop": 0.0006313131313131493
       }
     },
     "parity_reference": "onnx_q8",
-    "onnx_download_mb": 24.36,
-    "onnx_fp32_download_mb": 92.53
+    "onnx_download_mb": 24.31,
+    "onnx_fp32_download_mb": 92.47
   }
 }

nodd.json

@@ -1 +1 @@
-{"format":"nodd","format_version":2,"model":"sentiment@v1","labels":["positive","neutral","negative"],"temperature":0.5394950342333552,"threshold":0.8577380690700984,"max_chars":2000,"files":{"config.json":923,"onnx/model.onnx":91811939,"onnx/model_quantized.onnx":23649644,"tokenizer.json":711649,"tokenizer_config.json":594},"tier":"encoder","tokenizer":{"file":"tokenizer.json","add_special_tokens":true,"max_tokens":256,"truncation":true},"onnx":{"file":"onnx/model_quantized.onnx","fp32_file":"onnx/model.onnx","dtype":"q8","inputs":["input_ids","token_type_ids","attention_mask"],"output":"logits"}}
+{"format":"nodd","format_version":2,"model":"sentiment@v2","labels":["positive","neutral","negative"],"temperature":1.5494422923834061,"threshold":0.9467601639708129,"max_chars":2000,"files":{"config.json":923,"onnx/model.onnx":91752775,"onnx/model_quantized.onnx":23598378,"tokenizer.json":711649,"tokenizer_config.json":594},"tier":"encoder","tokenizer":{"file":"tokenizer.json","add_special_tokens":true,"max_tokens":256,"truncation":true},"onnx":{"file":"onnx/model_quantized.onnx","fp32_file":"onnx/model.onnx","dtype":"q8","inputs":["input_ids","token_type_ids","attention_mask"],"output":"logits"}}

onnx/model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0eaabe310c345b9986940112615d9b591e61dabf2340bb01e75a897f46c8573d
-size 91811939
+oid sha256:e15e2af81a60ccc1ebbd4894fbcd581f153dc975a10934cfbb398fea343eab68
+size 91752775

onnx/model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:562a22793fdf2c2a17bad38d03320cefcf776c9b11bbaf1e937c7fc46775e631
-size 23649644
+oid sha256:1fab106ffd3cf4992cb355dbc92af876cd70d008b1b2d12f3ce284a89ac91ccb
+size 23598378

parity.jsonl

@@ -1,51 +1,51 @@
-{"text": "The worst customer service I have ever dealt with.", "label": "negative", "probabilities": {"positive": 0.02545724231717005, "neutral": 0.023743408456705434, "negative": 0.9507993492261244}}
-{"text": "Rude staff and overpriced drinks. Avoid.", "label": "negative", "probabilities": {"positive": 0.012111247552389742, "neutral": 0.03582100490604493, "negative": 0.9520677475415653}}
-{"text": "My package was delivered to the wrong address and nobody cares.", "label": "negative", "probabilities": {"positive": 0.01351755795046229, "neutral": 0.03239043663889282, "negative": 0.9540920054106449}}
-{"text": "They cancelled my order without explanation. Pathetic.", "label": "negative", "probabilities": {"positive": 0.012824672141719194, "neutral": 0.03150854377017358, "negative": 0.9556667840881071}}
-{"text": "Took forever to set up and it still doesn't connect.", "label": "negative", "probabilities": {"positive": 0.014148068655987969, "neutral": 0.03423085567071655, "negative": 0.9516210756732955}}
-{"text": "The hotel photos are clearly fake, the place is run down.", "label": "negative", "probabilities": {"positive": 0.009239102300657389, "neutral": 0.22075207135849054, "negative": 0.7700088263408521}}
-{"text": "Poor quality stitching, came apart in the wash.", "label": "negative", "probabilities": {"positive": 0.021043686499243888, "neutral": 0.01943589078640507, "negative": 0.959520422714351}}
-{"text": "The mattress is lumpy and gave me back pain.", "label": "negative", "probabilities": {"positive": 0.02037007362302223, "neutral": 0.025891840105410648, "negative": 0.9537380862715671}}
-{"text": "Slow, clunky and expensive. No thanks.", "label": "negative", "probabilities": {"positive": 0.018886616555516784, "neutral": 0.022718985183480648, "negative": 0.9583943982610026}}
-{"text": "I'm cancelling my membership, the gym is always dirty.", "label": "negative", "probabilities": {"positive": 0.011216333552360631, "neutral": 0.0663155921080884, "negative": 0.922468074339551}}
-{"text": "Disgusting bathroom, broken lock, rude receptionist.", "label": "negative", "probabilities": {"positive": 0.015603009106652126, "neutral": 0.02765756082472178, "negative": 0.9567394300686259}}
-{"text": "ugh, broken again 😡", "label": "negative", "probabilities": {"positive": 0.04499463580547422, "neutral": 0.0169678586831921, "negative": 0.9380375055113336}}
-{"text": "1 star is too generous", "label": "positive", "probabilities": {"positive": 0.5072130757473364, "neutral": 0.13927667802443502, "negative": 0.3535102462282286}}
-{"text": "Waited on hold for two hours to be told to call back tomorrow.", "label": "neutral", "probabilities": {"positive": 0.005698486194519127, "neutral": 0.9294640774090298, "negative": 0.06483743639645093}}
-{"text": "Pretty design, shame it barely works.", "label": "negative", "probabilities": {"positive": 0.05490471797046441, "neutral": 0.01767932484151116, "negative": 0.9274159571880245}}
-{"text": "Thanks for making the app worse with every update.", "label": "negative", "probabilities": {"positive": 0.040686627466280934, "neutral": 0.015938452520312386, "negative": 0.9433749200134067}}
-{"text": "The store opens at 9am on weekdays.", "label": "neutral", "probabilities": {"positive": 0.00747896027055764, "neutral": 0.9766198186837354, "negative": 0.015901221045707077}}
-{"text": "We stayed for three nights in a double room.", "label": "neutral", "probabilities": {"positive": 0.006639561829318198, "neutral": 0.960934891056446, "negative": 0.032425547114235564}}
-{"text": "Our flight departs at 6:40 in the morning.", "label": "neutral", "probabilities": {"positive": 0.006704764083762528, "neutral": 0.9761994632565988, "negative": 0.017095772659638617}}
-{"text": "The sofa comes in grey, green and beige.", "label": "neutral", "probabilities": {"positive": 0.010118631295988448, "neutral": 0.9723883669677537, "negative": 0.01749300173625812}}
-{"text": "Does anyone know if this works with Alexa?", "label": "neutral", "probabilities": {"positive": 0.01817557897307635, "neutral": 0.9145484931113563, "negative": 0.06727592791556732}}
-{"text": "Is this dishwasher safe?", "label": "neutral", "probabilities": {"positive": 0.011530062482359972, "neutral": 0.8571846354876557, "negative": 0.13128530202998442}}
-{"text": "I'll check back after the next update.", "label": "neutral", "probabilities": {"positive": 0.01224470050889334, "neutral": 0.9459133032158612, "negative": 0.04184199627524555}}
-{"text": "This is my second order from this store.", "label": "neutral", "probabilities": {"positive": 0.013284661841959946, "neutral": 0.9698546154660278, "negative": 0.016860722692012442}}
-{"text": "The instructions are in English and Spanish.", "label": "neutral", "probabilities": {"positive": 0.00835290573554598, "neutral": 0.9740137967456504, "negative": 0.017633297518803517}}
-{"text": "It uses AA batteries.", "label": "neutral", "probabilities": {"positive": 0.007258513798940282, "neutral": 0.9662089742955725, "negative": 0.026532511905487312}}
-{"text": "The app syncs every 15 minutes.", "label": "neutral", "probabilities": {"positive": 0.005217907607760432, "neutral": 0.9665775701550822, "negative": 0.028204522237157442}}
-{"text": "We took the train instead of driving.", "label": "neutral", "probabilities": {"positive": 0.00613157104547193, "neutral": 0.7778720141688376, "negative": 0.21599641478569054}}
-{"text": "Took the bus to the museum, about 20 minutes.", "label": "neutral", "probabilities": {"positive": 0.007285880495427496, "neutral": 0.9753145538462283, "negative": 0.017399565658344256}}
-{"text": "Planning to upgrade next year.", "label": "neutral", "probabilities": {"positive": 0.013986552783786192, "neutral": 0.9682537427383371, "negative": 0.017759704477876764}}
-{"text": "Le colis est arrivé ce matin.", "label": "neutral", "probabilities": {"positive": 0.022766198700194813, "neutral": 0.7379755514398941, "negative": 0.23925824985991098}}
-{"text": "Pros: light, cheap. Cons: flimsy, loud.", "label": "negative", "probabilities": {"positive": 0.14484039325546536, "neutral": 0.02166184417076735, "negative": 0.8334977625737672}}
-{"text": "The pizza was alright.", "label": "neutral", "probabilities": {"positive": 0.006737677210684211, "neutral": 0.8744153898057384, "negative": 0.11884693298357733}}
-{"text": "Thank you so much for the quick refund, really appreciate it.", "label": "positive", "probabilities": {"positive": 0.981554376292129, "neutral": 0.003853136093605839, "negative": 0.014592487614265038}}
-{"text": "Delivery arrived a day early and everything was in perfect condition.", "label": "positive", "probabilities": {"positive": 0.51646531967425, "neutral": 0.42631641797313313, "negative": 0.05721826235261688}}
-{"text": "I'm obsessed with this moisturizer, my skin has never felt better.", "label": "negative", "probabilities": {"positive": 0.20192981251270403, "neutral": 0.016860172316803677, "negative": 0.7812100151704923}}
-{"text": "Delicious, generous portions, and the waiter was lovely.", "label": "positive", "probabilities": {"positive": 0.9868967167831515, "neutral": 0.004025654994472781, "negative": 0.009077628222375757}}
-{"text": "Kudos to the devs, the dark mode looks stunning.", "label": "positive", "probabilities": {"positive": 0.9850386060871908, "neutral": 0.003678597196590094, "negative": 0.011282796716218893}}
-{"text": "This podcast episode was so insightful, learned a ton.", "label": "positive", "probabilities": {"positive": 0.9831528298758287, "neutral": 0.004146593532583886, "negative": 0.012700576591587439}}
-{"text": "Very pleased with the repair, the car drives like new.", "label": "positive", "probabilities": {"positive": 0.9839911646153474, "neutral": 0.003628637884063868, "negative": 0.012380197500588759}}
-{"text": "The museum was fascinating and the guide was brilliant.", "label": "positive", "probabilities": {"positive": 0.9850187675972273, "neutral": 0.007778249652011528, "negative": 0.007202982750761198}}
-{"text": "The team was welcoming and made onboarding painless.", "label": "negative", "probabilities": {"positive": 0.20387843012376153, "neutral": 0.015664344761642673, "negative": 0.7804572251145958}}
-{"text": "This library saved me hours of work, thanks to the maintainers.", "label": "positive", "probabilities": {"positive": 0.9613733528047768, "neutral": 0.006985465944337493, "negative": 0.03164118125088578}}
-{"text": "Perfect fit and the fabric is super soft.", "label": "positive", "probabilities": {"positive": 0.9839112815706914, "neutral": 0.0037017975422324483, "negative": 0.012386920887076111}}
-{"text": "I'm really proud of how this project turned out, thanks everyone.", "label": "positive", "probabilities": {"positive": 0.9849241983927706, "neutral": 0.0034982335481877334, "negative": 0.011577568059041662}}
-{"text": "Great product, great price, great service.", "label": "positive", "probabilities": {"positive": 0.9846353550224716, "neutral": 0.0048618242614952125, "negative": 0.01050282071603324}}
-{"text": "This charger is tiny and charges my laptop super fast. Love it.", "label": "positive", "probabilities": {"positive": 0.9718353101996634, "neutral": 0.0036069900372691, "negative": 0.024557699763067365}}
-{"text": "Quick, friendly and efficient. What more could you want?", "label": "positive", "probabilities": {"positive": 0.9696840790295648, "neutral": 0.016513329456282182, "negative": 0.01380259151415321}}
-{"text": "A bit pricey, but the quality makes up for it. Happy.", "label": "positive", "probabilities": {"positive": 0.9168514136641516, "neutral": 0.003977769351213419, "negative": 0.07917081698463514}}
-{"text": "It's no Michelin star, but it was a lovely meal.", "label": "positive", "probabilities": {"positive": 0.9759013128458217, "neutral": 0.017008820083179922, "negative": 0.007089867070998468}}
-{"text": "Slow start, but the second half of the book is brilliant.", "label": "negative", "probabilities": {"positive": 0.053639907200989094, "neutral": 0.024066071044830557, "negative": 0.9222940217541804}}
+{"text": "The worst customer service I have ever dealt with.", "label": "negative", "probabilities": {"positive": 0.022732729664800333, "neutral": 0.02787839675993739, "negative": 0.9493888735752624}}
+{"text": "Rude staff and overpriced drinks. Avoid.", "label": "negative", "probabilities": {"positive": 0.03657828832126019, "neutral": 0.04027292973439461, "negative": 0.9231487819443451}}
+{"text": "My package was delivered to the wrong address and nobody cares.", "label": "negative", "probabilities": {"positive": 0.019697576217617276, "neutral": 0.0408890623700929, "negative": 0.9394133614122899}}
+{"text": "They cancelled my order without explanation. Pathetic.", "label": "negative", "probabilities": {"positive": 0.02156548427014997, "neutral": 0.024858529404526163, "negative": 0.9535759863253239}}
+{"text": "Took forever to set up and it still doesn't connect.", "label": "negative", "probabilities": {"positive": 0.0231232868458743, "neutral": 0.02112925087843885, "negative": 0.955747462275687}}
+{"text": "The hotel photos are clearly fake, the place is run down.", "label": "negative", "probabilities": {"positive": 0.06266887810683679, "neutral": 0.09870886800099776, "negative": 0.8386222538921654}}
+{"text": "Poor quality stitching, came apart in the wash.", "label": "negative", "probabilities": {"positive": 0.027432687681153554, "neutral": 0.02047087498398513, "negative": 0.9520964373348613}}
+{"text": "The mattress is lumpy and gave me back pain.", "label": "negative", "probabilities": {"positive": 0.025013547330756232, "neutral": 0.018892569177520666, "negative": 0.9560938834917231}}
+{"text": "Slow, clunky and expensive. No thanks.", "label": "negative", "probabilities": {"positive": 0.039961166120812956, "neutral": 0.025463345849973725, "negative": 0.9345754880292133}}
+{"text": "I'm cancelling my membership, the gym is always dirty.", "label": "negative", "probabilities": {"positive": 0.028108783170786623, "neutral": 0.04173747635476573, "negative": 0.9301537404744477}}
+{"text": "Disgusting bathroom, broken lock, rude receptionist.", "label": "negative", "probabilities": {"positive": 0.02487671942594719, "neutral": 0.02060096775213736, "negative": 0.9545223128219155}}
+{"text": "ugh, broken again 😡", "label": "negative", "probabilities": {"positive": 0.025791558875005033, "neutral": 0.023479953282957156, "negative": 0.9507284878420378}}
+{"text": "1 star is too generous", "label": "positive", "probabilities": {"positive": 0.8935329691015025, "neutral": 0.0425933850041057, "negative": 0.06387364589439185}}
+{"text": "Waited on hold for two hours to be told to call back tomorrow.", "label": "neutral", "probabilities": {"positive": 0.02351739979289841, "neutral": 0.8767317545528871, "negative": 0.09975084565421455}}
+{"text": "Pretty design, shame it barely works.", "label": "positive", "probabilities": {"positive": 0.5456890318394592, "neutral": 0.020666810390063177, "negative": 0.4336441577704776}}
+{"text": "Thanks for making the app worse with every update.", "label": "negative", "probabilities": {"positive": 0.15130164538731705, "neutral": 0.016544758098495347, "negative": 0.8321535965141876}}
+{"text": "The store opens at 9am on weekdays.", "label": "neutral", "probabilities": {"positive": 0.023255098870509187, "neutral": 0.9561585612908057, "negative": 0.020586339838685245}}
+{"text": "We stayed for three nights in a double room.", "label": "neutral", "probabilities": {"positive": 0.12045498867806145, "neutral": 0.7975807218958599, "negative": 0.08196428942607849}}
+{"text": "Our flight departs at 6:40 in the morning.", "label": "neutral", "probabilities": {"positive": 0.026883725479022513, "neutral": 0.9478315726096493, "negative": 0.025284701911328204}}
+{"text": "The sofa comes in grey, green and beige.", "label": "neutral", "probabilities": {"positive": 0.20577690297226564, "neutral": 0.7581402894499925, "negative": 0.03608280757774184}}
+{"text": "Does anyone know if this works with Alexa?", "label": "neutral", "probabilities": {"positive": 0.05236810649637475, "neutral": 0.9081180201541645, "negative": 0.03951387334946067}}
+{"text": "Is this dishwasher safe?", "label": "neutral", "probabilities": {"positive": 0.22276124468965597, "neutral": 0.5165383429241467, "negative": 0.2607004123861974}}
+{"text": "I'll check back after the next update.", "label": "neutral", "probabilities": {"positive": 0.02266674922832736, "neutral": 0.952141771711581, "negative": 0.025191479060091802}}
+{"text": "This is my second order from this store.", "label": "neutral", "probabilities": {"positive": 0.12486390448834193, "neutral": 0.8373956457413828, "negative": 0.037740449770275214}}
+{"text": "The instructions are in English and Spanish.", "label": "neutral", "probabilities": {"positive": 0.025459676694021414, "neutral": 0.9555874104992783, "negative": 0.018952912806700337}}
+{"text": "It uses AA batteries.", "label": "neutral", "probabilities": {"positive": 0.05621023043704398, "neutral": 0.8918792170889146, "negative": 0.051910552474041555}}
+{"text": "The app syncs every 15 minutes.", "label": "neutral", "probabilities": {"positive": 0.10292317968318757, "neutral": 0.7765542298305381, "negative": 0.12052259048627435}}
+{"text": "We took the train instead of driving.", "label": "negative", "probabilities": {"positive": 0.06846555012683701, "neutral": 0.299043599956491, "negative": 0.632490849916672}}
+{"text": "Took the bus to the museum, about 20 minutes.", "label": "neutral", "probabilities": {"positive": 0.03897910206734341, "neutral": 0.9262995257091182, "negative": 0.03472137222353839}}
+{"text": "Planning to upgrade next year.", "label": "neutral", "probabilities": {"positive": 0.04240472500425306, "neutral": 0.9260246916680073, "negative": 0.03157058332773968}}
+{"text": "Le colis est arrivé ce matin.", "label": "neutral", "probabilities": {"positive": 0.18884652230946813, "neutral": 0.5216173923775365, "negative": 0.2895360853129954}}
+{"text": "Pros: light, cheap. Cons: flimsy, loud.", "label": "negative", "probabilities": {"positive": 0.10533729176659042, "neutral": 0.027624897726048037, "negative": 0.8670378105073615}}
+{"text": "The pizza was alright.", "label": "neutral", "probabilities": {"positive": 0.2048309436191239, "neutral": 0.735223182741217, "negative": 0.05994587363965906}}
+{"text": "Thank you so much for the quick refund, really appreciate it.", "label": "positive", "probabilities": {"positive": 0.9452156730302687, "neutral": 0.014083655841315134, "negative": 0.04070067112841606}}
+{"text": "Delivery arrived a day early and everything was in perfect condition.", "label": "positive", "probabilities": {"positive": 0.951516194800968, "neutral": 0.023058526978325333, "negative": 0.02542527822070666}}
+{"text": "I'm obsessed with this moisturizer, my skin has never felt better.", "label": "negative", "probabilities": {"positive": 0.036281944019340304, "neutral": 0.029968475396225182, "negative": 0.9337495805844346}}
+{"text": "Delicious, generous portions, and the waiter was lovely.", "label": "positive", "probabilities": {"positive": 0.9584187219004168, "neutral": 0.018279471539756485, "negative": 0.023301806559826808}}
+{"text": "Kudos to the devs, the dark mode looks stunning.", "label": "positive", "probabilities": {"positive": 0.9561556363872546, "neutral": 0.01837029835049029, "negative": 0.02547406526225516}}
+{"text": "This podcast episode was so insightful, learned a ton.", "label": "positive", "probabilities": {"positive": 0.9555686752512573, "neutral": 0.017976855802408927, "negative": 0.026454468946333812}}
+{"text": "Very pleased with the repair, the car drives like new.", "label": "positive", "probabilities": {"positive": 0.9589325189363227, "neutral": 0.017700491604114982, "negative": 0.023366989459562303}}
+{"text": "The museum was fascinating and the guide was brilliant.", "label": "positive", "probabilities": {"positive": 0.9564181745141263, "neutral": 0.019581620802546728, "negative": 0.024000204683327038}}
+{"text": "The team was welcoming and made onboarding painless.", "label": "positive", "probabilities": {"positive": 0.5224835168122385, "neutral": 0.025439524542114286, "negative": 0.45207695864564723}}
+{"text": "This library saved me hours of work, thanks to the maintainers.", "label": "positive", "probabilities": {"positive": 0.9530391540775632, "neutral": 0.018957402994338558, "negative": 0.02800344292809813}}
+{"text": "Perfect fit and the fabric is super soft.", "label": "positive", "probabilities": {"positive": 0.9539594041422013, "neutral": 0.016872804985438965, "negative": 0.029167790872359975}}
+{"text": "I'm really proud of how this project turned out, thanks everyone.", "label": "positive", "probabilities": {"positive": 0.9594479827880417, "neutral": 0.017505981967099234, "negative": 0.02304603524485925}}
+{"text": "Great product, great price, great service.", "label": "positive", "probabilities": {"positive": 0.9513287689167623, "neutral": 0.01865588850544544, "negative": 0.03001534257779225}}
+{"text": "This charger is tiny and charges my laptop super fast. Love it.", "label": "positive", "probabilities": {"positive": 0.9546800773556958, "neutral": 0.01663138541010711, "negative": 0.028688537234196943}}
+{"text": "Quick, friendly and efficient. What more could you want?", "label": "positive", "probabilities": {"positive": 0.9162133507683954, "neutral": 0.030774072483803185, "negative": 0.05301257674780137}}
+{"text": "A bit pricey, but the quality makes up for it. Happy.", "label": "positive", "probabilities": {"positive": 0.9547402220118609, "neutral": 0.01743616702949393, "negative": 0.027823610958645162}}
+{"text": "It's no Michelin star, but it was a lovely meal.", "label": "positive", "probabilities": {"positive": 0.9563229043632224, "neutral": 0.019548317085197983, "negative": 0.024128778551579612}}
+{"text": "Slow start, but the second half of the book is brilliant.", "label": "positive", "probabilities": {"positive": 0.6405768336008174, "neutral": 0.05249405114830608, "negative": 0.3069291152508764}}

report.json

@@ -0,0 +1,181 @@
+{
+  "model": "sentiment@v2",
+  "tier": "encoder",
+  "base": "sentence-transformers/all-MiniLM-L6-v2",
+  "test": {
+    "n": 51,
+    "accuracy": 0.8823529411764706,
+    "macro_f1": 0.8808012558012558,
+    "per_label": {
+      "positive": {
+        "precision": 0.8947368421052632,
+        "recall": 0.9444444444444444,
+        "f1": 0.918918918918919,
+        "support": 18
+      },
+      "neutral": {
+        "precision": 0.9333333333333333,
+        "recall": 0.8235294117647058,
+        "f1": 0.875,
+        "support": 17
+      },
+      "negative": {
+        "precision": 0.8235294117647058,
+        "recall": 0.875,
+        "f1": 0.8484848484848485,
+        "support": 16
+      }
+    },
+    "confusion": [
+      [
+        17,
+        0,
+        1
+      ],
+      [
+        1,
+        14,
+        2
+      ],
+      [
+        1,
+        1,
+        14
+      ]
+    ]
+  },
+  "by_source": {
+    "synthetic": {
+      "n": 51,
+      "accuracy": 0.8823529411764706,
+      "macro_f1": 0.8808012558012558,
+      "per_label": {
+        "positive": {
+          "precision": 0.8947368421052632,
+          "recall": 0.9444444444444444,
+          "f1": 0.918918918918919,
+          "support": 18
+        },
+        "neutral": {
+          "precision": 0.9333333333333333,
+          "recall": 0.8235294117647058,
+          "f1": 0.875,
+          "support": 17
+        },
+        "negative": {
+          "precision": 0.8235294117647058,
+          "recall": 0.875,
+          "f1": 0.8484848484848485,
+          "support": 16
+        }
+      },
+      "confusion": [
+        [
+          17,
+          0,
+          1
+        ],
+        [
+          1,
+          14,
+          2
+        ],
+        [
+          1,
+          1,
+          14
+        ]
+      ]
+    }
+  },
+  "teacher_agreement": 0.8823529411764706,
+  "calibration": {
+    "temperature": 1.5494422923834061,
+    "test_ece_before": 0.13583191042865103,
+    "test_ece_after": 0.09304488274530132,
+    "val_ece_before": 0.10366493771366977,
+    "val_ece_after": 0.10166117276176741,
+    "val_coverage": 0.5294117647058824,
+    "val_precision_at_threshold": 1.0
+  },
+  "escalation": {
+    "threshold": 0.9467601639708129,
+    "target_precision": 0.97,
+    "coverage": 0.5098039215686274,
+    "accuracy_on_covered": 0.9615384615384616,
+    "escalation_rate": 0.4901960784313726
+  },
+  "latency": {
+    "p50_ms": 3.41874998412095,
+    "p95_ms": 3.5703955072676763,
+    "batch_ms_per_input": 0.451612745082992,
+    "backend": "python (cpu); browser numbers: web/ `npm run bench`"
+  },
+  "size_mb": 24.31,
+  "targets": {
+    "min_macro_f1": {
+      "target": 0.9,
+      "actual": 0.8808012558012558,
+      "met": false
+    },
+    "max_download_mb": {
+      "target": 30.0,
+      "actual": 24.31,
+      "met": true
+    }
+  },
+  "worst_errors": [
+    {
+      "text": "I'm obsessed with this moisturizer, my skin has never felt better.",
+      "label": "positive",
+      "predicted": "negative",
+      "confidence": 0.947,
+      "teacher_confidence": 0.95
+    },
+    {
+      "text": "Pros: light, cheap. Cons: flimsy, loud.",
+      "label": "neutral",
+      "predicted": "negative",
+      "confidence": 0.922,
+      "teacher_confidence": 0.75
+    },
+    {
+      "text": "1 star is too generous",
+      "label": "negative",
+      "predicted": "positive",
+      "confidence": 0.914,
+      "teacher_confidence": 0.95
+    },
+    {
+      "text": "Waited on hold for two hours to be told to call back tomorrow.",
+      "label": "negative",
+      "predicted": "neutral",
+      "confidence": 0.908,
+      "teacher_confidence": 0.95
+    },
+    {
+      "text": "This is my second order from this store.",
+      "label": "neutral",
+      "predicted": "positive",
+      "confidence": 0.529,
+      "teacher_confidence": 0.92
+    },
+    {
+      "text": "We took the train instead of driving.",
+      "label": "neutral",
+      "predicted": "negative",
+      "confidence": 0.508,
+      "teacher_confidence": 0.92
+    }
+  ],
+  "data": {
+    "train": 3238,
+    "val": 51,
+    "test": 51
+  },
+  "seed": 42,
+  "recommendations": [
+    "Macro F1 0.881 is below the 0.90 target. Options: more/better data (real inputs beat synthetic), the encoder tier (M4), or escalation-heavy mode (the micro model handles only confident cases).",
+    "All test data is synthetic: expect lower accuracy on real inputs. Add a human-labeled gold set."
+  ]
+}

report.md

@@ -0,0 +1,48 @@
+# sentiment@v2 — encoder tier
+
+Base `sentence-transformers/all-MiniLM-L6-v2` · test n=51 · train/val/test = 3238/51/51 · seed 42
+
+## Summary
+
+| metric | value |
+|---|---|
+| macro F1 | **0.881** (target 0.90 ❌) |
+| accuracy | 88.2% |
+| teacher agreement | 88.2% |
+| coverage @ threshold 0.947 | 51.0% handled alone, 96.2% accurate (target 97%) |
+| escalation rate | 49.0% |
+| ECE (test) | 0.136 → 0.093 after temperature 1.55 |
+| download size | 24.31 MB (budget 30.0 MB ✅) |
+| latency (python (cpu), single input) | p50 3.42 ms · p95 3.57 ms |
+
+## Recommendations
+
+- Macro F1 0.881 is below the 0.90 target. Options: more/better data (real inputs beat synthetic), the encoder tier (M4), or escalation-heavy mode (the micro model handles only confident cases).
+- All test data is synthetic: expect lower accuracy on real inputs. Add a human-labeled gold set.
+
+## Per label
+
+| label | precision | recall | F1 | support |
+|---|---|---|---|---|
+| positive | 0.895 | 0.944 | 0.919 | 18 |
+| neutral | 0.933 | 0.824 | 0.875 | 17 |
+| negative | 0.824 | 0.875 | 0.848 | 16 |
+
+## Confusion matrix (rows = true, cols = predicted)
+
+| | positive | neutral | negative |
+|---|---|---|---|
+| **positive** | 17 | 0 | 1 |
+| **neutral** | 1 | 14 | 2 |
+| **negative** | 1 | 1 | 14 |
+
+## Worst 6 errors (most confident mistakes)
+
+| conf | true → predicted | teacher conf | text |
+|---|---|---|---|
+| 0.95 | positive → negative | 0.95 | I'm obsessed with this moisturizer, my skin has never felt better. |
+| 0.92 | neutral → negative | 0.75 | Pros: light, cheap. Cons: flimsy, loud. |
+| 0.91 | negative → positive | 0.95 | 1 star is too generous |
+| 0.91 | negative → neutral | 0.95 | Waited on hold for two hours to be told to call back tomorrow. |
+| 0.53 | neutral → positive | 0.92 | This is my second order from this store. |
+| 0.51 | neutral → negative | 0.92 | We took the train instead of driving. |

synthetic_manifest.json

@@ -0,0 +1,24 @@
+{
+  "baseline": "runs/sentiment/v1",
+  "baseline_sha256": "fe5d48bb3f9f751fa8ec1869257a48db3bd7c90d6ee40964646bd927c9f99c2c",
+  "seed": 20260928,
+  "added_per_label": {
+    "positive": 1000,
+    "neutral": 1000,
+    "negative": 1000
+  },
+  "before": {
+    "train": 238,
+    "test": 51,
+    "val": 51
+  },
+  "after": {
+    "train": 3238,
+    "test": 51,
+    "val": 51
+  },
+  "generator": "offline-templates-v1",
+  "holdout_max_cosine": 0.9,
+  "rejected": {},
+  "limitations": "Template variants are correlated. Labels are assigned by construction, not an independent teacher. Original synthetic validation/test are unchanged. Real-world generalization is unmeasured."
+}