Maya v0.2 (DeBERTa-v3-large fine-tune)

README.md

@@ -1,40 +1,50 @@
 ---
 license: apache-2.0
-base_model: MoritzLaurer/ModernBERT-base-zeroshot-v2.0
+base_model: MoritzLaurer/deberta-v3-large-zeroshot-v2.0
 pipeline_tag: text-classification
-tags: [yes-no, guardrail, zero-shot, nli, modernbert, calibration]
+tags: [yes-no, guardrail, zero-shot, nli, deberta-v3, calibration]
 language: [en]
 ---
 
-# Maya: a dedicated yes/no gate
+# Maya v0.2: a dedicated yes/no gate
 
-Maya reads a text and a statement (or a yes/no question) and returns P(yes). 150M parameters, fine-tuned from [MoritzLaurer/ModernBERT-base-zeroshot-v2.0](https://huggingface.co/MoritzLaurer/ModernBERT-base-zeroshot-v2.0) on rule-labeled synthetic data in which every fact appears in plain and negated wordings with opposite labels (plus a consistency loss that, in an ablation, added nothing measurable).
+Maya reads a text and a statement (or a yes/no question) and returns P(yes). Version 0.2 has 435M parameters and is fine-tuned from [MoritzLaurer/deberta-v3-large-zeroshot-v2.0](https://huggingface.co/MoritzLaurer/deberta-v3-large-zeroshot-v2.0) on rule-labeled texts with varied wording, minimal pairs and word-overlap traps, plus BoolQ and MNLI pairs to keep its general skill.
 
-Code, data, evaluation and full results: https://github.com/vishalmysore/maya. Browser (int8 ONNX) build: [VishalMysore/mayaWasm](https://huggingface.co/VishalMysore/mayaWasm).
+- Code, data, tests and every result: https://github.com/vishalmysore/maya
+- Live demo in the browser: https://vishalmysore.github.io/maya/
+- Browser (int8 ONNX) build: [VishalMysore/mayaWasm](https://huggingface.co/VishalMysore/mayaWasm)
+- The previous model (v0.1, ModernBERT-base) is kept under the `v0.1` tag of this repo.
 
 ```python
-# pip install torch transformers ; then from the GitHub repo:
+# pip install torch transformers sentencepiece ; then from the GitHub repo:
 from maya.gate import Maya
 maya = Maya.load("VishalMysore/maya")
-maya.ask("Hi team, how do I export the dashboard to Excel? No rush.", ["Is the customer angry?"])
+maya.ask("Agent plan: delete the `sessions` table on the production database. A verified backup was taken ten minutes ago and the on-call engineer has reviewed the plan.",
+         ["The action is destructive and cannot be undone", "This action can be undone if needed"])
+# -> no (0.15), yes (0.90)
 ```
 
-Without the repo: tokenize the pair `(text, statement)`, run `AutoModelForSequenceClassification` (load with `dtype=torch.float32` on CPU), and take `sigmoid((logits[:, 0] - logits[:, 1]) / 2.8)`; label 0 is "yes". The temperature 2.8 is in `maya_config.json`.
+Without the repo: tokenize the pair `(text, statement)`, run `AutoModelForSequenceClassification` (load with `dtype=torch.float32` on CPU), and take `sigmoid((logits[:, 0] - logits[:, 1]) / 1.65)`; label 0 is "yes". The temperature is in `maya_config.json`.
 
-## Results (hand-labeled, never used for training)
+## Results (hand-labeled test sets, never used for training or model selection)
 
-| | v1: 120 answers, mostly in-domain | v2: 256 answers, 8 unseen domains |
-|---|---|---|
-| laya-typed-decisions (421M) | 68.3% acc, AUROC 0.792 | 80.1% acc, AUROC 0.889 |
-| ModernBERT-base-zeroshot-v2.0 (starting point) | 76.7%, 0.875 | 78.5%, 0.868 |
-| deberta-v3-large-zeroshot-v2.0 (435M) | 75.8%, 0.890 | **89.8%, 0.957** |
-| **Maya** | **78.3%, 0.905** | 81.2%, 0.893 |
+| Model | v3: judgment questions (192 answers) | v2: 8 unseen domains (256) | v1: familiar domains (120) |
+|---|---|---|---|
+| laya-typed-decisions (421M) | 62.5% | 80.1% | 68.3% |
+| deberta-v3-large-zeroshot-v2.0 (Maya's starting point) | 72.9% | 89.8% | 75.8% |
+| Maya v0.1 (ModernBERT-base, 150M) | 71.9% | 81.2% | 78.3% |
+| **Maya v0.2** | **87.5%** | **95.3%** | **87.5%** |
 
-Maya is the strongest model tested on its own domains (v1). On unseen domains (v2) it is level with Laya and smaller NLI models, and a 3x larger off-the-shelf NLI model is clearly better.
+- v3 was written and committed before the v0.2 training data existed; it includes word-overlap traps (Maya v0.2: 90%) and minimal pairs (71% of pairs fully right; starting point 33%).
+- Contradictions between a statement and its negation: 12% on v3 and 8% on v2 (starting point: 73% and 30%).
+- Calibration error (ECE) with the shipped temperature: 0.064 on v3, 0.025 on v2.
+- The checkpoint was chosen on a separate 64-answer dev set (87.5%).
 
-- Contradictions between a statement and its negation: 25% (v1) and 27% (v2), against 44-92% for the other models.
-- With the shipped temperature, v2 calibration error (ECE) is 0.061.
-- Weak spots: agent guardrails (61% on v1, misled by lexical overlap such as "no human has reviewed" vs "without a human"), and minimal pairs in unseen domains (53% vs Laya's 69%). Do not use it as the only safety check.
-- It only outputs P(yes): every answer is yes or no (or "not sure" if you set abstention thresholds). It cannot refuse non-yes/no questions.
+## Limits
+
+- It still makes mistakes: about one in eight answers on judgment questions is wrong. Example: for a production `DELETE` that "no human has reviewed", it correctly says the action is not safe to run without a human, but answers "no" to "A human should approve this action before it runs". Do not use it as the only safety check.
+- It only outputs P(yes): every answer is yes or no (or "not sure" if you set abstention thresholds). It cannot refuse a question that is not yes/no.
+- A "not sure" band with a guaranteed error rate has to be fitted on a few hundred labeled examples of your own inputs (`maya.conformal.Abstainer`).
+- English only; 435M parameters, about 1 s per question on a laptop CPU.
 
 Unofficial experiment, not affiliated with the authors of the base model or of Laya. Apache-2.0; see NOTICE.md.

config.json

@@ -1,86 +1,46 @@
 {
   "architectures": [
-    "ModernBertForSequenceClassification"
+    "DebertaV2ForSequenceClassification"
   ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 50281,
-  "classifier_activation": "gelu",
-  "classifier_bias": false,
-  "classifier_dropout": 0.0,
-  "classifier_pooling": "mean",
-  "cls_token_id": 50281,
-  "decoder_bias": true,
-  "deterministic_flash_attn": false,
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
   "dtype": "float32",
-  "embedding_dropout": 0.0,
-  "eos_token_id": 50282,
-  "global_attn_every_n_layers": 3,
-  "gradient_checkpointing": false,
-  "hidden_activation": "gelu",
-  "hidden_size": 768,
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
   "id2label": {
     "0": "yes",
     "1": "no"
   },
-  "initializer_cutoff_factor": 2.0,
   "initializer_range": 0.02,
-  "intermediate_size": 1152,
+  "intermediate_size": 4096,
   "label2id": {
     "no": 1,
     "yes": 0
   },
-  "layer_norm_eps": 1e-05,
-  "layer_types": [
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention"
+  "layer_norm_eps": 1e-07,
+  "legacy": true,
+  "max_position_embeddings": 512,
+  "max_relative_positions": -1,
+  "model_type": "deberta-v2",
+  "norm_rel_ebd": "layer_norm",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 0,
+  "pooler_dropout": 0,
+  "pooler_hidden_act": "gelu",
+  "pooler_hidden_size": 1024,
+  "pos_att_type": [
+    "p2c",
+    "c2p"
   ],
-  "local_attention": 128,
-  "max_position_embeddings": 8192,
-  "mlp_bias": false,
-  "mlp_dropout": 0.0,
-  "model_type": "modernbert",
-  "norm_bias": false,
-  "norm_eps": 1e-05,
-  "num_attention_heads": 12,
-  "num_hidden_layers": 22,
-  "pad_token_id": 50283,
-  "position_embedding_type": "absolute",
-  "problem_type": "single_label_classification",
-  "rope_parameters": {
-    "full_attention": {
-      "rope_theta": 160000.0,
-      "rope_type": "default"
-    },
-    "sliding_attention": {
-      "rope_theta": 10000.0,
-      "rope_type": "default"
-    }
-  },
-  "sep_token_id": 50282,
-  "sparse_pred_ignore_index": -100,
-  "sparse_prediction": false,
+  "position_biased_input": false,
+  "position_buckets": 256,
+  "relative_attention": true,
+  "share_att_key": true,
   "tie_word_embeddings": true,
   "transformers_version": "5.17.0",
-  "vocab_size": 50368
+  "type_vocab_size": 0,
+  "vocab_size": 128100
 }

maya_config.json

@@ -1,10 +1,13 @@
 {
  "p_yes": "sigmoid((logit_yes - logit_no) / temperature)",
- "temperature": 2.8,
- "temperature_fitted_on": "data/eval (v1, 120 hand-labeled answers); lowered v2 ECE from 0.152 to 0.061",
+ "temperature": 1.65,
+ "temperature_fitted_on": "data/dev (64 hand-labeled answers, never a test set)",
  "thresholds": {
   "t_yes": 0.5,
   "t_no": 0.5
  },
- "thresholds_note": "Default is strict yes/no. For 'not sure' with a bounded error rate, fit maya.conformal.Abstainer on a labeled sample of your own inputs (a few hundred answers per side are needed to certify ~10% error)."
+ "thresholds_note": "Default is strict yes/no. For 'not sure' with a bounded error rate, fit maya.conformal.Abstainer on a labeled sample of your own inputs (a few hundred answers per side are needed to certify ~10% error).",
+ "pad_token_id": 0,
+ "version": "0.2",
+ "base_model": "MoritzLaurer/deberta-v3-large-zeroshot-v2.0"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cb7d67b8f8303f137bd70055ecdeeef542a3cd9c91d7c1599b15ee04a2bc6ea2
-size 598439784
+oid sha256:c8040f98ed19a5a4ec19d8d0472befd6fe81a565f697993fb50942c8b22d699d
+size 1740304440

tokenizer_config.json

@@ -1,17 +1,21 @@
 {
+  "add_prefix_space": true,
   "backend": "tokenizers",
+  "bos_token": "[CLS]",
   "clean_up_tokenization_spaces": true,
   "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
   "is_local": false,
   "local_files_only": false,
   "mask_token": "[MASK]",
-  "model_input_names": [
-    "input_ids",
-    "attention_mask"
-  ],
   "model_max_length": 512,
   "pad_token": "[PAD]",
   "sep_token": "[SEP]",
-  "tokenizer_class": "TokenizersBackend",
-  "unk_token": "[UNK]"
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_id": 3,
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
 }