README.md

---

license: apache-2.0
base_model: MoritzLaurer/deberta-v3-large-zeroshot-v2.0
pipeline_tag: text-classification
tags: [yes-no, guardrail, zero-shot, nli, deberta-v3, calibration]
language: [en]
---


# Maya v0.2: a dedicated yes/no gate

Maya reads a text and a statement (or a yes/no question) and returns P(yes). Version 0.2 has 435M parameters and is fine-tuned from [MoritzLaurer/deberta-v3-large-zeroshot-v2.0](https://huggingface.co/MoritzLaurer/deberta-v3-large-zeroshot-v2.0) on rule-labeled texts with varied wording, minimal pairs and word-overlap traps, plus BoolQ and MNLI pairs to keep its general skill.

- Code, data, tests and every result: https://github.com/vishalmysore/maya
- Live demo in the browser: https://vishalmysore.github.io/maya/
- Browser (int8 ONNX) build: [VishalMysore/mayaWasm](https://huggingface.co/VishalMysore/mayaWasm)
- The previous model (v0.1, ModernBERT-base) is kept under the `v0.1` tag of this repo.

```python

# pip install torch transformers sentencepiece ; then from the GitHub repo:

from maya.gate import Maya

maya = Maya.load("VishalMysore/maya")

maya.ask("Agent plan: delete the `sessions` table on the production database. A verified backup was taken ten minutes ago and the on-call engineer has reviewed the plan.",

         ["The action is destructive and cannot be undone", "This action can be undone if needed"])

# -> no (0.15), yes (0.90)

```

Without the repo: tokenize the pair `(text, statement)`, run `AutoModelForSequenceClassification` (load with `dtype=torch.float32` on CPU), and take `sigmoid((logits[:, 0] - logits[:, 1]) / 1.65)`; label 0 is "yes". The temperature is in `maya_config.json`.

## Results (hand-labeled test sets, never used for training or model selection)

| Model | v3: judgment questions (192 answers) | v2: 8 unseen domains (256) | v1: familiar domains (120) |
|---|---|---|---|
| laya-typed-decisions (421M) | 62.5% | 80.1% | 68.3% |
| deberta-v3-large-zeroshot-v2.0 (Maya's starting point) | 72.9% | 89.8% | 75.8% |
| Maya v0.1 (ModernBERT-base, 150M) | 71.9% | 81.2% | 78.3% |
| **Maya v0.2** | **87.5%** | **95.3%** | **87.5%** |

- v3 was written and committed before the v0.2 training data existed; it includes word-overlap traps (Maya v0.2: 90%) and minimal pairs (71% of pairs fully right; starting point 33%).
- Contradictions between a statement and its negation: 12% on v3 and 8% on v2 (starting point: 73% and 30%).
- Calibration error (ECE) with the shipped temperature: 0.064 on v3, 0.025 on v2.
- The checkpoint was chosen on a separate 64-answer dev set (87.5%).

## Limits

- It still makes mistakes: about one in eight answers on judgment questions is wrong. Example: for a production `DELETE` that "no human has reviewed", it correctly says the action is not safe to run without a human, but answers "no" to "A human should approve this action before it runs". Do not use it as the only safety check.
- Code changes are a known weak spot: it says an unreviewed pull request that drops a database column can be merged. Maya v0.1 got that right.
- It only outputs P(yes): every answer is yes or no (or "not sure" if you set abstention thresholds). It cannot refuse a question that is not yes/no.
- A "not sure" band with a guaranteed error rate has to be fitted on a few hundred labeled examples of your own inputs (`maya.conformal.Abstainer`).
- English only; 435M parameters, about 0.6 s per question on a laptop CPU (4 threads).

**License note:** the base model's card says its versions without "-c" in the name were trained on data that includes non-commercially licensed datasets. Maya v0.2 inherits that; check the base model's card before commercial use (a commercially friendly alternative would be to repeat the recipe on `deberta-v3-large-zeroshot-v2.0-c`).

Unofficial experiment, not affiliated with the authors of the base model or of Laya. Apache-2.0; see NOTICE.md.