---
language: en
license: mit
library_name: transformers
base_model: microsoft/deberta-v3-large
pipeline_tag: text-classification
inference: false
tags:
  - deberta-v3
  - schema-conditioned
  - candidate-scoring
  - zero-shot-classification
  - structured-output
---

# Schema-conditioned candidate scorer (DeBERTa-v3-large)

One DeBERTa-v3-large encoder with a **single scalar head** scores `(state, question + candidate)` pairs.
Deterministic code groups the scalar logits per question and decodes them into three answer primitives:

| primitive | input schema | answer |
|---|---|---|
| `choice` | `criteria: {option_id: description}` | argmax option id + probabilities |
| `noul` | optional `criteria: {"true": ..., "false": ...}` | p(proposition is true) |
| `score` | `criteria: [level 0 description, level 1, ...]` | expected level index + per-level probabilities |

The question text, criteria and option ids are **read at inference time**, never baked into the weights,
so the same checkpoint answers new questions over new label sets without retraining.

Try it in the Space: **[mobarmg/jev-schema-scorer](https://huggingface.co/spaces/mobarmg/jev-schema-scorer)**.

## How it works

For every candidate of a question the model sees a sentence pair:

```
sequence_a = the state (free text, or a JSON object serialised)
sequence_b = {"candidate": {"id": "<option id>", "description": "<option description>"},
              "type": "choice", "instructions": "...", "criteria": {...}}
```

The candidate sits right after `[SEP]`, so the only tokens that differ between a question's candidates
are where the encoder attends most easily. Each pair yields one logit; a softmax over the question's
candidates gives the answer distribution. Training minimises cross-entropy between that grouped softmax
and a target distribution (one-hot for `choice` / `score`, `[1-p, p]` for `noul`).

## Usage

The repo ships `schema_scorer.py` with the request compiler, decoder and a small adapter.

```python
from huggingface_hub import hf_hub_download
import importlib.util

repo = "mobarmg/jev-schema-scorer-deberta-v3-large"
spec = importlib.util.spec_from_file_location("schema_scorer", hf_hub_download(repo, "schema_scorer.py"))
schema_scorer = importlib.util.module_from_spec(spec); spec.loader.exec_module(schema_scorer)

scorer = schema_scorer.LocalSystemOne(repo)
scorer.system_one(
    "Nine days of silence on a signed quote is a joke. Our launch event is on the 28th and we still "
    "do not have the licence keys your sales team promised. Somebody pick up a phone.",
    {
        "department": {"type": "choice", "instructions": "Which team should handle this?",
                       "criteria": {"billing": "Payment or subscription issues",
                                    "technical": "Bugs or integration problems",
                                    "sales": "Pricing or plan questions"}},
        "frustration": {"type": "score", "instructions": "How frustrated the customer appears",
                        "criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language"]},
        "is_urgent": {"type": "noul", "instructions": "The message conveys urgency"},
    },
)
# {'answers': {'department': {'type': 'choice', 'choice': 'sales', 'probabilities': {...}},
#              'frustration': {'type': 'score', 'score': 2.0, 'probabilities': [...], 'legend': {...}},
#              'is_urgent': {'type': 'noul', 'noul': 0.99}}}
```

Without the helper, it is a plain `DebertaV2ForSequenceClassification` with `num_labels=1`: tokenize
`(state, serialised question + candidate)` pairs, take `logits[:, 0]`, and softmax over each question's
candidates.

Limits: state + schema + candidate must fit in 512 tokens (trained at 384); every question needs at least two candidates.

## Training data

Fine-tuned from `microsoft/deberta-v3-large` (supervised, grouped-softmax cross-entropy, max 384 tokens)
on the union of two synthetic English datasets, 17,170 training questions in total:

- **v1** (7,650 train / 1,350 eval questions, 3,000 texts, 30 domains): support triage, moderation,
  product reviews, email routing, resume screening, banking, insurance claims, telehealth messages,
  IT helpdesk, dating safety, and more. One `choice`, one `score` and one `noul` question per domain.
- **v2** (9,520 train / 1,680 eval questions, 2,400 states, 24 domains): harder task shapes. JSON-object
  states (orders, transactions, API request logs, candidate profiles), multi-turn dialogue transcripts
  (support chats, sales calls, tutoring), code snippets and log excerpts, `choice` questions with 5-7
  options including an "other" bucket, 4-5 level `score` questions, `noul` labels defined by explicit
  rules over the state, sarcasm and negation, summary faithfulness, and rubric grading of student answers.
  Each domain has 4-6 questions.

To make the model read the schema instead of memorising label ids, each training example draws a
random instruction wording and criteria wording, shuffles the `choice` options, and replaces the option
ids with opaque ids (`opt_a`, `k2`, `bravo`, ...) half of the time. Eval splits hold out 15% of records
per domain.

## Evaluation

Measured with `bench_dataset.py` on the held-out eval splits.

| eval split | metric | this checkpoint | previous checkpoint (v1 only) |
|---|---|---|---|
| v2 (1,680 questions) | `choice` accuracy | **0.841** | 0.687 |
| v1 (1,350 questions) | `choice` / `noul` / `score` | unchanged from the previous checkpoint | 0.889 acc / 0.940 acc, 0.052 Brier / 0.183 MAE |

Chance on v2 `choice` is 0.214; on v1 it is 0.255. The earlier revision of this repo (v1-only training)
remains available in the commit history.

Known weak spots (near chance): `order_record.next_action`, `summary_faithfulness.error_type`, and
`student_answer` grading.

## Intended use and caveats

- Intended for experiments with schema-conditioned classification: routing, triage, moderation-style
  labelling, ordinal scoring, and yes/no propositions over short English texts.
- Trained on synthetic data. Expect degraded accuracy far from the training domains, on long inputs, on
  non-English text, and on criteria that require world knowledge or reasoning across the text.
- Probabilities are often very peaked (the grouped softmax is trained on one-hot targets); treat them as
  rankings rather than calibrated confidences.
- Not a safety classifier. Do not use its `moderation`, `health`, or `security` outputs to make
  consequential decisions without human review.