Upload scorer

README.md

@@ -0,0 +1,157 @@
+---
+language: en
+license: mit
+library_name: transformers
+base_model: microsoft/deberta-v3-large
+pipeline_tag: text-classification
+inference: false
+tags:
+  - deberta-v3
+  - schema-conditioned
+  - candidate-scoring
+  - zero-shot-classification
+  - structured-output
+---
+
+# Schema-conditioned candidate scorer (DeBERTa-v3-large)
+
+One DeBERTa-v3-large encoder with a **single scalar head** scores `(state, question + candidate)` pairs.
+Deterministic code groups the scalar logits per question and decodes them into three answer primitives:
+
+| primitive | input schema | answer |
+|---|---|---|
+| `choice` | `criteria: {option_id: description}` | argmax option id + probabilities |
+| `noul` | optional `criteria: {"true": ..., "false": ...}` | p(proposition is true) |
+| `score` | `criteria: [level 0 description, level 1, ...]` | expected level index + per-level probabilities |
+
+The question text, criteria and option ids are **read at inference time**, never baked into the weights,
+so the same checkpoint answers new questions over new label sets without retraining.
+
+Try it in the Space: **[mobarmg/jev-schema-scorer](https://huggingface.co/spaces/mobarmg/jev-schema-scorer)**.
+
+## How it works
+
+For every candidate of a question the model sees a sentence pair:
+
+```
+sequence_a = the state (free text, or a JSON object serialised)
+sequence_b = {"candidate": {"id": "<option id>", "description": "<option description>"},
+              "type": "choice", "instructions": "...", "criteria": {...}}
+```
+
+The candidate sits right after `[SEP]`, so the only tokens that differ between a question's candidates
+are where the encoder attends most easily. Each pair yields one logit; a softmax over the question's
+candidates gives the answer distribution. Training minimises cross-entropy between that grouped softmax
+and a target distribution (one-hot for `choice` / `score`, `[1-p, p]` for `noul`).
+
+## Usage
+
+The repo ships `schema_scorer.py` with the request compiler, decoder and a small adapter.
+
+```python
+from huggingface_hub import hf_hub_download
+import importlib.util
+
+repo = "mobarmg/jev-schema-scorer-deberta-v3-large"
+spec = importlib.util.spec_from_file_location("schema_scorer", hf_hub_download(repo, "schema_scorer.py"))
+schema_scorer = importlib.util.module_from_spec(spec); spec.loader.exec_module(schema_scorer)
+
+scorer = schema_scorer.LocalSystemOne(repo)
+scorer.system_one(
+    "Nine days of silence on a signed quote is a joke. Our launch event is on the 28th and we still "
+    "do not have the licence keys your sales team promised. Somebody pick up a phone.",
+    {
+        "department": {"type": "choice", "instructions": "Which team should handle this?",
+                       "criteria": {"billing": "Payment or subscription issues",
+                                    "technical": "Bugs or integration problems",
+                                    "sales": "Pricing or plan questions"}},
+        "frustration": {"type": "score", "instructions": "How frustrated the customer appears",
+                        "criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language"]},
+        "is_urgent": {"type": "noul", "instructions": "The message conveys urgency"},
+    },
+)
+# {'answers': {'department': {'type': 'choice', 'choice': 'sales', 'probabilities': {...}},
+#              'frustration': {'type': 'score', 'score': 2.0, 'probabilities': [...], 'legend': {...}},
+#              'is_urgent': {'type': 'noul', 'noul': 0.99}}}
+```
+
+Without the helper, it is a plain `DebertaV2ForSequenceClassification` with `num_labels=1`: tokenize
+`(state, serialised question + candidate)` pairs, take `logits[:, 0]`, and softmax over each question's
+candidates.
+
+Limits: state + schema + candidate must fit in 512 tokens; every question needs at least two candidates.
+
+## Training data
+
+Fine-tuned from `microsoft/deberta-v3-large` on 7,650 questions over 3,000 short English texts spanning
+30 domains (support triage, moderation, product reviews, email routing, resume screening, banking,
+insurance claims, telehealth messages, IT helpdesk, dating safety, and more). Each domain defines one
+`choice`, one `score` and one `noul` question. The texts and labels are synthetic, written to a per-domain
+spec. To make the model read the schema instead of memorising label ids, each training example draws a
+random instruction wording and criteria wording, shuffles the `choice` options, and replaces the option
+ids with opaque ids (`opt_a`, `k2`, `bravo`, ...) half of the time.
+
+Held-out eval split: 1,350 questions (15% of records per domain, stratified over label combinations).
+
+## Evaluation
+
+Held-out eval split (1,350 questions, 45 per domain), measured with `bench_dataset.py`:
+
+| primitive | metric | DeBERTa-v3-large scorer | chance |
+|---|---|---|---|
+| `choice` | accuracy | **0.889** | 0.255 |
+| `noul` | accuracy | **0.940** | 0.500 |
+| `noul` | Brier score (lower is better) | **0.052** | 0.250 |
+| `score` | mean absolute error in levels (lower is better) | **0.183** | 0.700 |
+
+<details>
+<summary>Per-domain results</summary>
+
+| domain | choice acc (chance) | noul acc / Brier | score MAE (uniform) |
+|---|---|---|---|
+| auto_service | 0.733 (0.200) | 0.933 / 0.067 | 0.105 (0.667) |
+| banking | 0.867 (0.200) | 1.000 / 0.002 | 0.283 (0.600) |
+| bug_report | 0.867 (0.250) | 0.933 / 0.067 | 0.097 (0.667) |
+| dating_safety | 0.933 (0.250) | 0.933 / 0.067 | 0.176 (0.533) |
+| ecommerce_order | 1.000 (0.250) | 1.000 / 0.000 | 0.133 (0.600) |
+| education | 1.000 (0.250) | 1.000 / 0.000 | 0.064 (0.600) |
+| email_routing | 0.800 (0.250) | 0.933 / 0.061 | 0.189 (0.733) |
+| fitness_nutrition | 0.867 (0.250) | 1.000 / 0.000 | 0.024 (0.733) |
+| gov_services | 0.800 (0.200) | 1.000 / 0.000 | 0.074 (0.667) |
+| health_symptom | 1.000 (0.200) | 1.000 / 0.000 | 0.213 (1.100) |
+| hr_workplace | 0.800 (0.200) | 1.000 / 0.000 | 0.041 (0.600) |
+| insurance_claim | 1.000 (0.250) | 0.933 / 0.067 | 0.266 (0.733) |
+| it_helpdesk | 0.800 (0.250) | 1.000 / 0.000 | 0.430 (0.533) |
+| job_posting | 0.933 (0.250) | 0.933 / 0.066 | 0.116 (0.667) |
+| legal_clause | 0.933 (0.250) | 0.800 / 0.205 | 0.421 (0.533) |
+| mental_health | 0.667 (0.200) | 1.000 / 0.000 | 0.072 (0.667) |
+| moderation | 1.000 (0.333) | 0.800 / 0.170 | 0.137 (0.667) |
+| news | 1.000 (0.250) | 0.867 / 0.105 | 0.101 (0.667) |
+| pharmacy | 0.933 (0.250) | 0.933 / 0.067 | 0.429 (0.600) |
+| product_review | 0.867 (0.333) | 0.933 / 0.028 | 0.071 (1.133) |
+| real_estate | 0.867 (0.250) | 1.000 / 0.000 | 0.000 (0.533) |
+| restaurant_review | 0.933 (0.250) | 0.933 / 0.067 | 0.175 (1.267) |
+| resume_screening | 0.933 (0.333) | 0.933 / 0.059 | 0.074 (0.667) |
+| scientific_abstract | 1.000 (0.250) | 1.000 / 0.003 | 0.217 (0.733) |
+| security_alert | 0.867 (0.200) | 0.800 / 0.134 | 0.814 (0.900) |
+| smart_home | 0.933 (0.250) | 0.933 / 0.025 | 0.000 (0.733) |
+| social_post | 0.800 (0.333) | 0.867 / 0.114 | 0.310 (0.667) |
+| support_triage | 0.933 (0.333) | 0.867 / 0.134 | 0.009 (0.600) |
+| survey | 0.800 (0.333) | 0.933 / 0.067 | 0.133 (0.600) |
+| travel | 0.800 (0.250) | 1.000 / 0.000 | 0.304 (0.600) |
+
+</details>
+
+Chance is the uniform-guess baseline: `1/k` accuracy for `choice`, 0.5 for `noul`, and the MAE of
+predicting the middle level for `score`.
+
+## Intended use and caveats
+
+- Intended for experiments with schema-conditioned classification: routing, triage, moderation-style
+  labelling, ordinal scoring, and yes/no propositions over short English texts.
+- Trained on synthetic data. Expect degraded accuracy far from the training domains, on long inputs, on
+  non-English text, and on criteria that require world knowledge or reasoning across the text.
+- Probabilities are often very peaked (the grouped softmax is trained on one-hot targets); treat them as
+  rankings rather than calibrated confidences.
+- Not a safety classifier. Do not use its `moderation`, `health`, or `security` outputs to make
+  consequential decisions without human review.

config.json

@@ -0,0 +1,44 @@
+{
+  "architectures": [
+    "DebertaV2ForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_eps": 1e-07,
+  "legacy": true,
+  "max_position_embeddings": 512,
+  "max_relative_positions": -1,
+  "model_type": "deberta-v2",
+  "norm_rel_ebd": "layer_norm",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 0,
+  "pooler_dropout": 0.0,
+  "pooler_hidden_act": "gelu",
+  "pooler_hidden_size": 1024,
+  "pos_att_type": [
+    "p2c",
+    "c2p"
+  ],
+  "position_biased_input": false,
+  "position_buckets": 256,
+  "relative_attention": true,
+  "share_att_key": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "type_vocab_size": 0,
+  "vocab_size": 128100
+}

schema_scorer.py

@@ -0,0 +1,337 @@
+"""
+Schema-conditioned BERT candidate scorer.
+
+One BERT encoder with a single scalar head scores (state, question+candidate)
+pairs. Deterministic code groups the scalar logits per question and decodes
+them into `choice`, `noul`, and `score` answers. The schema (instructions,
+criteria, candidate ids) is read at inference time, never baked into weights.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import random
+from dataclasses import dataclass
+from typing import Any
+
+import torch
+from torch import nn
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+SUPPORTED_PRIMITIVES = ("choice", "noul", "score")
+
+# Serialization layout. Must match between training and inference.
+# SCORER_CANDIDATE_FIRST=0 restores the trailing-candidate layout (ablation only).
+CANDIDATE_FIRST = os.environ.get("SCORER_CANDIDATE_FIRST", "1") == "1"
+
+
+# --------------------------------------------------------------------------- #
+# 1. Request compiler
+# --------------------------------------------------------------------------- #
+def serialize(value: Any) -> str:
+    return json.dumps(value, ensure_ascii=False, sort_keys=False)
+
+
+def compile_question(state: Any, question: dict) -> tuple[list[tuple[str, str]], list[tuple[str, str]]]:
+    """
+    Turn one question into (candidates, text_pairs).
+
+    candidates: [(candidate_id, description), ...] in decode order.
+    text_pairs: [(sequence_a, sequence_b), ...], one per candidate, where
+        sequence_a = the state and
+        sequence_b = the full question schema plus the candidate under review.
+    """
+    kind = question.get("type")
+
+    if kind == "choice":
+        criteria = question.get("criteria")
+        if not isinstance(criteria, dict):
+            raise ValueError("choice questions need a dict of criteria {id: description}")
+        candidates = [(str(k), str(v)) for k, v in criteria.items()]
+
+    elif kind == "score":
+        criteria = question.get("criteria")
+        if not isinstance(criteria, list):
+            raise ValueError("score questions need an ordered list of level descriptions")
+        candidates = [(str(i), str(d)) for i, d in enumerate(criteria)]
+
+    elif kind == "noul":
+        criteria = question.get("criteria") or {}
+        candidates = [
+            ("false", str(criteria.get("false", "No. The proposition is false for this state."))),
+            ("true", str(criteria.get("true", "Yes. The proposition is true for this state."))),
+        ]
+
+    else:
+        raise ValueError(f"Unsupported primitive: {kind!r} (expected one of {SUPPORTED_PRIMITIVES})")
+
+    if len(candidates) < 2:
+        raise ValueError("This implementation requires at least two candidates per question.")
+
+    state_text = state if isinstance(state, str) else serialize(state)
+    schema = {
+        "type": kind,
+        "instructions": question.get("instructions", ""),
+    }
+    if question.get("criteria") is not None:
+        schema["criteria"] = question["criteria"]
+
+    # Candidate first: the only tokens that differ between a question's
+    # candidates sit right after [SEP], where the encoder attends most easily.
+    # Trailing placement (candidate after the full schema) trains much slower.
+    if CANDIDATE_FIRST:
+        pairs = [
+            (state_text, serialize({"candidate": {"id": cid, "description": desc}, **schema}))
+            for cid, desc in candidates
+        ]
+    else:
+        pairs = [
+            (state_text, serialize({**schema, "candidate": {"id": cid, "description": desc}}))
+            for cid, desc in candidates
+        ]
+    return candidates, pairs
+
+
+# --------------------------------------------------------------------------- #
+# 2. Decoder (logits -> structured answer)
+# --------------------------------------------------------------------------- #
+def decode_answer(kind: str, candidates: list[tuple[str, str]], logits: torch.Tensor) -> dict:
+    probabilities = logits.float().softmax(dim=0)
+
+    if kind == "noul":
+        return {"type": "noul", "noul": float(probabilities[1])}
+
+    if kind == "choice":
+        selected = int(probabilities.argmax())
+        return {
+            "type": "choice",
+            "choice": candidates[selected][0],
+            "probabilities": {cid: float(probabilities[i]) for i, (cid, _) in enumerate(candidates)},
+        }
+
+    if kind == "score":
+        expected = sum(i * float(probabilities[i]) for i in range(len(candidates)))
+        return {
+            "type": "score",
+            "score": expected,
+            "probabilities": [float(p) for p in probabilities],
+            "legend": dict(candidates),
+        }
+
+    raise ValueError(f"Unsupported primitive: {kind!r}")
+
+
+# --------------------------------------------------------------------------- #
+# 3. Model wrapper: inference
+# --------------------------------------------------------------------------- #
+def pick_device() -> torch.device:
+    if torch.cuda.is_available():
+        return torch.device("cuda")
+    if torch.backends.mps.is_available():
+        return torch.device("mps")
+    return torch.device("cpu")
+
+
+class LocalSystemOne:
+    """TypeSafe-shaped adapter over a trained one-logit candidate scorer."""
+
+    def __init__(self, checkpoint: str, max_length: int | None = None, device: torch.device | None = None):
+        self.device = device or pick_device()
+        self.tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+        self.model = AutoModelForSequenceClassification.from_pretrained(checkpoint, dtype=torch.float32).to(self.device).eval()
+        if self.model.config.num_labels != 1:
+            raise ValueError("Expected a trained one-logit candidate scorer (num_labels == 1).")
+        # Default to the architecture's positional limit (512 for BERT/DeBERTa, 8192 for ModernBERT).
+        self.max_length = max_length or getattr(self.model.config, "max_position_embeddings", 512)
+
+    @torch.inference_mode()
+    def score_pairs(self, pairs: list[tuple[str, str]], batch_size: int = 32) -> torch.Tensor:
+        chunks = []
+        for start in range(0, len(pairs), batch_size):
+            batch = pairs[start : start + batch_size]
+            encoded = self.tokenizer(
+                [a for a, _ in batch],
+                [b for _, b in batch],
+                padding=True,
+                truncation=False,
+                return_tensors="pt",
+            )
+            if encoded["input_ids"].shape[1] > self.max_length:
+                raise ValueError(
+                    f"State + question + criteria exceed the input limit "
+                    f"({encoded['input_ids'].shape[1]} > {self.max_length} tokens)."
+                )
+            encoded = {k: v.to(self.device) for k, v in encoded.items()}
+            chunks.append(self.model(**encoded).logits.squeeze(-1).float().cpu())
+        return torch.cat(chunks) if chunks else torch.empty(0)
+
+    def system_one(self, state: Any, questions: dict[str, dict], batch_size: int = 32) -> dict:
+        all_pairs: list[tuple[str, str]] = []
+        groups = []
+        for name, question in questions.items():
+            candidates, pairs = compile_question(state, question)
+            start = len(all_pairs)
+            all_pairs.extend(pairs)
+            groups.append((name, question["type"], candidates, start, len(all_pairs)))
+
+        if not all_pairs:
+            return {"model": "local-bert-scorer", "answers": {}}
+
+        logits = self.score_pairs(all_pairs, batch_size=batch_size)
+        answers = {
+            name: decode_answer(kind, candidates, logits[start:end])
+            for name, kind, candidates, start, end in groups
+        }
+        return {"model": "local-bert-scorer", "answers": answers}
+
+
+# --------------------------------------------------------------------------- #
+# 4. Training: grouped distribution loss
+# --------------------------------------------------------------------------- #
+@dataclass
+class Example:
+    state: Any
+    question: dict
+    target: list[float]  # distribution over the compiled candidates, sums to 1
+
+
+def target_for(kind: str, label: Any, n_candidates: int) -> list[float]:
+    """Helper to build a target distribution from a plain label."""
+    if kind == "noul":
+        p = float(label)
+        return [1.0 - p, p]
+    if kind == "choice":
+        # label is the index of the correct candidate
+        t = [0.0] * n_candidates
+        t[int(label)] = 1.0
+        return t
+    if kind == "score":
+        # label is either a level index or a full distribution
+        if isinstance(label, (list, tuple)):
+            return [float(x) for x in label]
+        t = [0.0] * n_candidates
+        t[int(label)] = 1.0
+        return t
+    raise ValueError(kind)
+
+
+def grouped_distribution_loss(logits: torch.Tensor, targets: torch.Tensor, group_sizes: list[int]) -> torch.Tensor:
+    """
+    logits, targets: flat 1-D tensors, concatenation of per-question groups.
+    Loss = mean over questions of cross-entropy(target_dist, softmax(group logits)).
+    """
+    losses = []
+    offset = 0
+    for size in group_sizes:
+        g_logits = logits[offset : offset + size]
+        g_target = targets[offset : offset + size]
+        losses.append(-(g_target * g_logits.log_softmax(dim=0)).sum())
+        offset += size
+    return torch.stack(losses).mean()
+
+
+def train_scorer(
+    base_model: str,
+    examples: list[Example],
+    output_dir: str,
+    *,
+    epochs: int = 2,
+    questions_per_batch: int = 8,
+    lr: float = 3e-5,
+    max_length: int = 256,
+    seed: int = 0,
+    device: torch.device | None = None,
+    log_every: int = 25,
+    eval_fn=None,
+    bf16: bool | None = None,
+    gradient_checkpointing: bool = False,
+) -> str:
+    """
+    Fine-tune an encoder as a one-logit candidate scorer and save it.
+
+    bf16: autocast matmuls to bfloat16. Default False; see note below.
+    gradient_checkpointing: trade compute for activation memory (large models
+          on 24 GB cards).
+    """
+    random.seed(seed)
+    torch.manual_seed(seed)
+    device = device or pick_device()
+    # Default fp32. Candidates of one question differ by a handful of tokens, so
+    # their logits differ by tiny amounts early in training; bf16 rounds those to
+    # identical values and the grouped loss gets no gradient (observed: bert-base
+    # stuck at chance and DeBERTa-v3-large NaN under bf16 autocast).
+    if bf16 is None:
+        bf16 = False
+
+    tokenizer = AutoTokenizer.from_pretrained(base_model)
+    # dtype=float32: transformers 5 otherwise keeps the checkpoint's stored dtype;
+    # deberta-v3-large ships in fp16, and fp16 master weights train to NaN.
+    model = AutoModelForSequenceClassification.from_pretrained(base_model, num_labels=1, dtype=torch.float32)
+    if gradient_checkpointing:
+        model.gradient_checkpointing_enable()
+    model.to(device).train()
+    print(f"  base={base_model} params={sum(p.numel() for p in model.parameters()) / 1e6:.0f}M "
+          f"device={device} bf16={bf16} grad_ckpt={gradient_checkpointing}")
+
+    optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
+    total_steps = epochs * ((len(examples) + questions_per_batch - 1) // questions_per_batch)
+    warmup = max(1, int(0.06 * total_steps))
+
+    def lr_lambda(step: int) -> float:
+        if step < warmup:
+            return (step + 1) / warmup
+        return max(0.0, (total_steps - step) / max(1, total_steps - warmup))
+
+    scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)
+
+    step = 0
+    for epoch in range(epochs):
+        order = list(range(len(examples)))
+        random.shuffle(order)
+        running = 0.0
+        for b in range(0, len(order), questions_per_batch):
+            batch = [examples[i] for i in order[b : b + questions_per_batch]]
+            pairs, targets, sizes = [], [], []
+            for ex in batch:
+                candidates, ex_pairs = compile_question(ex.state, ex.question)
+                if len(ex.target) != len(candidates):
+                    raise ValueError("target distribution length must equal candidate count")
+                pairs.extend(ex_pairs)
+                targets.extend(ex.target)
+                sizes.append(len(candidates))
+
+            encoded = tokenizer(
+                [a for a, _ in pairs],
+                [b_ for _, b_ in pairs],
+                padding=True,
+                truncation="only_first",  # truncate the state, never the schema
+                max_length=max_length,
+                return_tensors="pt",
+            )
+            encoded = {k: v.to(device) for k, v in encoded.items()}
+            with torch.autocast(device_type=device.type, dtype=torch.bfloat16, enabled=bf16):
+                logits = model(**encoded).logits.squeeze(-1).float()
+            loss = grouped_distribution_loss(logits, torch.tensor(targets, device=device), sizes)
+
+            optimizer.zero_grad(set_to_none=True)
+            loss.backward()
+            nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+            optimizer.step()
+            scheduler.step()
+
+            running += loss.item()
+            step += 1
+            if step % log_every == 0:
+                print(f"  epoch {epoch + 1}/{epochs} step {step}/{total_steps} loss {running / log_every:.4f}")
+                running = 0.0
+
+        if eval_fn is not None:
+            model.eval()
+            eval_fn(model, tokenizer, epoch)
+            model.train()
+
+    model.eval()
+    model.save_pretrained(output_dir)
+    tokenizer.save_pretrained(output_dir)
+    return output_dir

tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_prefix_space": true,
+  "backend": "tokenizers",
+  "bos_token": "[CLS]",
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "extra_special_tokens": [
+    "[PAD]",
+    "[CLS]",
+    "[SEP]"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 256,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "split_by_punct": false,
+  "stride": 0,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "only_first",
+  "unk_id": 3,
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}