Laya Intent Router 150M: int8 ONNX, shortlist embedder, router.py, model card

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+laya.onnx.data filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,182 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: onnx
+pipeline_tag: zero-shot-classification
+base_model: jhu-clsp/ettin-encoder-150m
+tags:
+- intent-classification
+- intent-detection
+- zero-shot-classification
+- text-classification
+- chatbot
+- conversational-ai
+- customer-support
+- routing
+- out-of-scope-detection
+- onnx
+- int8
+- cpu
+- modernbert
+- ettin
+- laya
+- distillation
+datasets:
+- clinc_oos
+- PolyAI/banking77
+- FastFit/hwu_64
+- benayas/snips
+- bitext/Bitext-customer-support-llm-chatbot-training-dataset
+- bitext/Bitext-retail-ecommerce-llm-chatbot-training-dataset
+- bitext/Bitext-retail-banking-llm-chatbot-training-dataset
+metrics:
+- accuracy
+model-index:
+- name: laya-intent-router-150m-onnx
+  results:
+  - task:
+      type: zero-shot-classification
+      name: Zero-shot intent routing
+    dataset:
+      type: custom
+      name: Held-out routing suite (1,636 messages, 10 workflows)
+    metrics:
+    - type: accuracy
+      value: 0.949
+      name: Routing accuracy at threshold 0.625
+    - type: recall
+      value: 0.986
+      name: Out-of-scope recall
+---
+
+# Laya Intent Router 150M (ONNX, int8)
+
+**Zero-shot intent classification that runs on a CPU in about 60 ms.** You give it a message and a list of intents written in plain English. It tells you which intent the message belongs to, or that it belongs to none of them.
+
+No training. No labelled data. You write the intents when you call it, and you can change them on every request.
+
+```
+"i want to cancel my order 88213"         -> cancel_order     0.97
+"hey where's my parcel, it's been 5 days" -> order_status     0.97
+"ordered black shoes, got blue ones lol"  -> wrong_item       0.97
+"what's the weather in paris"             -> none             (0.98 none)
+"qwewqeqw"                                -> none             (0.98 none)
+```
+
+That last line is why I built this. The original Laya sent `qwewqeqw` to `order_not_received` with 0.73 confidence. This one puts 0.98 on none.
+
+## Try it
+
+```bash
+pip install onnxruntime tokenizers numpy huggingface_hub
+```
+
+```python
+from huggingface_hub import hf_hub_download
+import importlib.util, sys
+
+spec = importlib.util.spec_from_file_location(
+    "router", hf_hub_download("vrajnotviraj/laya-intent-router-150m-onnx", "router.py"))
+router = importlib.util.module_from_spec(spec); spec.loader.exec_module(router)
+
+r = router.Router.from_pretrained()   # downloads ~440 MB once
+
+intents = {
+    "check_balance":  ["What's my account balance"],
+    "transfer_money": ["Send money to someone", "Transfer funds between accounts"],
+    "block_card":     ["Block my card", "My card was lost or stolen"],
+    "loan_enquiry":   ["Questions about personal or home loans"],
+}
+
+r.route("i think someone stole my card", intents)
+# {'match': 'block_card', 'score': 0.98, 'probabilities': {..., '__none__': 0.01}}
+
+r.route("ok", intents)
+# {'match': None, 'score': 0.0, 'probabilities': {..., '__none__': 0.98}}
+```
+
+Each intent is a key plus 1 or more example phrasings or a short description. `match` is `None` when the message fits nothing, or when the best score is under the threshold (0.625 by default; pass `threshold=` to change it).
+
+Or just clone the repo and run `python router.py "your message here"`.
+
+## More examples
+
+A SaaS support bot, 4 intents, nothing fine-tuned:
+
+| message | match | score |
+|---|---|---|
+| cant get into my account | reset_password | 0.95 |
+| why was i charged twice this month | billing | 0.98 |
+| the export button does nothing | bug_report | 0.73 |
+| can i talk to an actual human please | talk_to_human | 0.99 |
+
+The intents were `reset_password: "User can't log in or forgot their password"`, `billing: "Questions about invoices, charges or refunds"`, `bug_report: "Something in the app is broken or showing an error"` and `talk_to_human: "User wants to speak to a real person"`. That's the whole setup.
+
+## How good is it
+
+I tested it on 1,636 hand-written messages across 10 routing setups: e-commerce, retail banking, insurance and telecom, plus an adversarial set of near-duplicates, typos, slang and "don't cancel, just tell me where it is" style traps. **Insurance and telecom were never seen in training.**
+
+| | Laya (original, zero-shot) | Laya-large fine-tuned (teacher) | **This model** |
+|---|---|---|---|
+| Routing accuracy | 0.728 | 0.950 | **0.949** |
+| Catches out-of-scope messages | 0.682 | 0.959 | **0.986** |
+| Wrongly accepts out-of-scope | 0.318 | 0.041 | **0.014** |
+| Big menus (20 to 45 intents) | 0.704 | 0.938 | **0.943** |
+| 6 brand new domains | 0.777 | | **0.920** |
+| 166 extra test messages, written last | 0.566 | 0.934 | **0.952** |
+| p95 latency, 4 CPU threads | 252 ms | 496 ms | **160 ms** |
+| Size on disk | 636 MB | 636 MB | **304 MB** (+134 MB shortlist) |
+
+So you get the big fine-tuned model's accuracy at a third of its latency and half its size. Latency was measured on an Apple M2 Pro. Your server's vCPUs are probably slower, so measure there.
+
+It also holds up when I poke at it. A perturbation test (shuffled intent order, removed correct intent, distractor intents, rewritten messages, gibberish) scores 0.92 averaged over 3 seeds, against 0.73 for the original Laya.
+
+## Big intent lists
+
+The model reads everything in one 512-token window, so it can only see so many intents at once. For lists longer than 4, a small embedder (`bge-small-en-v1.5`, bundled in `shortlist/`) picks the 4 closest intents first and the router decides between those and "none".
+
+That's on by default and it's why accuracy stays flat as the list grows:
+
+| intents | full list | with shortlist |
+|---|---|---|
+| 7 | 0.915 | 0.930 |
+| 12 | 0.969 | 0.979 |
+| 45 | 0.781 | 0.938 |
+| 148 | too long to run | 0.889 |
+
+The first call on a new intent list embeds all its phrasings (about 0.9 s for 50 intents), then it's cached. Later calls stay around 180 ms p95 whatever the list size. Pass `shortlist_k=0` to turn it off.
+
+## How it was made
+
+The base is [Laya](https://huggingface.co/convaiinnovations/laya), a ModernBERT-large decision model that scores a list of options in one forward pass. Out of the box it was too eager to match, so:
+
+1. I fine-tuned Laya-large as a teacher on about 122k routing episodes built from 7 public intent datasets (CLINC150, BANKING77, HWU64, SNIPS and 3 Bitext customer-support sets) plus 76 synthetic workflows across 16 domains. About 40% of episodes had the right intent removed, so the model learns to say "none". Loss was soft cross-entropy plus Laya's RL objective.
+2. I rewrote the prompt format. Each intent went from a wordy wrapper to `key: "phrasing 1" | "phrasing 2"`, with leftover token budget handed to intents that need it. That alone was worth about 3 points.
+3. I distilled the teacher into [Ettin-150M](https://huggingface.co/jhu-clsp/ettin-encoder-150m) with a fresh Laya head, mixing the teacher's probabilities 50/50 with the gold label. Checkpoints were picked by a perturbation-based intent score, since plain accuracy picked worse routers.
+4. I calibrated temperatures per menu size, then exported to ONNX with 8-bit weight quantization.
+
+Everything trained locally on a 32 GB M2 Pro. Training code: [github.com/vrajnotviraj/laya-intent-router](https://github.com/vrajnotviraj/laya-intent-router).
+
+## Where it slips
+
+- **English only.** It hasn't seen other languages.
+- **Filler on tiny menus.** With only 2 intents like "Hi" and "Bye", words like "ok", "well" and "nice" can land on "Bye" (0.67 to 0.81). Give short closing intents a clear description, or raise the threshold for small menus.
+- **It only knows what your phrasings say.** "My card was stolen" won't hit a `block_card` intent whose only phrasing is "Block my card". Add 2 or 3 phrasings that cover how people actually talk.
+- **Indirect requests and heavy typos** are the weakest slices (about 0.83 to 0.86).
+- My test messages were written by the same process as the synthetic training data. Real user logs might be harder. Treat the numbers as a strong hint and run your own messages through it.
+
+## Files
+
+| file | what |
+|---|---|
+| `laya.onnx`, `laya.onnx.data` | the router (int8 weights) |
+| `tokenizer.json`, `rl_agent_config.json` | tokenizer, prompt format, calibrated temperatures, threshold |
+| `shortlist/` | bge-small-en-v1.5 ONNX embedder for long intent lists |
+| `router.py` | the whole inference code, one file, no torch |
+
+## License and credits
+
+Apache-2.0. Built on [convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya) (Apache-2.0, ModernBERT-large), [jhu-clsp/ettin-encoder-150m](https://huggingface.co/jhu-clsp/ettin-encoder-150m) (MIT) and [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) (MIT).
+
+Training data: CLINC150 (CC BY 3.0, Larson et al. 2019), BANKING77 (CC BY 4.0, Casanueva et al. 2020), HWU64 (CC BY 4.0, Liu et al. 2019), SNIPS (CC0) and the Bitext customer-support, retail e-commerce and retail banking datasets (CDLA-Sharing-1.0). No training data is included here.

laya.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53e526532f19fff929506f3bffbd4faa0f6fdb684fa5842a2376e973e4091808
+size 2467583

laya.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17cfafb27b977cd0537a61da24edecad8d8f50eac2d0eb9f3e14e5bc3116cf9c
+size 299825152

rl_agent_config.json

@@ -0,0 +1,27 @@
+{
+  "head_layers": 2,
+  "max_len": 512,
+  "head_max_len": 384,
+  "max_prefixes": 6,
+  "act_costs": {
+    "escalate": 0.5
+  },
+  "cost_wrong_act": 3.0,
+  "encoder": "jhu-clsp/ettin-encoder-150m",
+  "model_name": "laya-student",
+  "amp_dtype": "bf16",
+  "temperature": [
+    1.1014399528503418,
+    1.0,
+    1.0
+  ],
+  "option_format": "compact_fill",
+  "temperature_by_options": {
+    "choice:2": 1.1014399528503418,
+    "choice:3-5": 1.1109503507614136,
+    "choice:6-10": 1.1113102436065674,
+    "choice:11+": 1.0
+  },
+  "match_threshold": 0.625,
+  "model_id": "vrajnotviraj/laya-intent-router-150m-onnx"
+}

router.py

@@ -0,0 +1,203 @@
+"""Zero-shot intent router: pick which of your paths a message belongs to, or none of them.
+
+    from router import Router
+    r = Router.from_pretrained()          # or Router("path/to/local/dir")
+    r.route("i want to cancel my order 88213", {
+        "order_status": ["Customer wants to know where their order is"],
+        "cancel_order": ["Customer wants to cancel an existing order"],
+    })
+    # {'match': 'cancel_order', 'score': 0.97, 'probabilities': {...}}
+
+Needs: pip install onnxruntime tokenizers numpy huggingface_hub
+"""
+
+import hashlib
+import json
+import os
+import re
+from collections import OrderedDict
+
+import numpy as np
+
+REPO_ID = "vrajnotviraj/laya-intent-router-150m-onnx"
+INSTRUCTIONS = (
+    "A user sent this message to a conversational workflow that branches into the paths below. "
+    "Which path is the message asking for?"
+)
+HISTORY_HINT = (
+    " `history` holds the earlier turns of this conversation, oldest first; a short or elliptical "
+    "message usually continues the intent of the most recent turns."
+)
+NO_MATCH_DESCRIPTION = "Gibberish, filler words, or a message unrelated to every other path."
+NONE = "__none__"
+OPTION_CAP, MIN_HEAD_TOKENS, PHRASING_SEP = 96, 16, '" | "'
+
+
+def _water_fill(lengths, budget, floor=4):
+    open_ = list(range(len(lengths)))
+    while open_:
+        fair = budget // len(open_)
+        short = [i for i in open_ if lengths[i] <= fair]
+        if not short:
+            break
+        budget -= sum(lengths[i] for i in short)
+        open_ = [i for i in open_ if lengths[i] > fair]
+    alloc = list(lengths)
+    for j, i in enumerate(open_):
+        alloc[i] = max(floor, budget // len(open_) + (j < budget % len(open_)))
+    return alloc
+
+
+def _humanise(path_id):
+    s = re.sub(r"([a-z0-9])([A-Z])", r"\1 \2", path_id)
+    return re.sub(r"[_\-.:/]+", " ", s).strip().lower()
+
+
+def _session(path, threads):
+    import onnxruntime as ort
+
+    o = ort.SessionOptions()
+    o.intra_op_num_threads, o.inter_op_num_threads = threads, 1
+    return ort.InferenceSession(path, o, providers=["CPUExecutionProvider"])
+
+
+class Shortlist:
+    """bge-small embedder: keeps the k paths closest to the message (best phrasing wins). Cached per path set."""
+
+    def __init__(self, model_dir, threads=4, cache_size=512):
+        from tokenizers import Tokenizer
+
+        self.tok = Tokenizer.from_file(os.path.join(model_dir, "tokenizer.json"))
+        self.tok.enable_truncation(128)
+        self.tok.enable_padding(pad_id=self.tok.token_to_id("[PAD]") or 0)
+        self.sess = _session(os.path.join(model_dir, "model.onnx"), threads)
+        self.cache, self.cache_size = OrderedDict(), cache_size
+
+    def _embed(self, texts):
+        encs = self.tok.encode_batch(texts)
+        ids = np.array([e.ids for e in encs], dtype=np.int64)
+        mask = np.array([e.attention_mask for e in encs], dtype=np.int64)
+        v = self.sess.run(None, {"input_ids": ids, "attention_mask": mask, "token_type_ids": np.zeros_like(ids)})[0][:, 0]
+        return v / np.maximum(np.linalg.norm(v, axis=1, keepdims=True), 1e-8)
+
+    def top(self, message, paths, k):
+        key = hashlib.blake2b(json.dumps(paths).encode(), digest_size=16).hexdigest()
+        if key not in self.cache:
+            texts, owner = [], []
+            for j, texts_j in enumerate(paths.values()):
+                for t in list(texts_j) + [_humanise(list(paths)[j])]:
+                    texts.append(t)
+                    owner.append(j)
+            self.cache[key] = (self._embed(texts), np.array(owner))
+            if len(self.cache) > self.cache_size:
+                self.cache.popitem(last=False)
+        self.cache.move_to_end(key)
+        vecs, owner = self.cache[key]
+        sims = vecs @ self._embed([message])[0]
+        best = np.array([sims[owner == j].max() for j in range(len(paths))])
+        keep = set(np.argsort(-best, kind="stable")[:k].tolist())
+        return {p: t for j, (p, t) in enumerate(paths.items()) if j in keep}
+
+
+class Router:
+    def __init__(self, model_dir, threads=4, shortlist_k=4):
+        from tokenizers import Tokenizer
+
+        self.cfg = json.load(open(os.path.join(model_dir, "rl_agent_config.json")))
+        self.tok = Tokenizer.from_file(os.path.join(model_dir, "tokenizer.json"))
+        self.cls, self.sep, self.mask = (self.tok.token_to_id(t) for t in ("[CLS]", "[SEP]", "[MASK]"))
+        self.sess = _session(os.path.join(model_dir, "laya.onnx"), threads)
+        self.threshold = self.cfg.get("match_threshold", 0.625)
+        self.temps = self.cfg["temperature_by_options"]
+        sl_dir = os.path.join(model_dir, "shortlist")
+        self.k = shortlist_k if os.path.isdir(sl_dir) else 0
+        self.shortlist = Shortlist(sl_dir, threads) if self.k else None
+
+    @classmethod
+    def from_pretrained(cls, repo_id=REPO_ID, **kw):
+        from huggingface_hub import snapshot_download
+
+        return cls(snapshot_download(repo_id), **kw)
+
+    def _tokens(self, text):
+        return self.tok.encode(text.replace("[MASK]", " "), add_special_tokens=False)
+
+    def _options(self, criteria, head_max_len):
+        texts = [f" {k}: {d}".replace("[MASK]", " ") for k, d in criteria.items()]
+        encs = [self.tok.encode(t, add_special_tokens=False) for t in texts]
+        options = [[self.mask] + e.ids[:OPTION_CAP] for e in encs]
+        if head_max_len - sum(map(len, options)) < MIN_HEAD_TOKENS:  # too many paths: share the budget fairly
+            alloc = _water_fill([len(o) for o in options], head_max_len - MIN_HEAD_TOKENS)
+            for i, (t, e, a) in enumerate(zip(texts, encs, alloc)):
+                n = a - 1
+                if n >= len(options[i]) - 1:
+                    continue
+                ends, keep = [end for _, end in e.offsets], n
+                p = t.find(PHRASING_SEP)
+                while p >= 0:  # prefer cutting between phrasings
+                    b = sum(x <= p + 1 for x in ends)
+                    if b > n:
+                        break
+                    if 4 * (n - b) <= a:
+                        keep = b
+                    p = t.find(PHRASING_SEP, p + 1)
+                options[i] = [self.mask] + e.ids[:keep]
+        return options
+
+    def _encode(self, message, criteria, history):
+        max_len, head_max_len = self.cfg["max_len"], self.cfg["head_max_len"]
+        options = self._options(criteria, head_max_len)
+        instructions = INSTRUCTIONS + (HISTORY_HINT if history else "")
+        head = self._tokens(f"choice question: {instructions}").ids[: max(8, head_max_len - sum(map(len, options)))]
+        ids, markers = [self.cls, *head, self.sep], []
+        for o in options:
+            markers.append(len(ids))
+            ids += o
+        ids.append(self.sep)
+        state = {"message": message, **({"history": list(history)} if history else {})}
+        room = max(0, max_len - len(ids) - 1)
+        ids = (ids + self._tokens(json.dumps(state, ensure_ascii=False)).ids[:room] + [self.sep])[:max_len]
+        if any(m >= max_len for m in markers):
+            raise ValueError("too many paths for 512 tokens; keep the shortlist on")
+        return ids, markers
+
+    def _temperature(self, n):
+        b = "2" if n <= 2 else "3-5" if n <= 5 else "6-10" if n <= 10 else "11+"
+        return min(5.0, max(0.5, float(self.temps.get(f"choice:{b}", self.cfg["temperature"][0]))))
+
+    def route(self, message, paths, history=None, threshold=None):
+        """paths: {path_id: [example phrasings or a description, ...]}. Returns the match (or None) and every probability."""
+        none_key = NONE
+        while none_key in paths:
+            none_key += "_"
+        kept = self.shortlist.top(message, paths, self.k) if self.k and len(paths) > self.k else paths
+        criteria = {p: " | ".join(f'"{t}"' for t in texts) for p, texts in kept.items()}
+        criteria[none_key] = NO_MATCH_DESCRIPTION
+        ids, markers = self._encode(message, criteria, history)
+        logits = self.sess.run(["logits"], {
+            "input_ids": np.array([ids], dtype=np.int64), "attention_mask": np.ones((1, len(ids)), dtype=np.int64),
+            "marker_pos": np.array([markers], dtype=np.int64), "marker_mask": np.ones((1, len(markers)), dtype=bool),
+            "qtype": np.zeros(1, dtype=np.int64)})[0][0]
+        z = logits / self._temperature(len(criteria))
+        p = np.exp(z - z.max())
+        p /= p.sum()
+        got = dict(zip(criteria, p.tolist()))
+        probs = {k: got.get(k, 0.0) for k in paths} | {NONE: got[none_key]}
+        best = max(paths, key=probs.get)
+        ok = probs[best] > probs[NONE] and probs[best] >= (self.threshold if threshold is None else threshold)
+        return {"match": best if ok else None, "score": probs[best], "probabilities": probs}
+
+
+if __name__ == "__main__":
+    import sys
+
+    r = Router(os.path.dirname(os.path.abspath(__file__)))
+    paths = {
+        "order_status": ["Customer wants to know the status or location of their order"],
+        "cancel_order": ["Customer wants to cancel an existing order"],
+        "return_order": ["Customer wants to return an order"],
+        "wrong_item": ["Customer received an item different from what they ordered"],
+    }
+    for msg in sys.argv[1:] or ["where is my parcel #A-7721", "qwewqeqw", "i got blue shoes but ordered black"]:
+        out = r.route(msg, paths)
+        print(f"{msg!r:45} -> {out['match']} ({out['score']:.2f})")

shortlist/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:828e1496d7fabb79cfa4dcd84fa38625c0d3d21da474a00f08db0f559940cf35
+size 133093490

shortlist/shortlist_config.json

@@ -0,0 +1 @@
+{"pooling": "cls", "max_len": 128}