Duplicate from convaiinnovations/laya-typed-decisions

Co-authored-by: Convai Innovations <convaiinnovations@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,137 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-classification
+language: [en]
+tags: [laya, system-one, calibrated-decisions, rlcd, typed-decisions, classification, agent-observability, invoice-processing, security-incidents, customer-service, commercial-use]
+---
+
+<p align="center">
+  <img src="https://huggingface.co/convaiinnovations/laya/resolve/main/assets/logo-mark.png" alt="" width="72" />
+</p>
+
+# Laya Typed-Decisions
+
+Non-autoregressive **System 1 decision model**, fine-tuned on the
+[typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) workflows: agent-trace
+observability, customer service, invoice processing and security incidents.
+
+Part of the [Laya family](https://huggingface.co/convaiinnovations/laya).
+
+| checkpoint | encoder | params | context | use it for |
+|---|---|---|---|---|
+| [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) | ModernBERT-large | 421M | 512 | English, general |
+| [`convaiinnovations/laya-multilingual`](https://huggingface.co/convaiinnovations/laya-multilingual) | mmBERT-base | 322M | 1024 | 100+ languages |
+| **`convaiinnovations/laya-typed-decisions`** (this repo) | ModernBERT-large | 421M | 1024 | these four workflows |
+
+## Benchmark
+
+400 test cases, 2,000 decisions, measured on the official test split.
+
+| model | accuracy | soft acc | Brier | ECE | score MAE |
+|---|---|---|---|---|---|
+| **this checkpoint** | **0.766** | 0.471 | **0.062** | 0.213 | **0.242** |
+| *TypeSafe Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* |
+| *teacher self-agreement ceiling* | *0.735* | | | | |
+| *ModernBERT-base specialist (published)* | *0.646* | | | | |
+| *per-question majority class* | *0.461* | | | | |
+| *random guess* | *0.318* | | | | |
+| `laya` (not fine-tuned) | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 |
+| `laya-multilingual` (not fine-tuned) | 0.342 | 0.326 | 0.439 | 0.285 | 0.687 |
+
+**+3.9 points over Jev's published 0.727, above the 0.735 teacher ceiling**, with 2.4x better
+Brier and 1.6x better score MAE.
+
+Jev figures are third-party published, not measured here — there is no TypeSafe API access in
+this project, and sample sizes and prompts differ. Treat the comparison as indicative.
+
+### By workflow
+
+| workflow | accuracy |
+|---|---|
+| invoice processing | **0.804** |
+| security incidents | 0.766 |
+| customer service | 0.764 |
+| agent-trace observability | 0.730 |
+
+### By primitive
+
+| type | accuracy | ECE | n |
+|---|---|---|---|
+| `noul` | **0.857** | 0.192 | 600 |
+| `choice` | 0.733 | 0.255 | 600 |
+| `score` | 0.723 | 0.199 | 800 |
+
+## Quickstart
+
+```bash
+pip install laya
+```
+
+```python
+import laya
+
+agent = laya.load("convaiinnovations/laya-typed-decisions")
+result = agent.predict(state, questions)
+```
+
+Or route to it explicitly:
+
+```python
+from laya import Router
+
+router = Router()
+router.predict(state, questions, model="typed-decisions")
+```
+
+`Router` will not select this checkpoint automatically unless you construct it with
+`auto_task_detection=True` — it is specialised to four synthetic workflows and should not be a
+silent default.
+
+If this checkpoint is on a hot path, keep it resident rather than loading it per request:
+
+```python
+router = Router()
+router.preload(["typed-decisions"])          # or router.attach("typed-decisions", agent)
+```
+
+`preload` fetches and builds the checkpoints you name once; `attach` registers an `Agent` you
+already hold, so nothing is loaded twice.
+
+> **If `laya.load()` hangs:** `transformers` probes for TensorFlow at import, and when TF is
+> installed its abseil runtime can deadlock model construction. Run with `USE_TF=0`.
+
+## Training
+
+Fine-tuned from [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) on the
+benchmark's 1,200-case training split (6,000 decisions) with **RLCD**: the policy reports a
+distribution, exploration adds zero-mean Gaussian noise to the logits, and the reward is a
+strictly proper scoring rule (log + spherical, plus ranked probability score for ordinal
+questions), so expected reward is maximised only by honest probabilities. Updates are REINFORCE
+with a group-mean baseline, alongside soft cross-entropy against the teacher's distributions.
+
+Reproduce it:
+[`laya_finetune_typed_decisions_2xT4_kaggle.ipynb`](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb)
+— about 4–5 hours on Kaggle's free 2xT4.
+
+## Limits
+
+- **This is a specialist.** It was fine-tuned on four specific synthetic workflows. Expect it to
+  behave like the base `laya` checkpoint, or worse, on anything else.
+- **Soft accuracy trails Jev** (0.471 vs 0.580): its argmax is better, but its probability
+  *distributions* match the teacher less well.
+- **Still over-confident** (ECE 0.213 vs Jev's 0.144). Its `temperature_by_options` was inherited
+  from the base checkpoint and overrides the per-type temperatures fitted for this model — refit
+  on your own held-out data before relying on the probabilities.
+- English only. Use `laya-multilingual` for other languages.
+- **Keep `choice` questions under ~20 options.** Options share a fixed 256-token head budget, so
+  a large label space leaves few tokens per label and accuracy falls off sharply.
+
+## Links
+
+- **Hub / family** https://huggingface.co/convaiinnovations/laya
+- **GitHub** https://github.com/NandhaKishorM/laya · full benchmark data on the `research` branch
+- **PyPI** https://pypi.org/project/laya/
+- **Benchmark dataset** https://huggingface.co/datasets/LocalLLaMA/typed-decisions
+
+Apache 2.0 · Convai Innovations

encoder/config.json

@@ -0,0 +1,84 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "vocab_size": 50368
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fa56de72383a9d3efa9cfa78955733c81b9fc8067a587ca4beb82c78107a24e
+size 842609220

rl_agent_config.json

@@ -0,0 +1,36 @@
+{
+  "encoder": "answerdotai/ModernBERT-large",
+  "head_layers": 2,
+  "max_len": 1024,
+  "head_max_len": 256,
+  "max_prefixes": 6,
+  "act_costs": {
+    "escalate": 0.5
+  },
+  "cost_wrong_act": 3.0,
+  "amp_dtype": "bf16",
+  "model_name": "laya-typed-decisions",
+  "temperature": [
+    1.0148024559020996,
+    1.0374259948730469,
+    1.0575125217437744
+  ],
+  "temperature_by_options": {
+    "choice:3-5": 1.7601518630981445,
+    "choice:6-10": 1.0000158548355103,
+    "score:3-5": 1.2514300346374512,
+    "noul:2": 1.983399510383606,
+    "choice:11+": 0.10058280825614929,
+    "choice:2": 1.9063563346862793
+  },
+  "training": {
+    "updates": 7313,
+    "epochs_completed": 1,
+    "hours": 1.96,
+    "world_size": 1,
+    "fine_tuned_from_checkpoint": true
+  },
+  "gradient_checkpointing": true,
+  "max_tokens_per_batch": 4096,
+  "fine_tuned": true
+}

tokenizer/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}