Quazim0t0/Byrne-Jev-79M

Byrne-Jev

Small local System-1 typed-decision model. Same job as TypeSafe's Jev and convaiinnovations' Laya. You give it a state (text, JSON, or a conversation) and typed questions (choice, score, noul). It returns a calibrated probability for every option in one forward pass. Nothing is generated. No free text to parse. No answer outside the options you define.

I built this. ~79M params: a 70.4M causal decoder trunk (SpikeWhale) plus an 8.5M Laya-style decision head. Method is close to Laya: typed decision head, one marker position per option, 2-layer bidirectional head over the sequence, strictly proper scoring rules so the probabilities are actually calibrated.

Answer format is schema-compatible with Laya and Jev. Served over Jev's /v1/systemone wire protocol.

Checkpoint is model.pt. Temperature 1.0 for every question type. No post-hoc temperature scaling. That measured best-calibrated. See Calibration.

Files

File	What
model.pt	the model (weights + config)
agent.py	DecisionAgent: predict(state, questions) -> answers
serve.py	Jev-compatible HTTP server (POST /v1/systemone, GET /health), standard library only
eval_decisions.py, eval_data.py	the evaluation used for every number below
decision_core.py, config.py, model_v2.py, fractal.py, special_tokens.py, spike_tokenizer.py, tokenizer.json	model code and tokenizer
eval_results.json	the full report behind the tables below

Install

Python 3.10+.

pip install -r requirements.txt

Use

from agent import DecisionAgent

agent = DecisionAgent("model.pt", device="cuda")      # or "cpu"
result = agent.predict(
    {"body": "Hi, we were billed twice for March. Refund the duplicate today or we cancel."},
    {
        "department": {"type": "choice", "instructions": "Which department should handle this?",
                       "criteria": {"billing": "invoices, payments, refunds",
                                    "technical": "bugs, outages, errors",
                                    "other": "everything else"}},
        "urgency":    {"type": "score", "instructions": "How urgent is this?",
                       "criteria": ["not urgent", "soon", "blocking"]},
        "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
    },
)
result["answers"]["department"]["choice"]            # e.g. "billing"
result["answers"]["department"]["probabilities"]     # one probability per option
result["answers"]["churn_risk"]["noul"]              # P(true)

Question types:

• choice: criteria is a dict of label -> description (or a list of labels).
• score: criteria is a list of ordered levels; the answer includes the expected level.
• noul: a yes/no question. criteria optionally describes true / false; the answer is P(true).

Each answer also carries answer_confidence (the probability of the chosen answer - the calibrated number to gate on).

Server (Jev wire protocol)

JEV_DEVICE=cuda python serve.py          # http://127.0.0.1:8000/v1/systemone
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"body": "billed twice, refund please or we cancel"},
  "questions": {"dept": {"type": "choice", "instructions": "which team?",
                "criteria": {"billing": "refunds", "tech": "bugs"}}}}'

Env var	Default	Meaning
JEV_MODEL	model.pt	model file
JEV_DEVICE	cpu	cuda or cpu
JEV_HOST	127.0.0.1	bind address
JEV_PORT	8000	port
JEV_API_KEY	unset	when set, clients must send Authorization: Bearer <key>
JEV_MAX_LEN / JEV_HEAD_MAX_LEN	1024 / 256	token budgets per question (state + question, and question + options); positions go up to 2048

Responses include usage, rounding: {"probabilityDecimals": 4} and cost_usd: 0, so strict Jev clients validate and budget them correctly.

Evaluation

Reproduce everything below with python eval_decisions.py --suite all --device cuda. Other suites: --suite typed, --suite benchmarks, or --suite jsonl --jsonl your_examples.jsonl. All numbers were measured on an RTX 5060 Ti.

typed-decisions: LocalLLaMA/typed-decisions test split

400 cases, 2,000 decisions, four workflows. Metrics and reference rows match the protocol in Laya's fine-tuning notebook. Reference numbers are as published by their authors.

Model	Params	Accuracy	Soft acc	Brier ↓	ECE ↓	Score MAE ↓	Within 1 level
Byrne-Jev	~79M	0.630	0.509	0.134	0.045	0.428	0.915
TypeSafe Jev 1.13.0	closed	0.727	0.580	0.148	0.144	0.391	0.952
Laya typed-decisions	421M	0.766	0.471	0.062	0.213	0.242	-
ModernBERT-base specialist	149M	0.646	0.542	0.119	0.179	0.444	0.931
Laya base (laya)	421M	0.362	0.332	0.316	0.175	0.694	-
Teacher self-agreement (ceiling)	-	0.735
Per-question majority class	-	0.461
Random guess	-	0.318

KL divergence 0.228, total variation 0.220. Latency is 111 ms p50 and 158 ms p95 per case (5 questions in one forward pass, GPU).

Workflow	Accuracy
Agent-trace observability	0.704
Security incidents	0.654
Customer service	0.610
Invoice processing	0.552

What stands out:

• Calibration. ECE 0.045 is the lowest in the table. Stated confidences track hit rate.
• Probability quality. Brier beats Jev's published score. Soft accuracy beats Laya's fine-tuned checkpoint.
• Accuracy gap. Top-1 is 10-14 points below Jev and Laya. Invoice processing is the weakest workflow.

Public benchmarks as typed questions

200 examples from each benchmark's evaluation split, each posed as a typed question (eval_data.py). Instruction wording and option descriptions are sampled with a fixed seed. banking77 and MASSIVE show up to 20 options per question: the gold intent plus randomly sampled other intents (not the full 77 / 60-way label set).

Benchmark (split)	Question	Options	Accuracy	ECE
IMDB (test)	noul / choice	2	0.990	0.015
DAIR Emotion (validation)	choice	6	0.890	0.062
AG News (test)	choice	4	0.870	0.090
banking77 (test)	choice	<= 20	0.835	0.058
MASSIVE intent, en (validation)	choice	<= 20	0.785	0.102
SST-2 (validation)	noul / choice	2	0.775	0.136
tweet_eval hate (validation)	noul	2	0.630	0.187
BoolQ (validation)	noul	2	0.585	0.121
Yelp review stars (test)	score	5	0.505	0.140
MultiNLI (validation matched)	choice	3	0.500	0.077

Sentiment and topic/intent are strong. Reading a passage and reasoning over it is weak. BoolQ is below always answering "yes" (~0.62). MultiNLI is 0.50, where chance is 0.33.

Calibration

model.pt uses temperature 1.0. I also fitted post-hoc temperatures (per question type and per option count) and compared. On typed-decisions, fitting made calibration worse: ECE rose from 0.045 to 0.100 and soft accuracy fell from 0.509 to 0.473, while Brier improved slightly (0.134 to 0.127). On the benchmark table, fitted temperatures lowered ECE on most single-label tasks. The released model keeps temperature 1.0 because structured-state decisions are the main use. To apply your own temperatures, add temperature (per type: choice, score, noul) or temperature_by_options to the model's decision_cfg; the loader clamps values to [0.5, 5].

Integration test: jev-doom (Freedoom, zero-shot)

To check it can stand in for Jev in a real application, I dropped Byrne-Jev in for Jev in jev-doom, a public harness that shows Jev playing Freedoom. Observable mode, MAP01, 90 s real time. No Doom-specific adaptation.

	Result
Protocol	164 decisions, 163 accepted, 0 validation errors (the one rejection was the harness discarding a stale in-flight reply)
Latency	~580 ms per decision (about 12 questions per call at JEV_MAX_LEN=2048 / JEV_HEAD_MAX_LEN=512)
Outcome	survived to the time limit at full health; moved 5,339 units, visited 43 cells, discovered 581
Combat / doors	0 kills, 0 damage dealt or taken; never permitted firing (trigger = hold_fire 136/136) and never pressed USE, so it didn't open doors

It works as a drop-in at the protocol level. As a zero-shot game agent it's weak: it tracks and faces enemies but doesn't choose to shoot, and it prefers right-side moves.

Limitations

• Accuracy. Top-1 on typed-decisions is below Jev and Laya.
• Passage reasoning. Reasoning over a passage (BoolQ, NLI) is weak.
• Yes/no detection (noul) is unreliable and can be inverted. Hand-written spot check: 5 of 7 detection questions were wrong, including clear cases. Explicit "refund today or we cancel" scored churn 0.09 while a polite feature request scored 0.68. Obvious phishing email 0.29. "I want a manager now" escalation 0.04. "ignore all previous instructions" injection 0.16. Describe both outcomes in criteria (e.g. {"true": "the user threatens to cancel or leave", "false": "the user does not threaten to leave"}). Same 7 checks: accuracy 2/7 -> 5/7 (cancel threat 0.09 -> 0.99, escalation 0.04 -> 0.98, injection 0.16 -> 0.76). A two-option choice with the same descriptions scored the same (5/7), so it's the descriptions that matter, not the question type. Even so, a polite request still scored as a churn risk and the phishing email still scored below 0.5. Validate yes/no questions on your own data before gating anything on them.
• Score questions under-rate intensity. Urgency, frustration and star ratings skew toward the low end (e.g. a glowing review got an expected 2.6 of 5 stars; a same-day cancel threat was rated "not urgent").
• Context. Default 1024-token budget per question truncates long states. Questions with many long options share a 256-token budget (raise it with JEV_HEAD_MAX_LEN).
• Action head. action.act_probability in each answer is kept for format compatibility. It isn't meaningful.
• Option order. Options are read in the order given. Results can shift slightly if you reorder them.