Publish evaluated auto-0.4b-2 checkpoint and reproducibility artifacts

README.md

@@ -0,0 +1,114 @@
+---
+license: apache-2.0
+base_model:
+  - ProCreations/auto-0.4b
+pipeline_tag: text-classification
+library_name: transformers
+datasets:
+  - ProCreations/auto-1b-data
+tags:
+  - modernbert
+  - agent-safety
+  - tool-calling
+  - long-context
+model-index:
+  - name: auto-0.4b-2
+    results:
+      - task:
+          type: text-classification
+          name: Agentic tool-call approve/deny classification
+        dataset:
+          name: Approve-or-Deny
+          type: ProCreations/approve-or-deny
+          split: test
+          revision: a38b625913dd46ca9702063f1597c979e6ace34e
+        metrics:
+          - type: accuracy
+            value: 0.9676666666666667
+          - type: f1
+            value: 0.9654927072216293
+            name: F1 (deny)
+          - type: roc_auc
+            value: 0.9943784012045359
+          - type: false_approve_rate
+            value: 0.03140613847251963
+          - type: false_deny_rate
+            value: 0.03314571607254534
+---
+
+# auto-0.4b-2
+
+A 395.8M-parameter ModernBERT encoder for deciding whether an agent's proposed tool call is authorized and safe in context. Continued full-parameter training of [auto-0.4b](https://huggingface.co/ProCreations/auto-0.4b), with supervised training on short and full-length long-context examples; teacher distillation was skipped because the post-SFT benchmark accuracy exceeded the teacher, on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU.
+
+## Measured results
+
+The same pinned 3,000-item benchmark, input serialization, tokenizer, full input lengths, attention implementation, and default `P(deny) >= 0.5` threshold were used for all three models below. Original and teacher results are fresh measurements; older model-card scores can differ across data/runtime revisions.
+
+| Model | Accuracy | False approve | False deny | AUROC |
+|---|---:|---:|---:|---:|
+| Original auto-0.4b | 89.63% | 6.07% | 14.13% | 0.9655 |
+| Teacher auto-1b-bf16 | 96.60% | 4.14% | 2.75% | 0.9930 |
+| **auto-0.4b-2** | 96.77% | 3.14% | 3.31% | 0.9944 |
+
+Accuracy improved by **7.10 percentage points**; paired bootstrap 95% interval: 6.07 to 8.17 points. Accuracy Wilson 95% interval: 96.07%–97.34%.
+
+Accuracy on inputs of 16,384–65,536 tokens: **94.56%** (239 items). The untouched validation audit partition scored **97.88%** across 2595 items. Full category, difficulty, language, length, confusion counts, confidence intervals, and threshold results are in `eval_results.json`.
+
+The separately calibrated recommended threshold is `0.340`. It was chosen only on 2,581 validation calibration rows. The table above uses 0.5, without test-set threshold tuning.
+
+## Training and evaluation integrity
+
+- 711,985 retained training examples, 516,733,212 tokens. Exact normalized-text deduplication and request-plus-call group separation removed 15 rows. There was no detected benchmark overlap.
+- 13,000 validation rows were split by request-plus-call group into 7,824 selection, 2,581 calibration, and 2,595 untouched audit rows. Benchmark examples were excluded from training targets, teacher-target generation, threshold tuning, and checkpoint selection.
+- At the user's request, a checkpoint frozen on validation was evaluated after long SFT, scoring 96.77% benchmark accuracy. This intermediate benchmark decided whether to run distillation: distill if the student did not strictly beat the teacher's 96.60%; otherwise skip it. Consequently the benchmark was used for this training-procedure decision and is not an untouched one-use final test. The independent validation audit remained unused until final evaluation.
+- Two short-context supervised epochs; one full-length long-context supervised epoch with short replay; teacher distillation was skipped under the user-requested benchmark comparison rule.
+- Every retained training row was used at full tokenized length. No training example required truncation. Configured capacity is 65,536 tokens; the supplied training corpus reaches 51,408 tokens and benchmark reaches 56,176 with this tokenizer. The 65,536-token path is a capacity smoke test, not a claim of measured accuracy at exactly 65,536 tokens.
+- Full-parameter AdamW with FP32 master weights and moments, BF16 compute, cosine learning-rate schedules, gradient clipping, and activation checkpointing for long inputs. BF16 safetensors are published.
+- No teacher distillation was applied to this release.
+- Each checkpoint was frozen using validation selection scores before its benchmark evaluation. Training sources and immutable revisions, data audit, plan, environment versions, and runnable scripts are included.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+"""Serialize exactly the information available to an approve/deny classifier."""
+def build_input(user_request, history, call):
+    parts = ['### PROPOSED TOOL CALL', f"tool: {call['tool']}", f"args: {call['args']}", '',
+             '### USER REQUEST', user_request, '', '### AGENT HISTORY']
+    if not history:
+        parts.append('(no prior actions)')
+    else:
+        for i, h in enumerate(history):
+            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result', '')}")
+    return '\n'.join(parts)
+
+
+repo = "ProCreations/auto-0.4b-2"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForSequenceClassification.from_pretrained(
+    repo, dtype=torch.bfloat16,
+    attn_implementation="kernels-community/flash-attn2@81fb77c12b2ad5d69380669b46739d5868614502",
+).cuda().eval()
+
+text = build_input(
+    user_request="Clean up the build artifacts and reinstall dependencies.",
+    history=[{"tool": "Bash", "args": "ls", "result": "node_modules dist package.json"}],
+    call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
+)
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda")
+with torch.inference_mode():
+    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
+print("deny" if p_deny >= 0.5 else "approve", p_deny)
+```
+
+Install the tested versions listed in `environment.json`, including `kernels`; the pinned Hugging Face FlashAttention kernel loads automatically on supported CUDA systems. A compatible local FlashAttention 2 installation can also use `attn_implementation="flash_attention_2"`. Ordinary SDPA is suitable for shorter inputs but can materialize large masks for long sequences.
+
+Labels are `0 = approve`, `1 = deny`. Preserve the exact `### PROPOSED TOOL CALL`, `### USER REQUEST`, and `### AGENT HISTORY` sections. The model only consumes this serialized text; training rationales and metadata are excluded.
+
+## Scope and limitations
+
+This is a classification model, not a chat model. It approves routine authorized work and denies consequential unauthorized actions or actions following injected instructions. Its synthetic labels and held-out benchmark do not establish reliability on every real agent workload. A false approval remains possible; the model cannot inspect hidden file contents, resolve opaque executables, or determine a URL's runtime behavior from text alone. Long-context results are slice measurements on the supplied benchmark, not a universal guarantee. Evaluate on representative deployment traffic and treat uncertain decisions appropriately for the application.
+
+The base model and teacher were previously developed using their own validation histories; this run prevents new training/evaluation overlap but cannot independently establish that all historical model-development decisions were untouched by public benchmarks.

artifact_hashes.json

@@ -0,0 +1,14 @@
+{
+  "eval_results.json": "dc4bb0d424873883e0ff1c1e0bb53a1630a34649dc11b031b2749e11386ca9df",
+  "README.md": "abd0e0fd8e83f6b595ce439845b5331c29a902392f1ba50aecf1fedf738940fc",
+  "config.json": "d6f114a76d668e44b2720fd8d8a8e48d42a9ee969b0b174d391912008a14f9ec",
+  "environment.json": "de9a91afa9816139ab20e489fc3a7ca3887324d8a2b8291d5ceca5fd8d53fe95",
+  "data_audit.json": "659d61976ceee6241d66612b452693a2d4af5eeeaf870dd95e4521e57a21d839",
+  "benchmark_predictions.npz": "5ef6fed2fb1faae86c25b9290b65ce50204249a778b48510e7e8b7536c8783e8",
+  "model.safetensors": "10a27866e72a8fd7af878eb3d41e68f2de8e0f025cc20e55ef531554d1abc626",
+  "tokenizer.json": "abf58f8fa056b6f6c5bb0b667af9dd5893bfe96022f8981d6daa7571fdbc830c",
+  "tokenizer_config.json": "f0fcf16a9b5a657f9441b1fe7d66a0060c1151c645cf8e44d26aca5697f5c7ca",
+  "gate.py": "ae657066ddcaaf7065e21327a216d78c34f8cf6871e30083d79432b531f4c690",
+  "source_revisions.json": "e574b8bc8d025c33bbb2f1d564e8abb3247bb23c04bcd6b73d58c0c099fbce85",
+  "training_plan.json": "3a87e7c6b1e7534fdf35ef903f9f07225ce9bb10696bd6d5f953b5b7a8d8d25d"
+}

benchmark_predictions.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ef6fed2fb1faae86c25b9290b65ce50204249a778b48510e7e8b7536c8783e8
+size 72762

config.json

@@ -0,0 +1,92 @@
+{
+  "architectures": [
+    "ModernBertForSequenceClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "cls",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "approve",
+    "1": "deny"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "label2id": {
+    "approve": 0,
+    "deny": 1
+  },
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 65536,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 2560000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
+  "vocab_size": 50368
+}

data_audit.json

@@ -0,0 +1,99 @@
+{
+  "sources": {
+    "student": {
+      "repo": "ProCreations/auto-0.4b",
+      "revision": "2f6ec6ca79d733fa8c1677676a77cc85b739f0d2"
+    },
+    "teacher": {
+      "repo": "ProCreations/auto-1b-bf16",
+      "revision": "328c2d93e0390610bbeab4dea0000d0670a51be2"
+    },
+    "data": {
+      "repo": "ProCreations/auto-1b-data",
+      "revision": "d265bbf7bac76cd55daaf81a1d185dfe089a412b"
+    },
+    "benchmark": {
+      "repo": "ProCreations/approve-or-deny",
+      "revision": "a38b625913dd46ca9702063f1597c979e6ace34e"
+    }
+  },
+  "deduplication": "SHA256 whitespace-normalized text; request+canonical call group exclusion across splits; conflicting exact-text labels removed from train.",
+  "seed": 20260908,
+  "benchmark_rows": 3000,
+  "validation_original": 13000,
+  "validation_partitions": {
+    "selection": 7824,
+    "calibration": 2581,
+    "audit": 2595,
+    "monitor": 2025
+  },
+  "train_original": 712000,
+  "removed_train": {
+    "validation_group_overlap": 14,
+    "duplicate_text": 1
+  },
+  "train": {
+    "rows": 711985,
+    "tokens": 516733212,
+    "label_counts": {
+      "approve": 355072,
+      "deny": 356913
+    },
+    "length_percentiles": {
+      "p0": 44.0,
+      "p25": 221.0,
+      "p50": 374.0,
+      "p75": 509.0,
+      "p90": 657.0,
+      "p95": 815.0,
+      "p99": 11582.0,
+      "p100": 51408.0
+    },
+    "over_model_limit": 0,
+    "gt4096": 18578,
+    "gt16384": 4202
+  },
+  "validation": {
+    "rows": 13000,
+    "tokens": 15660962,
+    "label_counts": {
+      "approve": 6475,
+      "deny": 6525
+    },
+    "length_percentiles": {
+      "p0": 47.0,
+      "p25": 227.0,
+      "p50": 391.0,
+      "p75": 531.0,
+      "p90": 777.0,
+      "p95": 5837.399999999994,
+      "p99": 21976.960000000043,
+      "p100": 48710.0
+    },
+    "over_model_limit": 0,
+    "gt4096": 827,
+    "gt16384": 185
+  },
+  "benchmark": {
+    "rows": 3000,
+    "tokens": 11710540,
+    "label_counts": {
+      "deny": 1401,
+      "approve": 1599
+    },
+    "length_percentiles": {
+      "p0": 49.0,
+      "p25": 229.75,
+      "p50": 403.0,
+      "p75": 824.25,
+      "p90": 11993.599999999999,
+      "p95": 25400.399999999998,
+      "p99": 51709.639999999985,
+      "p100": 56176.0
+    },
+    "over_model_limit": 0,
+    "gt4096": 450,
+    "gt16384": 239
+  },
+  "monitor_sampling": "All selection rows over 4096 tokens plus 1536 fixed random short selection rows"
+}

environment.json

@@ -0,0 +1,10 @@
+{
+  "torch": "2.13.0",
+  "transformers": "5.16.1",
+  "datasets": "5.0.1",
+  "huggingface_hub": "1.30.0",
+  "kernels": "0.16.1",
+  "accelerate": "1.14.0",
+  "numpy": "2.5.2",
+  "scikit-learn": "1.9.0"
+}

gate.py

@@ -0,0 +1,10 @@
+"""Serialize exactly the information available to an approve/deny classifier."""
+def build_input(user_request, history, call):
+    parts = ['### PROPOSED TOOL CALL', f"tool: {call['tool']}", f"args: {call['args']}", '',
+             '### USER REQUEST', user_request, '', '### AGENT HISTORY']
+    if not history:
+        parts.append('(no prior actions)')
+    else:
+        for i, h in enumerate(history):
+            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result', '')}")
+    return '\n'.join(parts)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10a27866e72a8fd7af878eb3d41e68f2de8e0f025cc20e55ef531554d1abc626
+size 791684972

source_revisions.json

@@ -0,0 +1,22 @@
+{
+  "student": {
+    "repo": "ProCreations/auto-0.4b",
+    "revision": "2f6ec6ca79d733fa8c1677676a77cc85b739f0d2",
+    "kind": "model"
+  },
+  "teacher": {
+    "repo": "ProCreations/auto-1b-bf16",
+    "revision": "328c2d93e0390610bbeab4dea0000d0670a51be2",
+    "kind": "model"
+  },
+  "data": {
+    "repo": "ProCreations/auto-1b-data",
+    "revision": "d265bbf7bac76cd55daaf81a1d185dfe089a412b",
+    "kind": "dataset"
+  },
+  "benchmark": {
+    "repo": "ProCreations/approve-or-deny",
+    "revision": "a38b625913dd46ca9702063f1597c979e6ace34e",
+    "kind": "dataset"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 65536,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 65536,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "tokenizer_class": "TokenizersBackend",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

training/download.py

@@ -0,0 +1,27 @@
+import concurrent.futures, json, pathlib
+from huggingface_hub import snapshot_download
+
+ROOT = pathlib.Path('/home/user/.local/share/rtx-pro-apps/auto-0.4b-2')
+SOURCES = {
+    'student': ('ProCreations/auto-0.4b', 'model', '2f6ec6ca79d733fa8c1677676a77cc85b739f0d2'),
+    'teacher': ('ProCreations/auto-1b-bf16', 'model', '328c2d93e0390610bbeab4dea0000d0670a51be2'),
+    'data': ('ProCreations/auto-1b-data', 'dataset', 'd265bbf7bac76cd55daaf81a1d185dfe089a412b'),
+    'benchmark': ('ProCreations/approve-or-deny', 'dataset', 'a38b625913dd46ca9702063f1597c979e6ace34e'),
+}
+
+def get(item):
+    name, (repo, kind, rev) = item
+    path = snapshot_download(repo, repo_type=kind, revision=rev, max_workers=4)
+    print(json.dumps({'downloaded': name, 'path': path}), flush=True)
+    return name, {'repo': repo, 'kind': kind, 'revision': rev, 'path': path}
+
+if __name__ == '__main__':
+    with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
+        manifest = dict(pool.map(get, SOURCES.items()))
+    (ROOT / 'sources.json').write_text(json.dumps(manifest, indent=2))
+    import pyarrow.parquet as pq
+    for name, filename in [('data', 'train.parquet'), ('data', 'validation.parquet'), ('benchmark', 'test.parquet')]:
+        f = pq.ParquetFile(pathlib.Path(manifest[name]['path']) / filename)
+        print(name, filename, f.metadata.num_rows, f.schema_arrow, flush=True)
+        row = next(f.iter_batches(batch_size=1)).to_pylist()[0]
+        print(json.dumps({k: str(v)[:200] for k, v in row.items()}, ensure_ascii=False), flush=True)

training/engine.py

@@ -0,0 +1,320 @@
+"""Reproducible full-parameter ModernBERT fine-tuning and binary logit distillation."""
+import os
+os.environ.setdefault('TOKENIZERS_PARALLELISM', 'false')
+os.environ.setdefault('OMP_NUM_THREADS', '8')
+import gc, hashlib, json, math, pathlib, random, shutil, signal, time
+import numpy as np
+import torch
+import torch.nn.functional as F
+from datasets import load_from_disk
+from scipy.special import softmax
+from sklearn.metrics import accuracy_score, balanced_accuracy_score, f1_score, roc_auc_score, log_loss
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+from torch.utils.data import DataLoader
+
+ROOT = pathlib.Path('/home/user/.local/share/rtx-pro-apps/auto-0.4b-2')
+DATA = pathlib.Path('/home/user/datasets/auto-0.4b-2')
+CKPT = pathlib.Path('/home/user/checkpoints/auto-0.4b-2')
+LOG = pathlib.Path('/home/user/logs/auto-0.4b-2')
+ATTENTION = 'kernels-community/flash-attn2@81fb77c12b2ad5d69380669b46739d5868614502'
+SEED = 20260908
+STOP = False
+torch.set_num_threads(8)
+torch.set_float32_matmul_precision('high')
+torch.backends.cuda.matmul.allow_tf32 = True
+
+def atomic_json(path, value):
+    path = pathlib.Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_suffix(path.suffix + '.tmp')
+    tmp.write_text(json.dumps(value, indent=2, allow_nan=False))
+    tmp.replace(path)
+
+def event(kind, **kwargs):
+    item = {'event': kind, 'time': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), **kwargs}
+    print(json.dumps(item, allow_nan=False), flush=True)
+    LOG.mkdir(exist_ok=True, parents=True)
+    with (LOG / 'events.jsonl').open('a') as f:
+        f.write(json.dumps(item, allow_nan=False) + '\n')
+    atomic_json(ROOT / 'status.json', item)
+
+def seed_all(seed=SEED):
+    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
+
+def load_model(path, train=False):
+    model = AutoModelForSequenceClassification.from_pretrained(
+        str(path), dtype=torch.float32 if train else torch.bfloat16, attn_implementation=ATTENTION).cuda()
+    assert model.config.id2label == {0: 'approve', 1: 'deny'}
+    assert model.config.max_position_embeddings == 65536
+    model.train(train)
+    return model
+
+def release(model):
+    del model
+    gc.collect(); torch.cuda.empty_cache()
+
+def metrics(labels, logits, threshold=0.5):
+    labels = np.asarray(labels)
+    probs = softmax(np.asarray(logits, dtype=np.float64), axis=1)[:, 1]
+    pred = probs >= threshold
+    deny, approve = labels == 1, labels == 0
+    recalls = ([float(pred[deny].mean())] if deny.any() else []) + ([float((~pred[approve]).mean())] if approve.any() else [])
+    out = {'n': len(labels), 'accuracy': float(accuracy_score(labels, pred)),
+           'balanced_accuracy': float(np.mean(recalls)),
+           'f1_deny': float(f1_score(labels, pred, zero_division=0)),
+           'auroc': float(roc_auc_score(labels, probs)) if deny.any() and approve.any() else None,
+           'nll': float(log_loss(labels, np.stack([1-probs, probs], axis=1), labels=[0, 1])),
+           'false_approve_rate': float((~pred[deny]).mean()) if deny.any() else None,
+           'false_deny_rate': float(pred[approve].mean()) if approve.any() else None,
+           'false_approve_count': int((~pred[deny]).sum()), 'false_deny_count': int(pred[approve].sum()),
+           'deny_count': int(deny.sum()), 'approve_count': int(approve.sum()), 'threshold': float(threshold),
+           'brier': float(np.mean((probs-labels)**2))}
+    # Wilson binomial interval for accuracy.
+    p, n, z = out['accuracy'], len(labels), 1.959963984540054
+    center = (p + z*z/(2*n)) / (1+z*z/n)
+    half = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / (1+z*z/n)
+    out['accuracy_ci95'] = [center-half, center+half]
+    return out
+
+def report(ds, indices, logits, threshold=0.5):
+    sub = ds.select([int(i) for i in indices])
+    labels = np.array(sub['labels'])
+    out = {'overall': metrics(labels, logits, threshold), 'slices': {}}
+    lens = np.array(sub['length'])
+    buckets = np.where(lens < 1024, '<1k', np.where(lens < 4096, '1k-4k', np.where(lens < 16384, '4k-16k', '16k-64k')))
+    for field, values in [('length', buckets), ('category', np.array(sub['category'])),
+                          ('difficulty', np.array(sub['difficulty'])), ('lang', np.array(sub['lang']))]:
+        out['slices'][field] = {str(v): metrics(labels[values == v], logits[values == v], threshold) for v in np.unique(values)}
+    return out
+
+def collate(rows):
+    length = ((max(len(r['input_ids']) for r in rows)+7)//8)*8
+    ids = torch.full((len(rows), length), 50283, dtype=torch.long)
+    mask = torch.zeros_like(ids)
+    for i, row in enumerate(rows):
+        n = len(row['input_ids']); ids[i, :n] = torch.tensor(row['input_ids']); mask[i, :n] = 1
+    return {'input_ids': ids, 'attention_mask': mask, 'labels': torch.tensor([r['labels'] for r in rows], dtype=torch.long)}
+
+def batches(indices, lengths, token_budget=16384, max_batch=32, seed=None):
+    indices = np.asarray(indices, dtype=np.int64).copy()
+    rng = np.random.default_rng(seed)
+    if seed is None:
+        indices = indices[np.argsort(lengths[indices], kind='stable')]
+        chunks = [indices]
+    else:
+        rng.shuffle(indices)
+        chunks = [c[np.argsort(lengths[c], kind='stable')] for c in np.array_split(indices, max(1, math.ceil(len(indices)/2048)))]
+    result = []
+    for chunk in chunks:
+        batch, maxlen = [], 0
+        for idx in chunk:
+            n = int(lengths[idx])
+            if batch and ((len(batch)+1) * max(maxlen, n) > token_budget or len(batch) >= max_batch):
+                result.append(batch); batch, maxlen = [], 0
+            batch.append(int(idx)); maxlen = max(maxlen, n)
+        if batch: result.append(batch)
+    if seed is not None: rng.shuffle(result)
+    return result
+
+def optimizer_groups(microbatches, lengths, examples=128, tokens=131072):
+    groups, group, count, total = [], [], 0, 0
+    for batch in microbatches:
+        group.append(batch); count += len(batch); total += int(lengths[batch].sum())
+        if count >= examples or total >= tokens:
+            groups.append(group); group, count, total = [], 0, 0
+    if group: groups.append(group)
+    return groups
+
+@torch.inference_mode()
+def predict(model, ds, indices, name, output=None, token_budget=32768):
+    indices = np.asarray(indices, dtype=np.int64)
+    if output is not None and pathlib.Path(output).exists():
+        saved = np.load(output)
+        assert np.array_equal(saved['indices'], indices), 'Prediction cache index mismatch'
+        return saved['logits']
+    lengths = np.array(ds['length'])
+    bs = batches(indices, lengths, token_budget=token_budget, max_batch=64)
+    loader = DataLoader(ds, batch_sampler=bs, collate_fn=collate, num_workers=2, pin_memory=True)
+    logits = np.full((len(ds), 2), np.nan, dtype=np.float32)
+    was_training = model.training; model.eval()
+    start, done, last = time.monotonic(), 0, 0
+    for batch_indices, batch in zip(bs, loader):
+        x = {k: v.cuda(non_blocking=True) for k, v in batch.items() if k != 'labels'}
+        with torch.autocast('cuda', dtype=torch.bfloat16):
+            pred = model(**x).logits.float().cpu().numpy()
+        if not np.isfinite(pred).all(): raise RuntimeError('Nonfinite evaluation logits')
+        logits[batch_indices] = pred; done += len(batch_indices)
+        if time.monotonic()-last > 45:
+            event('evaluating', name=name, done=done, total=len(indices), elapsed_seconds=time.monotonic()-start)
+            last = time.monotonic()
+    model.train(was_training)
+    result = logits[indices]
+    assert np.isfinite(result).all()
+    if output is not None:
+        pathlib.Path(output).parent.mkdir(parents=True, exist_ok=True)
+        temp = str(output) + '.tmp.npz'; np.savez(temp, indices=indices, logits=result); os.replace(temp, output)
+    event('evaluation_complete', name=name, n=len(indices), elapsed_seconds=time.monotonic()-start)
+    return result
+
+def selection_score(r):
+    short_and_long = r['overall']['balanced_accuracy']
+    long = r['slices']['length'].get('16k-64k', {}).get('balanced_accuracy', short_and_long)
+    return 0.7 * short_and_long + 0.3 * long
+
+def export_model(model, path, tokenizer):
+    path = pathlib.Path(path)
+    staging = path.with_name(path.name + '.staging')
+    backup = path.with_name(path.name + '.previous')
+    if staging.exists(): shutil.rmtree(staging)
+    staging.mkdir(parents=True, exist_ok=True)
+    state = {k: v.detach().cpu().to(torch.bfloat16) if v.is_floating_point() else v.detach().cpu() for k,v in model.state_dict().items()}
+    old_dtype = model.config.dtype
+    model.config.dtype = torch.bfloat16
+    model.save_pretrained(str(staging), state_dict=state, safe_serialization=True)
+    model.config.dtype = old_dtype
+    tokenizer.save_pretrained(str(staging))
+    if backup.exists(): shutil.rmtree(backup)
+    if path.exists(): path.rename(backup)
+    staging.rename(path)
+    if backup.exists(): shutil.rmtree(backup)
+    del state
+
+def save_resume(model, optimizer, scheduler, path, **progress):
+    state = {'model': model.state_dict(), 'optimizer': optimizer.state_dict(), 'scheduler': scheduler.state_dict(),
+             'rng_torch': torch.get_rng_state(), 'rng_cuda': torch.cuda.get_rng_state_all(),
+             'rng_numpy': np.random.get_state(), 'rng_python': random.getstate(), **progress}
+    tmp = str(path) + '.tmp'; torch.save(state, tmp); os.replace(tmp, path)
+
+def stop_handler(*_):
+    global STOP
+    STOP = True
+
+def train_phase(name, start_path, indices, epochs, lr, teacher_logits=None, alpha=0.5, temperature=2.0):
+    phase = CKPT / name; phase.mkdir(parents=True, exist_ok=True)
+    done_path = phase / 'complete.json'
+    if done_path.exists(): return json.loads(done_path.read_text())['best_path']
+    seed_all()
+    train = load_from_disk(str(DATA / 'train')); val = load_from_disk(str(DATA / 'validation'))
+    lengths = np.array(train['length'])
+    monitor = np.load(DATA / 'validation_partitions.npz')['monitor']
+    tokenizer = AutoTokenizer.from_pretrained(str(start_path))
+    model = load_model(start_path, train=True)
+    params = [p for p in model.parameters() if p.requires_grad]
+    optimizer = torch.optim.AdamW(params, lr=lr, betas=(0.9, 0.95), eps=1e-8, weight_decay=0.01, fused=True)
+    all_groups = [optimizer_groups(batches(indices, lengths, seed=SEED+e), lengths) for e in range(epochs)]
+    total_steps = sum(map(len, all_groups)); warmup = max(20, int(total_steps*0.03))
+    def schedule(step):
+        if step < warmup: return (step+1)/warmup
+        fraction = min(1., (step-warmup)/max(1,total_steps-warmup))
+        return 0.1 + 0.9*0.5*(1+math.cos(math.pi*fraction))
+    scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, schedule)
+    start_epoch = start_group = global_step = 0; best_score = -1.; best_path = str(phase/'best')
+    resume_path = phase / 'resume.pt'
+    if resume_path.exists():
+        resume = torch.load(resume_path, map_location='cpu', weights_only=False)
+        model.load_state_dict(resume['model']); optimizer.load_state_dict(resume['optimizer']); scheduler.load_state_dict(resume['scheduler'])
+        start_epoch, start_group, global_step = resume['epoch'], resume['next_group'], resume['step']
+        best_score = resume['best_score']
+        if (phase/'best_metrics.json').exists():
+            best_score = max(best_score, json.loads((phase/'best_metrics.json').read_text())['score'])
+        torch.set_rng_state(resume['rng_torch']); torch.cuda.set_rng_state_all(resume['rng_cuda'])
+        np.random.set_state(resume['rng_numpy']); random.setstate(resume['rng_python'])
+        del resume
+        event('resumed', phase=name, epoch=start_epoch, group=start_group, step=global_step)
+    else:
+        initial = predict(model, val, monitor, name+'-initial')
+        initial_report = report(val, monitor, initial); best_score = selection_score(initial_report)
+        export_model(model, best_path, tokenizer)
+        atomic_json(phase/'best_metrics.json', {'score': best_score, 'step': 0, 'report': initial_report})
+    teacher = np.load(teacher_logits, mmap_mode='r') if teacher_logits else None
+    if teacher is not None: assert teacher.shape == (len(train), 2) and np.isfinite(teacher).all()
+    event('phase_started', phase=name, rows=len(indices), epochs=epochs, lr=lr, total_steps=total_steps, distillation=teacher is not None)
+    last_log = last_save = time.monotonic(); started = last_log; seen = tokens_seen = 0; loss_sum = 0.; loss_examples = 0
+    ckpt_enabled = False
+    eval_interval = max(100, math.ceil(total_steps / (epochs*4)))
+    signal.signal(signal.SIGTERM, stop_handler); signal.signal(signal.SIGINT, stop_handler)
+    for epoch, groups in enumerate(all_groups):
+        if epoch < start_epoch: continue
+        begin = start_group if epoch == start_epoch else 0
+        # Background CPU collation, in exact deterministic group order.
+        remaining_batches = [b for group in groups[begin:] for b in group]
+        loader = iter(DataLoader(train, batch_sampler=remaining_batches, collate_fn=collate, num_workers=4, pin_memory=True, prefetch_factor=2))
+        for group_idx in range(begin, len(groups)):
+            group = groups[group_idx]; n_group = sum(map(len, group)); optimizer.zero_grad(set_to_none=True)
+            for batch_indices in group:
+                batch = next(loader)
+                want_checkpoint = batch['input_ids'].shape[1] > 8192
+                if want_checkpoint != ckpt_enabled:
+                    if want_checkpoint: model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={'use_reentrant': False})
+                    else: model.gradient_checkpointing_disable()
+                    ckpt_enabled = want_checkpoint
+                x = {k: v.cuda(non_blocking=True) for k,v in batch.items()}
+                labels = x.pop('labels')
+                with torch.autocast('cuda', dtype=torch.bfloat16):
+                    logits = model(**x).logits.float()
+                    ce = F.cross_entropy(logits, labels, reduction='none')
+                    if teacher is not None:
+                        tl = torch.tensor(np.array(teacher[batch_indices]), device='cuda', dtype=torch.float32)
+                        target = F.softmax(tl/temperature, dim=-1)
+                        kl = F.kl_div(F.log_softmax(logits/temperature, dim=-1), target, reduction='none').sum(-1)*temperature**2
+                        # Preserve hard-label evidence on teacher mistakes while still learning soft uncertainty.
+                        weight = torch.where(tl.argmax(-1) == labels, alpha, alpha*0.25)
+                        per_example = (1-weight)*ce + weight*kl
+                    else: per_example = ce
+                    loss = per_example.sum()/n_group
+                if not torch.isfinite(loss): raise RuntimeError('Nonfinite training loss')
+                loss.backward()
+                loss_sum += float(per_example.detach().sum()); loss_examples += len(batch_indices)
+                seen += len(batch_indices); tokens_seen += int(lengths[batch_indices].sum())
+                del logits, loss, per_example, ce, x, labels
+            grad_norm = torch.nn.utils.clip_grad_norm_(params, 1.0, error_if_nonfinite=True)
+            optimizer.step(); scheduler.step(); global_step += 1
+            now = time.monotonic()
+            if now-last_log > 45 or global_step % 100 == 0:
+                event('training', phase=name, epoch=epoch+1, step=global_step, total_steps=total_steps,
+                      loss=loss_sum/max(1,loss_examples), grad_norm=float(grad_norm), lr=scheduler.get_last_lr()[0],
+                      examples_this_run=seen, tokens_per_second=tokens_seen/max(1,now-started), elapsed_seconds=now-started)
+                loss_sum = 0.; loss_examples = 0; last_log=now
+            if global_step % eval_interval == 0 or group_idx == len(groups)-1:
+                pred = predict(model, val, monitor, name+f'-step{global_step}')
+                r = report(val, monitor, pred); score=selection_score(r)
+                atomic_json(phase/f'monitor-{global_step}.json', r)
+                event('validation', phase=name, step=global_step, score=score, metrics=r['overall'], long_metrics=r['slices']['length'].get('16k-64k'))
+                if score > best_score:
+                    best_score = score; export_model(model, best_path, tokenizer)
+                    atomic_json(phase/'best_metrics.json', {'score':score,'step':global_step,'report':r})
+            if now-last_save > 600 or STOP or group_idx == len(groups)-1:
+                save_resume(model, optimizer, scheduler, resume_path, epoch=epoch, next_group=group_idx+1, step=global_step, best_score=best_score)
+                last_save = time.monotonic()
+            if STOP:
+                event('stopped_safely', phase=name, step=global_step)
+                raise SystemExit(75)
+    export_model(model, phase/'last', tokenizer)
+    atomic_json(done_path, {'best_path':best_path,'score':best_score,'steps':global_step,'last_path':str(phase/'last')})
+    event('phase_complete', phase=name, steps=global_step, best_score=best_score)
+    del optimizer, scheduler, params, model
+    gc.collect(); torch.cuda.empty_cache()
+    return best_path
+
+def baseline():
+    sources = json.loads((ROOT/'sources.json').read_text())
+    ds = load_from_disk(str(DATA/'benchmark')); val = load_from_disk(str(DATA/'validation'))
+    selection = np.load(DATA/'validation_partitions.npz')['selection']
+    out = CKPT/'baseline'; out.mkdir(exist_ok=True, parents=True)
+    student_tok=AutoTokenizer.from_pretrained(sources['student']['path']); teacher_tok=AutoTokenizer.from_pretrained(sources['teacher']['path'])
+    assert student_tok.get_vocab() == teacher_tok.get_vocab()
+    assert student_tok('unicode Ω 中文 test')['input_ids'] == teacher_tok('unicode Ω 中文 test')['input_ids']
+    for name in ['student','teacher']:
+        if (out/f'{name}.json').exists(): continue
+        model=load_model(sources[name]['path'])
+        logits=predict(model,ds,np.arange(len(ds)),name+'-benchmark',out/f'{name}-benchmark.npz')
+        r={'benchmark':report(ds,np.arange(len(ds)),logits)}
+        logits=predict(model,val,selection,name+'-validation',out/f'{name}-selection.npz')
+        r['selection']=report(val,selection,logits)
+        atomic_json(out/f'{name}.json',r)
+        event('baseline_complete',model=name,benchmark=r['benchmark']['overall'],selection=r['selection']['overall'])
+        del model; gc.collect(); torch.cuda.empty_cache()
+
+if __name__ == '__main__':
+    import sys
+    if sys.argv[1] == 'baseline': baseline()

training/prepare.py

@@ -0,0 +1,116 @@
+"""Pinned data preparation; no benchmark labels are used to build training data."""
+import os
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+import collections, hashlib, json, pathlib, re
+import numpy as np
+from datasets import Dataset, load_from_disk
+from transformers import AutoTokenizer
+
+ROOT = pathlib.Path('/home/user/.local/share/rtx-pro-apps/auto-0.4b-2')
+DATA = pathlib.Path('/home/user/datasets/auto-0.4b-2')
+SOURCES = json.loads((ROOT / 'sources.json').read_text())
+TOKENIZER = None
+KEEP = ['label', 'category', 'difficulty', 'lang', 'is_long', 'needle_depth', 'subset', 'text_hash', 'group_hash', 'source_row']
+
+def digest(s):
+    return hashlib.sha256(re.sub(r'\s+', ' ', s).strip().encode()).hexdigest()
+
+def fingerprints(batch, indices):
+    groups = []
+    for req, call in zip(batch['user_request'], batch['call']):
+        try:
+            call = json.dumps(json.loads(call), sort_keys=True, ensure_ascii=False)
+        except (ValueError, TypeError):
+            pass
+        groups.append(digest(req + '\n' + call))
+    return {'text_hash': [digest(t) for t in batch['text']], 'group_hash': groups, 'source_row': indices}
+
+def tokenize(batch):
+    global TOKENIZER
+    if TOKENIZER is None:
+        TOKENIZER = AutoTokenizer.from_pretrained(SOURCES['student']['path'])
+    ids = TOKENIZER(batch['text'], truncation=False, return_attention_mask=False)['input_ids']
+    original = [len(x) for x in ids]
+    # Model limit only. Short-context stages exclude long rows instead of corrupting their labels by truncation.
+    ids = [x if len(x) <= 65536 else x[:65535] + [TOKENIZER.sep_token_id] for x in ids]
+    return {'input_ids': ids, 'length': [len(x) for x in ids], 'original_length': original,
+            'labels': [0 if x == 'approve' else 1 for x in batch['label']]}
+
+def load_raw(name, filename):
+    ds = Dataset.from_parquet(str(pathlib.Path(SOURCES[name]['path']) / filename), cache_dir=str(DATA / 'cache'))
+    assert set(ds.unique('label')) == {'approve', 'deny'}
+    return ds.map(fingerprints, batched=True, batch_size=256, with_indices=True, num_proc=8, desc='Fingerprint ' + filename)
+
+def save_tokens(ds, name):
+    path = DATA / name
+    if path.exists():
+        return load_from_disk(str(path))
+    ds = ds.map(tokenize, batched=True, batch_size=64, num_proc=8,
+                remove_columns=[k for k in ds.column_names if k not in KEEP], desc='Tokenize ' + name)
+    ds.save_to_disk(str(path), max_shard_size='512MB')
+    return ds
+
+if __name__ == '__main__':
+    DATA.mkdir(parents=True, exist_ok=True)
+    audit = {'sources': {k: {a: v[a] for a in ['repo', 'revision']} for k, v in SOURCES.items()},
+             'deduplication': 'SHA256 whitespace-normalized text; request+canonical call group exclusion across splits; conflicting exact-text labels removed from train.',
+             'seed': 20260908}
+    bench = load_raw('benchmark', 'test.parquet')
+    bench_text, bench_groups = set(bench['text_hash']), set(bench['group_hash'])
+    audit['benchmark_rows'] = len(bench)
+    bench = save_tokens(bench, 'benchmark')
+    val = load_raw('data', 'validation.parquet')
+    vtexts, vgroups = list(val['text_hash']), list(val['group_hash'])
+    seen, keep = set(), []
+    for i, (t, g) in enumerate(zip(vtexts, vgroups)):
+        if t not in bench_text and g not in bench_groups and t not in seen:
+            keep.append(i)
+            seen.add(t)
+    audit['validation_original'] = len(val)
+    val = val.select(keep)
+    val_text, val_groups = set(val['text_hash']), set(val['group_hash'])
+    val = save_tokens(val, 'validation')
+    partition = np.array([int(h[:8], 16) % 10 for h in val['group_hash']])
+    selection = np.flatnonzero(partition < 6)
+    calibration = np.flatnonzero((partition >= 6) & (partition < 8))
+    audit_indices = np.flatnonzero(partition >= 8)
+    rng = np.random.default_rng(20260908)
+    # All long selection examples plus a fixed random short subset for stable long-context monitoring.
+    lengths = np.array(val['length'])
+    short_selection = selection[lengths[selection] <= 4096]
+    monitor = np.sort(np.concatenate([rng.choice(short_selection, min(1536, len(short_selection)), replace=False), selection[lengths[selection] > 4096]]))
+    np.savez(DATA / 'validation_partitions.npz', selection=selection, calibration=calibration, audit=audit_indices, monitor=monitor)
+    audit['validation_partitions'] = {k: len(v) for k, v in [('selection', selection), ('calibration', calibration), ('audit', audit_indices), ('monitor', monitor)]}
+    (DATA / 'eval_ready.json').write_text(json.dumps(audit, indent=2))
+    train = load_raw('data', 'train.parquet')
+    texts, groups, labels = list(train['text_hash']), list(train['group_hash']), list(train['label'])
+    by_text, conflicts = {}, set()
+    for t, y in zip(texts, labels):
+        if t in by_text and by_text[t] != y:
+            conflicts.add(t)
+        by_text[t] = y
+    seen, keep, removed = set(), [], collections.Counter()
+    for i, (t, g) in enumerate(zip(texts, groups)):
+        reason = ('benchmark_text_overlap' if t in bench_text else
+                  'benchmark_group_overlap' if g in bench_groups else
+                  'validation_text_overlap' if t in val_text else
+                  'validation_group_overlap' if g in val_groups else
+                  'conflicting_labels' if t in conflicts else
+                  'duplicate_text' if t in seen else None)
+        if reason:
+            removed[reason] += 1
+        else:
+            keep.append(i)
+            seen.add(t)
+    audit['train_original'] = len(train)
+    audit['removed_train'] = dict(removed)
+    train = save_tokens(train.select(keep), 'train')
+    for name, ds in [('train', train), ('validation', val), ('benchmark', bench)]:
+        lengths = np.array(ds['length'])
+        audit[name] = {'rows': len(ds), 'tokens': int(lengths.sum()),
+                       'label_counts': dict(collections.Counter(ds['label'])),
+                       'length_percentiles': dict(zip(['p0','p25','p50','p75','p90','p95','p99','p100'], np.percentile(lengths, [0,25,50,75,90,95,99,100]).tolist())),
+                       'over_model_limit': int(sum(n > 65536 for n in ds['original_length'])),
+                       'gt4096': int((lengths > 4096).sum()), 'gt16384': int((lengths > 16384).sum())}
+    (DATA / 'data_audit.json').write_text(json.dumps(audit, indent=2))
+    print(json.dumps(audit, indent=2), flush=True)

training/publish.py

@@ -0,0 +1,260 @@
+"""Freeze on validation, audit once, publish only an improvement that meets declared quality gates."""
+import gc, hashlib, json, pathlib, shutil, time
+import numpy as np
+import torch
+from datasets import load_from_disk
+from scipy.special import softmax
+from scipy.stats import binomtest
+from huggingface_hub import HfApi, snapshot_download
+from huggingface_hub.errors import RepositoryNotFoundError
+from engine import ROOT, DATA, CKPT, LOG, ATTENTION, SEED, atomic_json, event, load_model, predict, report, metrics, selection_score
+
+REPO='ProCreations/auto-0.4b-2'
+
+def paired_comparison(labels, original, final):
+    b=(original.argmax(-1)==labels); f=(final.argmax(-1)==labels)
+    difference=f.astype(float)-b.astype(float)
+    rng=np.random.default_rng(SEED)
+    means=np.array([rng.choice(difference,size=len(difference),replace=True).mean() for _ in range(10000)])
+    improved=int((f&~b).sum()); regressed=int((b&~f).sum())
+    return {'accuracy_gain':float(difference.mean()),'gain_ci95':np.quantile(means,[0.025,0.975]).tolist(),
+            'fixed_errors':improved,'new_errors':regressed,
+            'mcnemar_exact_p':float(binomtest(improved,improved+regressed,0.5).pvalue) if improved+regressed else 1.0}
+
+def finalize(selected_sft=None):
+    from run import PLAN
+    val=load_from_disk(str(DATA/'validation'));bench=load_from_disk(str(DATA/'benchmark'))
+    parts=dict(np.load(DATA/'validation_partitions.npz'))
+    evaluated=CKPT/'final_evaluation';evaluated.mkdir(exist_ok=True,parents=True)
+    frozen_path=evaluated/'frozen_selection.json'
+    if frozen_path.exists(): frozen=json.loads(frozen_path.read_text())
+    elif selected_sft is not None:
+        frozen={'chosen':selected_sft,'selection_data':'Validation-only checkpoint selection; distillation skipped by the user-requested post-SFT benchmark comparison rule.'}
+        atomic_json(evaluated/'selection_candidates.json',[selected_sft])
+        atomic_json(frozen_path,frozen)
+    else:
+        candidates=[]
+        for phase in ['sft_short','sft_long','kd_short','kd_long']:
+            for which in ['best','last']:
+                path=CKPT/phase/which
+                model=load_model(path)
+                logits=predict(model,val,parts['selection'],phase+'-'+which+'-selection',evaluated/f'{phase}-{which}-selection.npz')
+                r=report(val,parts['selection'],logits)
+                kd_updates=0
+                if phase.startswith('kd_'):
+                    progress=json.loads((CKPT/phase/('complete.json' if which=='last' else 'best_metrics.json')).read_text())
+                    kd_updates=int(progress['steps'] if which=='last' else progress['step'])
+                    if phase=='kd_long':kd_updates+=int(json.loads((CKPT/'kd_short/best_metrics.json').read_text())['step'])
+                candidates.append({'path':str(path),'name':phase+'-'+which,'score':selection_score(r),'distillation_optimizer_steps':kd_updates,'report':r})
+                del model;gc.collect();torch.cuda.empty_cache()
+        atomic_json(evaluated/'selection_candidates.json',candidates)
+        # Final release must have completed distillation. Compare KD candidates; retain SFT comparisons for the report.
+        eligible=[x for x in candidates if x['name'].startswith('kd_') and x['distillation_optimizer_steps']>0]
+        chosen=max(eligible,key=lambda x:(x['score'],-x['report']['overall']['nll']))
+        frozen={'chosen':chosen,'selection_data':'validation selection only; benchmark and validation audit excluded'}
+        atomic_json(frozen_path,frozen)
+    chosen=frozen['chosen'];model=load_model(chosen['path'])
+    cal_logits=predict(model,val,parts['calibration'],'final-calibration',evaluated/'calibration.npz')
+    cal_labels=np.array(val.select(parts['calibration'].tolist())['labels'])
+    # A separately documented operating point; raw model/default-threshold quality is what gates publication.
+    thresholds=np.arange(0.05,0.951,0.005)
+    opts=[metrics(cal_labels,cal_logits,float(t)) for t in thresholds]
+    operating=max(opts,key=lambda x:(x['balanced_accuracy'],-abs(x['threshold']-0.5)))['threshold']
+    atomic_json(evaluated/'calibration.json',{'recommended_threshold':operating,'selection':'maximum balanced accuracy; closest to 0.5 on ties','rows':len(cal_labels)})
+    audit_logits=predict(model,val,parts['audit'],'final-validation-audit',evaluated/'audit.npz')
+    final_logits=predict(model,bench,np.arange(len(bench)),'final-benchmark',evaluated/'benchmark.npz')
+    audit=report(val,parts['audit'],audit_logits)
+    benchmark=report(bench,np.arange(len(bench)),final_logits)
+    tuned=report(bench,np.arange(len(bench)),final_logits,operating)
+    baseline_student=json.loads((CKPT/'baseline/student.json').read_text())
+    baseline_teacher=json.loads((CKPT/'baseline/teacher.json').read_text())
+    original=np.load(CKPT/'baseline/student-benchmark.npz')['logits']
+    pair=paired_comparison(np.array(bench['labels']),original,final_logits)
+    b=baseline_student['benchmark']['overall'];f=benchmark['overall'];gate=PLAN['publish_gate']
+    checks={
+        'accuracy_at_least_95_percent':f['accuracy']>=gate['benchmark_accuracy_min'],
+        'long_context_at_least_90_percent':benchmark['slices']['length']['16k-64k']['accuracy']>=gate['benchmark_long_accuracy_min'],
+        'false_approve_improved':b['false_approve_rate']-f['false_approve_rate']>=gate['false_approve_improvement_min'],
+        'false_deny_improved':b['false_deny_rate']-f['false_deny_rate']>=gate['false_deny_improvement_min'],
+        'validation_audit_at_least_95_percent':audit['overall']['accuracy']>=gate['validation_audit_accuracy_min'],
+        'paired_improvement_ci_excludes_zero':pair['gain_ci95'][0]>gate['paired_accuracy_ci95_lower_min'],
+    }
+    result={'selected_checkpoint':chosen,'benchmark_default':benchmark,'benchmark_calibrated_threshold':tuned,
+            'validation_audit':audit,'recommended_threshold':operating,'paired_comparison':pair,
+            'baseline_student':baseline_student,'baseline_teacher':baseline_teacher,'quality_checks':checks,
+            'runtime_attention':ATTENTION,'benchmark_revision':json.loads((ROOT/'sources.json').read_text())['benchmark']['revision'],
+            'distillation_performed':selected_sft is None,
+            'post_sft_comparison':json.loads((ROOT/'post_sft_benchmark.json').read_text())}
+    atomic_json(evaluated/'results.json',result)
+    if not all(checks.values()):
+        event('quality_gate_failed',checks=checks,benchmark=f,audit=audit['overall'],results_path=str(evaluated/'results.json'))
+        raise SystemExit(20)
+    publish(model,chosen['path'],result,bench,final_logits)
+
+def model_card(r,audit,plan):
+    b=r['baseline_student']['benchmark']['overall'];t=r['baseline_teacher']['benchmark']['overall'];f=r['benchmark_default']['overall']
+    long=r['benchmark_default']['slices']['length']['16k-64k'];gain=r['paired_comparison']
+    distilled=r['distillation_performed']
+    teacher_base='  - ProCreations/auto-1b-bf16\n' if distilled else ''
+    distillation_tag='  - distillation\n' if distilled else ''
+    method='followed by logit distillation from [auto-1b-bf16](https://huggingface.co/ProCreations/auto-1b-bf16)' if distilled else 'with supervised training on short and full-length long-context examples; teacher distillation was skipped because the post-SFT benchmark accuracy exceeded the teacher'
+    stages='then teacher logits over all training rows; one short distillation epoch and one long distillation epoch with replay' if distilled else 'teacher distillation was skipped under the user-requested benchmark comparison rule'
+    kd_detail=f"Distillation uses temperature {plan['kd_short']['temperature']} and a mixture of hard-label cross entropy and teacher KL divergence. Teacher disagreement reduces the KL weight so teacher errors do not replace the dataset labels." if distilled else 'No teacher distillation was applied to this release.'
+    intermediate=r['post_sft_comparison']['benchmark']['overall']['accuracy']
+    def row(name,m): return f"| {name} | {100*m['accuracy']:.2f}% | {100*m['false_approve_rate']:.2f}% | {100*m['false_deny_rate']:.2f}% | {m['auroc']:.4f} |"
+    return f'''---
+license: apache-2.0
+base_model:
+  - ProCreations/auto-0.4b
+{teacher_base}pipeline_tag: text-classification
+library_name: transformers
+datasets:
+  - ProCreations/auto-1b-data
+tags:
+  - modernbert
+{distillation_tag}  - agent-safety
+  - tool-calling
+  - long-context
+model-index:
+  - name: auto-0.4b-2
+    results:
+      - task:
+          type: text-classification
+          name: Agentic tool-call approve/deny classification
+        dataset:
+          name: Approve-or-Deny
+          type: ProCreations/approve-or-deny
+          split: test
+          revision: {r['benchmark_revision']}
+        metrics:
+          - type: accuracy
+            value: {f['accuracy']}
+          - type: f1
+            value: {f['f1_deny']}
+            name: F1 (deny)
+          - type: roc_auc
+            value: {f['auroc']}
+          - type: false_approve_rate
+            value: {f['false_approve_rate']}
+          - type: false_deny_rate
+            value: {f['false_deny_rate']}
+---
+
+# auto-0.4b-2
+
+A 395.8M-parameter ModernBERT encoder for deciding whether an agent's proposed tool call is authorized and safe in context. Continued full-parameter training of [auto-0.4b](https://huggingface.co/ProCreations/auto-0.4b), {method}, on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU.
+
+## Measured results
+
+The same pinned 3,000-item benchmark, input serialization, tokenizer, full input lengths, attention implementation, and default `P(deny) >= 0.5` threshold were used for all three models below. Original and teacher results are fresh measurements; older model-card scores can differ across data/runtime revisions.
+
+| Model | Accuracy | False approve | False deny | AUROC |
+|---|---:|---:|---:|---:|
+{row('Original auto-0.4b',b)}
+{row('Teacher auto-1b-bf16',t)}
+{row('**auto-0.4b-2**',f)}
+
+Accuracy improved by **{100*gain['accuracy_gain']:.2f} percentage points**; paired bootstrap 95% interval: {100*gain['gain_ci95'][0]:.2f} to {100*gain['gain_ci95'][1]:.2f} points. Accuracy Wilson 95% interval: {100*f['accuracy_ci95'][0]:.2f}%–{100*f['accuracy_ci95'][1]:.2f}%.
+
+Accuracy on inputs of 16,384–65,536 tokens: **{100*long['accuracy']:.2f}%** ({long['n']} items). The untouched validation audit partition scored **{100*r['validation_audit']['overall']['accuracy']:.2f}%** across {r['validation_audit']['overall']['n']} items. Full category, difficulty, language, length, confusion counts, confidence intervals, and threshold results are in `eval_results.json`.
+
+The separately calibrated recommended threshold is `{r['recommended_threshold']:.3f}`. It was chosen only on 2,581 validation calibration rows. The table above uses 0.5, without test-set threshold tuning.
+
+## Training and evaluation integrity
+
+- {audit['train']['rows']:,} retained training examples, {audit['train']['tokens']:,} tokens. Exact normalized-text deduplication and request-plus-call group separation removed 15 rows. There was no detected benchmark overlap.
+- 13,000 validation rows were split by request-plus-call group into 7,824 selection, 2,581 calibration, and 2,595 untouched audit rows. Benchmark examples were excluded from training targets, teacher-target generation, threshold tuning, and checkpoint selection.
+- At the user's request, a checkpoint frozen on validation was evaluated after long SFT, scoring {100*intermediate:.2f}% benchmark accuracy. This intermediate benchmark decided whether to run distillation: distill if the student did not strictly beat the teacher's {100*t['accuracy']:.2f}%; otherwise skip it. Consequently the benchmark was used for this training-procedure decision and is not an untouched one-use final test. The independent validation audit remained unused until final evaluation.
+- Two short-context supervised epochs; one full-length long-context supervised epoch with short replay; {stages}.
+- Every retained training row was used at full tokenized length. No training example required truncation. Configured capacity is 65,536 tokens; the supplied training corpus reaches 51,408 tokens and benchmark reaches 56,176 with this tokenizer. The 65,536-token path is a capacity smoke test, not a claim of measured accuracy at exactly 65,536 tokens.
+- Full-parameter AdamW with FP32 master weights and moments, BF16 compute, cosine learning-rate schedules, gradient clipping, and activation checkpointing for long inputs. BF16 safetensors are published.
+- {kd_detail}
+- Each checkpoint was frozen using validation selection scores before its benchmark evaluation. Training sources and immutable revisions, data audit, plan, environment versions, and runnable scripts are included.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+{(ROOT/'gate.py').read_text()}
+
+repo = "ProCreations/auto-0.4b-2"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForSequenceClassification.from_pretrained(
+    repo, dtype=torch.bfloat16,
+    attn_implementation="{ATTENTION}",
+).cuda().eval()
+
+text = build_input(
+    user_request="Clean up the build artifacts and reinstall dependencies.",
+    history=[{{"tool": "Bash", "args": "ls", "result": "node_modules dist package.json"}}],
+    call={{"tool": "Bash", "args": "rm -rf node_modules dist && npm install"}},
+)
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda")
+with torch.inference_mode():
+    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
+print("deny" if p_deny >= 0.5 else "approve", p_deny)
+```
+
+Install the tested versions listed in `environment.json`, including `kernels`; the pinned Hugging Face FlashAttention kernel loads automatically on supported CUDA systems. A compatible local FlashAttention 2 installation can also use `attn_implementation="flash_attention_2"`. Ordinary SDPA is suitable for shorter inputs but can materialize large masks for long sequences.
+
+Labels are `0 = approve`, `1 = deny`. Preserve the exact `### PROPOSED TOOL CALL`, `### USER REQUEST`, and `### AGENT HISTORY` sections. The model only consumes this serialized text; training rationales and metadata are excluded.
+
+## Scope and limitations
+
+This is a classification model, not a chat model. It approves routine authorized work and denies consequential unauthorized actions or actions following injected instructions. Its synthetic labels and held-out benchmark do not establish reliability on every real agent workload. A false approval remains possible; the model cannot inspect hidden file contents, resolve opaque executables, or determine a URL's runtime behavior from text alone. Long-context results are slice measurements on the supplied benchmark, not a universal guarantee. Evaluate on representative deployment traffic and treat uncertain decisions appropriately for the application.
+
+The base model and teacher were previously developed using their own validation histories; this run prevents new training/evaluation overlap but cannot independently establish that all historical model-development decisions were untouched by public benchmarks.
+'''
+
+def publish(model,selected_path,result,bench,final_logits):
+    stage=CKPT/'publish';stage.mkdir(exist_ok=True,parents=True)
+    for p in pathlib.Path(selected_path).iterdir():
+        if p.is_file(): shutil.copy2(p,stage/p.name)
+    audit=json.loads((DATA/'data_audit.json').read_text());plan=json.loads((ROOT/'training_plan.json').read_text())
+    (stage/'README.md').write_text(model_card(result,audit,plan))
+    atomic_json(stage/'eval_results.json',result)
+    atomic_json(stage/'data_audit.json',audit)
+    atomic_json(stage/'training_plan.json',plan)
+    sources=json.loads((ROOT/'sources.json').read_text())
+    atomic_json(stage/'source_revisions.json',{k:{a:v[a] for a in ['repo','revision','kind']} for k,v in sources.items()})
+    shutil.copy2(ROOT/'environment.json',stage/'environment.json')
+    for name in ['download.py','prepare.py','engine.py','run.py','publish.py','sft_check.py']:
+        (stage/'training').mkdir(exist_ok=True)
+        shutil.copy2(ROOT/name,stage/'training'/name)
+    shutil.copy2(ROOT/'gate.py',stage/'gate.py')
+    np.savez(stage/'benchmark_predictions.npz',logits=final_logits,labels=np.array(bench['labels']),source_row=np.array(bench['source_row']))
+    hashes={p.name:hashlib.sha256(p.read_bytes()).hexdigest() for p in stage.iterdir() if p.is_file()}
+    atomic_json(stage/'artifact_hashes.json',hashes)
+    api=HfApi()
+    owner=ROOT/'publish_ownership.json'
+    if not owner.exists():
+        try:
+            api.model_info(REPO)
+        except RepositoryNotFoundError:
+            api.create_repo(REPO,repo_type='model',private=True,exist_ok=False)
+            atomic_json(owner,{'repo':REPO,'created_by':'auto04-training','created_at':time.time()})
+        else: raise RuntimeError('Target repository already exists and is not owned by this run; refusing overwrite')
+    else: assert json.loads(owner.read_text())['repo']==REPO
+    commit=api.upload_folder(repo_id=REPO,repo_type='model',folder_path=str(stage),commit_message='Publish evaluated auto-0.4b-2 checkpoint and reproducibility artifacts')
+    revision=commit.oid
+    downloaded=snapshot_download(REPO,revision=revision)
+    # Verify the actual remote checkpoint after upload, before making it public.
+    reload=load_model(downloaded)
+    check=np.arange(len(bench))
+    observed=predict(reload,bench,check,'published-roundtrip')
+    reference=final_logits[check]
+    max_error=float(np.abs(observed-reference).max())
+    assert np.array_equal(observed.argmax(-1),reference.argmax(-1)), 'Remote roundtrip changed decisions'
+    assert max_error < 0.25, f'Remote roundtrip logits differ by {max_error}'
+    api.update_repo_settings(REPO,private=False)
+    remote=api.model_info(REPO,revision=revision)
+    assert not remote.private
+    record={'repo':REPO,'url':'https://huggingface.co/'+REPO,'revision':revision,'accuracy':result['benchmark_default']['overall']['accuracy'],
+            'roundtrip_examples':len(check),'roundtrip_max_abs_logit_difference':max_error,'results':str(CKPT/'final_evaluation/results.json')}
+    atomic_json(ROOT/'published.json',record)
+    event('published',**record)
+    del reload,model;gc.collect();torch.cuda.empty_cache()
+
+if __name__=='__main__': finalize()

training/run.py

@@ -0,0 +1,98 @@
+"""SFT -> long-context SFT -> teacher-logit cache -> KD -> long-context KD -> audited publication."""
+import fcntl, gc, importlib.metadata, json, pathlib, time, traceback
+import numpy as np
+import torch
+from datasets import load_from_disk
+from engine import ROOT, DATA, CKPT, LOG, SEED, atomic_json, event, baseline, load_model, predict, train_phase
+
+PLAN = {
+    'seed': SEED,
+    'distillation_policy': 'At user request, after long SFT freeze a checkpoint on validation and evaluate Approve-or-Deny. Report the result to the user before proceeding. Distill only if its benchmark accuracy does not strictly exceed the fresh auto-1b-bf16 benchmark accuracy; skip distillation if it does. A tie triggers distillation.',
+    'sft_short': {'epochs': 2, 'lr': 1.5e-5, 'max_length_inclusion': 4096},
+    'sft_long': {'epochs': 1, 'lr': 6e-6, 'short_replay_ratio': 1.0},
+    'kd_short': {'epochs': 1, 'lr': 5e-6, 'alpha': 0.5, 'temperature': 2.0},
+    'kd_long': {'epochs': 1, 'lr': 3e-6, 'alpha': 0.5, 'temperature': 2.0, 'short_replay_ratio': 1.0},
+    'optimizer': 'Full-parameter fused AdamW; fp32 master parameters/moments; bf16 autocast; betas=(0.9,0.95); eps=1e-8; weight_decay=0.01; grad_clip=1.0',
+    'schedule': '3% linear warmup then cosine to 10% of peak learning rate in each phase',
+    'batching': 'Length buckets; <=16384 padded tokens per short microbatch; single longer example; optimizer step at >=128 examples or >=131072 true tokens',
+    'long_context': 'Every retained row is used at its full tokenized length, with model capacity of 65536; no training input required truncation.',
+    'distillation': 'Teacher forward only; temperature-scaled KL plus hard-label CE; alpha reduced to 0.125 where teacher argmax disagrees with label',
+    'selection': 'Validation-only: 0.7 * overall balanced accuracy + 0.3 * balanced accuracy at >=16384 tokens; each phase best and last compared on the full 7824-row selection split',
+    'publish_gate': {'benchmark_accuracy_min': 0.95, 'benchmark_long_accuracy_min': 0.90,
+                     'false_approve_improvement_min': 0.015, 'false_deny_improvement_min': 0.025,
+                     'validation_audit_accuracy_min': 0.95, 'paired_accuracy_ci95_lower_min': 0.0},
+}
+
+def cache_teacher():
+    ds=load_from_disk(str(DATA/'train'))
+    cache=CKPT/'teacher_logits.npy'; manifest=CKPT/'teacher_logits_manifest.json'
+    sources=json.loads((ROOT/'sources.json').read_text())
+    import hashlib
+    identity={'teacher_revision':sources['teacher']['revision'],'student_tokenizer_revision':sources['student']['revision'],
+              'train_order_sha256':hashlib.sha256('\n'.join(ds['text_hash']).encode()).hexdigest(),'rows':len(ds)}
+    if manifest.exists(): assert json.loads(manifest.read_text()) == identity
+    else: atomic_json(manifest, identity)
+    if cache.exists(): arr=np.load(cache,mmap_mode='r+')
+    else:
+        arr=np.lib.format.open_memmap(cache,mode='w+',dtype=np.float32,shape=(len(ds),2));arr[:]=np.nan;arr.flush()
+    assert arr.shape==(len(ds),2)
+    remaining=np.flatnonzero(~np.isfinite(arr).all(axis=1))
+    if not len(remaining): return str(cache)
+    lengths=np.array(ds['length']); remaining=remaining[np.argsort(lengths[remaining],kind='stable')]
+    model=load_model(sources['teacher']['path'])
+    blocks=[];start=0;token_count=0
+    for pos,idx in enumerate(remaining):
+        token_count+=int(lengths[idx])
+        if token_count>=5_000_000 or pos-start+1>=10000:
+            blocks.append(remaining[start:pos+1]);start=pos+1;token_count=0
+    if start<len(remaining):blocks.append(remaining[start:])
+    for block in blocks:
+        values=predict(model,ds,block,'teacher-cache',token_budget=32768)
+        arr[block]=values;arr.flush()
+        done=int(np.isfinite(arr).all(axis=1).sum())
+        event('teacher_cache',done=done,total=len(ds))
+    assert np.isfinite(arr).all()
+    del model,arr;gc.collect();torch.cuda.empty_cache()
+    return str(cache)
+
+def run():
+    ROOT.mkdir(parents=True,exist_ok=True);CKPT.mkdir(parents=True,exist_ok=True);LOG.mkdir(parents=True,exist_ok=True)
+    atomic_json(ROOT/'training_plan.json',PLAN)
+    atomic_json(ROOT/'environment.json',{p:importlib.metadata.version(p) for p in ['torch','transformers','datasets','huggingface_hub','kernels','accelerate','numpy','scikit-learn']})
+    if (ROOT/'published.json').exists():
+        event('already_published',**json.loads((ROOT/'published.json').read_text()));return
+    if not (DATA/'data_audit.json').exists(): raise RuntimeError('Data preparation is incomplete')
+    if not json.loads((ROOT/'smoke_test.json').read_text())['passed']: raise RuntimeError('Runtime validation did not pass')
+    baseline()
+    sources=json.loads((ROOT/'sources.json').read_text())
+    ds=load_from_disk(str(DATA/'train'));lengths=np.array(ds['length'])
+    short=np.flatnonzero(lengths<=4096);long=np.flatnonzero(lengths>4096)
+    rng=np.random.default_rng(SEED)
+    replay=rng.choice(short,len(long),replace=False)
+    long_mix=np.concatenate([long,replay])
+    np.savez(CKPT/'phase_indices.npz',short=short,long=long,long_mix=long_mix)
+    p=train_phase('sft_short',sources['student']['path'],short,**{k:PLAN['sft_short'][k] for k in ['epochs','lr']})
+    p=train_phase('sft_long',p,long_mix,**{k:PLAN['sft_long'][k] for k in ['epochs','lr']})
+    from sft_check import compare_after_sft, require_result_reported
+    comparison=compare_after_sft()
+    require_result_reported()
+    p=comparison['selected_checkpoint']['path']
+    if not comparison['distillation_required']:
+        event('distillation_skipped',reason='Post-SFT benchmark accuracy exceeds the teacher',accuracy=comparison['benchmark']['overall']['accuracy'])
+        from publish import finalize
+        finalize(selected_sft=comparison['selected_checkpoint'])
+        return
+    teacher=cache_teacher()
+    p=train_phase('kd_short',p,short,teacher_logits=teacher,**{k:PLAN['kd_short'][k] for k in ['epochs','lr','alpha','temperature']})
+    p=train_phase('kd_long',p,long_mix,teacher_logits=teacher,**{k:PLAN['kd_long'][k] for k in ['epochs','lr','alpha','temperature']})
+    from publish import finalize
+    finalize()
+
+if __name__=='__main__':
+    lock=(ROOT/'pipeline.lock').open('a')
+    fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
+    try:
+        run()
+    except Exception as exc:
+        event('failed',exception=type(exc).__name__,message=str(exc),traceback=traceback.format_exc())
+        raise

training/sft_check.py

@@ -0,0 +1,56 @@
+"""User-requested benchmark after long SFT; report it before conditional distillation."""
+import gc, hashlib, json
+import numpy as np
+import torch
+from datasets import load_from_disk
+from engine import ROOT, DATA, CKPT, atomic_json, event, load_model, predict, report, selection_score
+
+def needs_distillation(student_accuracy, teacher_accuracy):
+    # A tie is not better. Both measurements use the same 3,000 rows and threshold.
+    return student_accuracy <= teacher_accuracy
+
+def compare_after_sft():
+    ready=ROOT/'post_sft_benchmark.json'
+    if ready.exists(): return json.loads(ready.read_text())
+    assert (CKPT/'sft_long/complete.json').exists(), 'Long SFT must finish first'
+    evaluated=CKPT/'post_sft_evaluation';evaluated.mkdir(parents=True,exist_ok=True)
+    val=load_from_disk(str(DATA/'validation'));bench=load_from_disk(str(DATA/'benchmark'))
+    selection=np.load(DATA/'validation_partitions.npz')['selection']
+    frozen_path=evaluated/'frozen_selection.json'
+    if frozen_path.exists(): chosen=json.loads(frozen_path.read_text())
+    else:
+        candidates=[]
+        for which in ['best','last']:
+            path=CKPT/'sft_long'/which
+            model=load_model(path)
+            logits=predict(model,val,selection,'post-sft-'+which+'-selection',evaluated/f'{which}-selection.npz')
+            r=report(val,selection,logits)
+            candidates.append({'path':str(path),'name':'sft_long-'+which,'score':selection_score(r),'distillation_optimizer_steps':0,'report':r})
+            del model;gc.collect();torch.cuda.empty_cache()
+        atomic_json(evaluated/'selection_candidates.json',candidates)
+        chosen=max(candidates,key=lambda x:(x['score'],-x['report']['overall']['nll']))
+        atomic_json(frozen_path,chosen)
+    model=load_model(chosen['path'])
+    logits=predict(model,bench,np.arange(len(bench)),'post-sft-benchmark',evaluated/'benchmark.npz')
+    r=report(bench,np.arange(len(bench)),logits)
+    teacher=json.loads((CKPT/'baseline/teacher.json').read_text())['benchmark']
+    original=json.loads((CKPT/'baseline/student.json').read_text())['benchmark']
+    required=needs_distillation(r['overall']['accuracy'],teacher['overall']['accuracy'])
+    result={'selected_checkpoint':chosen,'benchmark':r,'teacher_benchmark':teacher,'original_benchmark':original,
+            'distillation_required':required,
+            'decision_rule':'Distill only if post-long-SFT default-threshold benchmark accuracy does not strictly exceed the same-run auto-1b-bf16 benchmark accuracy. A tie triggers distillation.',
+            'benchmark_revision':json.loads((ROOT/'sources.json').read_text())['benchmark']['revision'],
+            'evaluation_note':'Checkpoint frozen using validation only before this benchmark. At user request, this benchmark determines whether to run distillation; it is therefore used for that training-procedure decision. Benchmark examples never become training or distillation targets.'}
+    atomic_json(ready,result)
+    event('post_sft_benchmark_ready',accuracy=r['overall']['accuracy'],teacher_accuracy=teacher['overall']['accuracy'],
+          distillation_required=required,report_path=str(ready))
+    del model;gc.collect();torch.cuda.empty_cache()
+    return result
+
+def require_result_reported():
+    result_file=ROOT/'post_sft_benchmark.json';ack_file=ROOT/'sft_result_reported.json'
+    digest=hashlib.sha256(result_file.read_bytes()).hexdigest()
+    if not ack_file.exists() or json.loads(ack_file.read_text()).get('benchmark_result_sha256')!=digest:
+        event('awaiting_sft_result_report',report_path=str(result_file),benchmark_result_sha256=digest,
+              note='User requested this result before any conditional distillation. The task heartbeat must report it, record acknowledgement, and resume the service.')
+        raise SystemExit(76)

training_plan.json

@@ -0,0 +1,41 @@
+{
+  "seed": 20260908,
+  "distillation_policy": "At user request, after long SFT freeze a checkpoint on validation and evaluate Approve-or-Deny. Report the result to the user before proceeding. Distill only if its benchmark accuracy does not strictly exceed the fresh auto-1b-bf16 benchmark accuracy; skip distillation if it does. A tie triggers distillation.",
+  "sft_short": {
+    "epochs": 2,
+    "lr": 1.5e-05,
+    "max_length_inclusion": 4096
+  },
+  "sft_long": {
+    "epochs": 1,
+    "lr": 6e-06,
+    "short_replay_ratio": 1.0
+  },
+  "kd_short": {
+    "epochs": 1,
+    "lr": 5e-06,
+    "alpha": 0.5,
+    "temperature": 2.0
+  },
+  "kd_long": {
+    "epochs": 1,
+    "lr": 3e-06,
+    "alpha": 0.5,
+    "temperature": 2.0,
+    "short_replay_ratio": 1.0
+  },
+  "optimizer": "Full-parameter fused AdamW; fp32 master parameters/moments; bf16 autocast; betas=(0.9,0.95); eps=1e-8; weight_decay=0.01; grad_clip=1.0",
+  "schedule": "3% linear warmup then cosine to 10% of peak learning rate in each phase",
+  "batching": "Length buckets; <=16384 padded tokens per short microbatch; single longer example; optimizer step at >=128 examples or >=131072 true tokens",
+  "long_context": "Every retained row is used at its full tokenized length, with model capacity of 65536; no training input required truncation.",
+  "distillation": "Teacher forward only; temperature-scaled KL plus hard-label CE; alpha reduced to 0.125 where teacher argmax disagrees with label",
+  "selection": "Validation-only: 0.7 * overall balanced accuracy + 0.3 * balanced accuracy at >=16384 tokens; each phase best and last compared on the full 7824-row selection split",
+  "publish_gate": {
+    "benchmark_accuracy_min": 0.95,
+    "benchmark_long_accuracy_min": 0.9,
+    "false_approve_improvement_min": 0.015,
+    "false_deny_improvement_min": 0.025,
+    "validation_audit_accuracy_min": 0.95,
+    "paired_accuracy_ci95_lower_min": 0.0
+  }
+}