Kev-4B round 10: skills delta (hard-v1 + devtools-v1) on top of round 8

README.md

@@ -30,42 +30,96 @@ metrics:
 model-index:
   - name: Kev-4B
     results:
-      - task: { type: text-classification, name: typed decision, real documents, locked test }
-        dataset: { type: mixed, name: "documents-v1 test (936 questions on CFPB complaint narratives; read once)" }
+      - task: { type: text-classification, name: typed decision, skill records, locked test }
+        dataset: { type: mixed, name: "hard-v1 test (1,088 questions; programmatic labels, held-out templates; read once)" }
         metrics:
-          - { type: accuracy, value: 0.904 }
-          - { type: brier_score, value: 0.156 }
+          - { type: accuracy, value: 0.803 }
+          - { type: brier_score, value: 0.278 }
+      - task: { type: text-classification, name: typed decision, developer tooling, locked test }
+        dataset: { type: mixed, name: "devtools-v1 test (1,071 questions; six public developer-tooling sources; read once)" }
+        metrics:
+          - { type: accuracy, value: 0.756 }
+          - { type: brier_score, value: 0.342 }
       - task: { type: text-classification, name: typed decision (choice / noul / score) }
-        dataset: { type: mixed, name: "decision-v7 development (1,204 records; ten trained public sources + programmatic policy data)" }
+        dataset: { type: mixed, name: "decision-v7 development (1,264 questions; ten trained public sources + programmatic policy data)" }
         metrics:
-          - { type: accuracy, value: 0.872 }
-          - { type: expected_calibration_error, value: 0.075, name: "ECE, raw probabilities" }
+          - { type: accuracy, value: 0.873 }
+          - { type: expected_calibration_error, value: 0.013, name: "ECE, as served" }
       - task: { type: text-classification, name: typed decision, out-of-domain }
-        dataset: { type: mixed, name: "transfer-v4 development (764 records; six never-trained sources + held-out policy structures)" }
+        dataset: { type: mixed, name: "transfer-v4 development (656 questions; six never-trained sources + held-out policy structures)" }
         metrics:
-          - { type: accuracy, value: 0.797 }
-          - { type: brier_score, value: 0.299 }
+          - { type: accuracy, value: 0.817 }
+          - { type: brier_score, value: 0.243 }
       - task: { type: text-classification, name: typed decision, out-of-domain, locked test }
         dataset: { type: mixed, name: "transfer-v4 test (read once)" }
         metrics:
-          - { type: accuracy, value: 0.837 }
-          - { type: brier_score, value: 0.255 }
+          - { type: accuracy, value: 0.838 }
+          - { type: brier_score, value: 0.224 }
 ---
 
 # Kev-4B
 
 Kev-4B is a **decision model**: one document (the *state*) and a set of typed questions in, a probability distribution per question out, in one forward pass. No text generation. It is a LoRA adapter (r=16, 33.8M trainable parameters) plus a pointer head on `Qwen/Qwen3.5-4B-Base` (revision `1001bb4d`), serving TypeSafe's public `/v1/systemone` contract.
 
-**This version (2026-09-24): real-document delta.** The previous Kev-4B plus one epoch (lr 2e-5) on `documents-v1` train: 5,219 real US consumer-finance complaint narratives (CFPB, 2015-2024, up to ~7k tokens) with 7,488 questions (which product, which main issue), labels kept only where two open-weight teachers agreed with the consumer's own filing, mixed with 2,000 replayed `decision-v7` records. On complaint narratives it has never seen, accuracy goes from 0.804 to **0.904** on the locked test (+9.9 pp [+7.5, +12.4], 936 questions) and from 0.811 to **0.891** on a private held-out set (`documents-v2`, 953 questions); on the development split it scores 0.895 against Jev's 0.868. Everything else is unchanged within noise: locked out-of-domain test 0.835 (previous 0.837), served Brier 0.233 (0.232).
+**This version (2026-09-24, second update): skills delta.** The round-8 Kev-4B (below) plus one epoch (lr 2e-5) on 11,320 new training records mixed with 4,000 replayed `decision-v7` records. 6,000 come from `hard-v1`, our programmatically labelled suite of the skills Kev was worst at: long policy documents with exceptions and sublimits, trade-offs under stated priorities, probability and expected value, multi-hop over several facts, dates and arithmetic, judging a proposed answer, and abstaining when a fact is missing. 5,320 come from `devtools-v1`, developer-tooling decisions from four licence-checked public datasets (CodeReviewer, CommitPackFT, FlakeFlagger, Aegis). On the held-out test splits, read once, accuracy goes from 0.540 to **0.803** on `hard-v1` (+26.3 pp [+23.3, +29.5], 1,088 questions) and from 0.623 to **0.756** on `devtools-v1` (+13.4 pp [+10.1, +16.1], 1,071 questions). On the development splits it scores 0.786 on hard-v1 against Jev's 0.777 and 0.739 on devtools-v1 against Jev's 0.713. The locked out-of-domain test is unchanged within noise (0.835 → 0.838, +0.3 pp [−1.8, +2.3]; served Brier 0.233 → 0.224). On JevBench's public items, which no training or selection step saw, the hard tier goes from 0.450 to **0.541**.
 
-**Read this before relying on the documents numbers.** The gain is measured **in distribution**: training and every documents suite share one source (CFPB complaints) and the same two question templates. It shows Kev-4B learns real long documents from a few thousand labelled examples; it does not show the same gain on other kinds of documents. Evaluation labels are AI-adjudicated (a unanimous three-model judge panel, or two agreeing adjudications) and human spot-checked (47/50 and 50/50).
+**Read this before relying on the hard-v1 and devtools-v1 numbers.**
+
+- **Both gains are measured in distribution.** `hard-v1` is generated: every label is computed by its family's solver, and the splits hold out *templates* (0-3 train, 4 development, 5 test) of the same seven generators. A held-out template is a new wording of a skill the model was trained on, not a new skill. JevBench's public hard tier is the out-of-distribution check, and there the gain is about a third as large (+9.0 pp, below).
+- **devtools-v1 labels are the public datasets' own, not adjudicated for this suite.** Some are human (CodeReviewer: whether a reviewer commented on the hunk; Aegis: human safety labels), some heuristic or by construction (CommitPackFT: the commit type is the first verb of the subject; FlakeFlagger: the test both passed and failed over reruns; When2Call: built by NVIDIA's pipeline). Before this delta, every model we scored was near chance on two of the binary sources, Jev included: on development, CodeReviewer 0.473 to 0.553 and FlakeFlagger 0.500 to 0.520 across Kev-0.8B, Kev-4B, Kev-9B, Kev-27B and Jev. This version reaches 0.633 and 0.693 on them after training on the same sources, which may be the labelling proxy being learned rather than the decision. When2Call and the prompt-injection source are never trained on: When2Call rises 0.573 → 0.660, prompt injection stays at 0.753 (Jev 0.893).
+- **Two reads went down.** The locked in-distribution test moved 0.875 → 0.865, and TypeSafe's 89 answered rows moved 0.843 → 0.798. That is 4 questions, but the paired interval (−4.5 pp [−9.3, −1.0]) excludes zero. TypeSafe is too small to gate on, so the rule only counts it inside the pooled external guard. Real documents are unchanged (development 0.895 → 0.891, −0.3 pp [−1.5, +0.9]).
 
-- Hub: `jaredpalmer/kev-4b` (this repo; trial `r8-small/00-trial-0`; the registration and every read are in `PLAN.md` round 8 on the `research/overnight-r6` branch). The previous version is at tag `night2-du-release`; the pre-delta v7 checkpoint at `v7-base`; the Qwen3 generation at `qwen3` ([its card](kev-4b-qwen3.md)).
-- Code, suites, every trial with hashes and paired bootstraps: [github.com/jaredpalmer/kev](https://github.com/jaredpalmer/kev). The numbers below are in `runs/release/kev-4b-r8.json`.
+- Hub: `jaredpalmer/kev-4b` (this repo; trial `r10-skills/00-trial-0`; the registration and every read are in `PLAN.md` round 10 on the `research/overnight-r6` branch). The previous (round-8) version is at tag `r8-documents-release`; the `night2-du` version at `night2-du-release`; the pre-delta v7 checkpoint at `v7-base`; the Qwen3 generation at `qwen3` ([its card](kev-4b-qwen3.md)).
+- Code, suites, every trial with hashes and paired bootstraps: [github.com/jaredpalmer/kev](https://github.com/jaredpalmer/kev). The numbers below are in `runs/release/kev-4b-r10.json`; JevBench in `runs/jevbench-public/kev-4b-r10/`.
 
 ## Results (as served: each checkpoint at its own fitted temperature)
 
-| | **Kev-4B (this version, T = 2.96)** | previous Kev-4B (T = 2.14) | Jev |
+| | **Kev-4B (this version, T = 2.41)** | round-8 Kev-4B (T = 2.96) | Jev |
+|---|---|---|---|
+| **hard-v1**, test (1,088 questions, read once) | **0.803** | 0.540 | – |
+| hard-v1, development (1,083) | **0.786** | 0.503 | 0.777 |
+| hard-v1 ECE, test / development | 0.084 / 0.095 | 0.112 / 0.137 | – / 0.035 |
+| **devtools-v1**, test (1,071, read once) | **0.756** | 0.623 | – |
+| devtools-v1, development (1,072) | **0.739** | 0.605 | 0.713 |
+| real documents, development (`documents-v1`, 920) | 0.891 | 0.895 | 0.868 |
+| in-distribution accuracy (decision-v7 dev, 1,264 questions) | 0.873 | 0.873 | 0.845 |
+| out-of-domain accuracy (transfer-v4 dev, 656) | 0.817 | 0.802 | 0.857 |
+| out-of-domain Brier / ECE | 0.243 / 0.042 | 0.265 / 0.043 | 0.211 / 0.049 |
+| confident errors out of domain (p ≥ 0.9 and wrong) | 0.9% | 2.9% | 3.7% |
+| coverage at ≤ 5% error | 0.620 | 0.552 | 0.70 |
+| held-out policy structures, both siblings correct | 0.812 | 0.781 | 0.86 |
+| unknowable items answered at ≥ 0.9 (transfer-v9) | 0.00 | 0.00 | 0.09 |
+| MMLU-Pro (transfer-v9 dev, 10-way) | 0.565 | 0.515 | 0.840 |
+| **locked test**, out-of-domain accuracy / Brier | **0.838 / 0.224** | 0.835 / 0.233 | – |
+| **locked test**, in-distribution accuracy | 0.865 | 0.875 | – |
+| SemIf (144 authored decisions) | 0.889 | 0.882 | – |
+| scienthoon (873 support tickets) | 0.723 | 0.723 | – |
+| WANLI-v2 (1,002 NLI pairs) | 0.693 | 0.691 | – |
+| TypeSafe (89 answered rows) | 0.798 | 0.843 | ��� |
+| JevBench public items, all 231 (report only) | 0.758 | 0.714 | – |
+| JevBench public, hard tier (111): accuracy / ECE | 0.541 / 0.112 | 0.450 / 0.263 | – |
+
+Jev's devtools-v1 figure is over all 1,074 development questions. Kev's rows drop the one CodeReviewer id that the builder reused for two different records (2 questions; `PLAN.md` round-10 amendment), because paired bootstraps need unique ids.
+
+Paired against the round-8 version (record-clustered bootstrap, 95 %): hard-v1 development +28.3 pp [+25.1, +31.4], test +26.3 [+23.3, +29.5]; devtools-v1 development +13.3 [+11.3, +15.4], test +13.4 [+10.1, +16.1]; the two tests pooled +19.9 [+17.8, +21.8]; documents development −0.3 [−1.5, +0.9]; SemIf +0.7 [−2.8, +4.2]; scienthoon 0.0 [−1.4, +1.5]; WANLI-v2 +0.2 [−1.8, +2.2]; TypeSafe −4.5 [−9.3, −1.0]; locked out-of-domain test +0.3 [−1.8, +2.3].
+
+**How the release was decided.** By a rule registered before the training data was built (`PLAN.md` round 10, `research/overnight-r6` branch). The primary criterion is the pooled hard-v1 + devtools-v1 development accuracy, with a lower bound above zero; this arm scored +20.8 pp [+18.8, +22.8]. Guards on short states, documents, WANLI-v2, scienthoon, the pooled externals and unknowable confidence are each sized to what the suite can resolve (short states +1.5 pp [−0.3, +3.5], pooled externals 0.0 [−1.2, +1.1]). Hard-set ECE may be no worse than the parent's plus 0.01. Then come one read of the two untouched test splits (pooled lower bound above zero) and one locked read. Three arms were trained: both sets, hard-v1 only, and devtools-v1 only. The two single-set arms failed external guards; this arm is the only one that passed.
+
+**JevBench public items, report only.** `runs/jevbench-public/kev-4b-r10` holds JevBench's unchanged harness run against this checkpoint, served by the `kev-deploy` template. Across all 231 public items, accuracy goes from 0.714 (round-8 version) to 0.758. On the hard tier it goes from 0.450 to 0.541: paired over the 111 hard items that is +9.0 pp [+2.7, +15.3], with 12 items newly right and 2 newly wrong (exact McNemar p = 0.013). Hard-tier ECE falls 0.263 → 0.112. No JevBench item was used for training or selection. `evals/hard-v1/overlap.json` checks all 7,400 hard-v1 records against the 231 public items (8-gram Jaccard, threshold 0.2) and finds none above the threshold. JevBench's sealed half has not been read.
+
+**Calibration.** This delta softened the raw logits (fitted temperature 2.96 → 2.41; raw out-of-domain Brier 0.269 on development, 0.242 on the locked test), the reverse of round 8. As served, out-of-domain Brier improved from 0.265 to 0.243 on development and confident errors from 2.9% to 0.9%. `KEV_TEMPERATURE=1.0` gives the raw values.
+
+## Previous version: round-8 real-document delta (2026-09-24) at tag `r8-documents-release`
+
+**Real-document delta.** The `night2-du` Kev-4B plus one epoch (lr 2e-5) on `documents-v1` train: 5,219 real US consumer-finance complaint narratives (CFPB, 2015-2024, up to ~7k tokens) with 7,488 questions (which product, which main issue), labels kept only where two open-weight teachers agreed with the consumer's own filing, mixed with 2,000 replayed `decision-v7` records. On complaint narratives it has never seen, accuracy goes from 0.804 to **0.904** on the locked test (+9.9 pp [+7.5, +12.4], 936 questions) and from 0.811 to **0.891** on a private held-out set (`documents-v2`, 953 questions); on the development split it scores 0.895 against Jev's 0.868. Everything else is unchanged within noise: locked out-of-domain test 0.835 (previous 0.837), served Brier 0.233 (0.232).
+
+**Read this before relying on the documents numbers.** The gain is measured **in distribution**: training and every documents suite share one source (CFPB complaints) and the same two question templates. It shows Kev-4B learns real long documents from a few thousand labelled examples; it does not show the same gain on other kinds of documents. Evaluation labels are AI-adjudicated (a unanimous three-model judge panel, or two agreeing adjudications) and human spot-checked (47/50 and 50/50).
+
+- Trial `r8-small/00-trial-0` (Hub revision `957b91e7`); the registration and every read are in `PLAN.md` round 8 on the `research/overnight-r6` branch. The numbers below are in `runs/release/kev-4b-r8.json`.
+
+### Results (as served: each checkpoint at its own fitted temperature)
+
+| | **round-8 Kev-4B (T = 2.96)** | `night2-du` Kev-4B (T = 2.14) | Jev |
 |---|---|---|---|
 | **real documents**, locked test (`documents-v1`, 936 questions) | **0.904** | 0.804 | – |
 | real documents, private held-out (`documents-v2`, 953) | **0.891** | 0.811 | – |
@@ -86,11 +140,11 @@ Kev-4B is a **decision model**: one document (the *state*) and a set of typed qu
 | WANLI-v2 (1,002 NLI pairs) | 0.691 | 0.699 | – |
 | TypeSafe (89 answered rows) | 0.843 | 0.843 | – |
 
-Paired against the previous version (record-clustered bootstrap, 95 %): documents dev +8.4 pp [+6.0, +10.6], locked test +9.9 [+7.5, +12.4], private held-out +8.0 [+5.6, +10.3]; scienthoon +2.6 [+0.8, +4.4]; SemIf −0.7 [−2.8, +1.4]; WANLI-v2 −0.8 [−2.0, +0.4]; TypeSafe identical on all 89 rows. The release was decided by a rule registered before any read (`PLAN.md` round 8, `research/overnight-r6` branch): documents development lower bound > 0, short-state and external guards sized to what each suite can resolve, then one read of the untouched documents test and one locked read. A first seed (round 7) gave the same documents gain and failed only per-suite lower bounds on the two smallest suites; it was not released.
+Paired against the `night2-du` version (record-clustered bootstrap, 95 %): documents dev +8.4 pp [+6.0, +10.6], locked test +9.9 [+7.5, +12.4], private held-out +8.0 [+5.6, +10.3]; scienthoon +2.6 [+0.8, +4.4]; SemIf −0.7 [−2.8, +1.4]; WANLI-v2 −0.8 [−2.0, +0.4]; TypeSafe identical on all 89 rows. The release was decided by a rule registered before any read (`PLAN.md` round 8, `research/overnight-r6` branch): documents development lower bound > 0, short-state and external guards sized to what each suite can resolve, then one read of the untouched documents test and one locked read. A first seed (round 7) gave the same documents gain and failed only per-suite lower bounds on the two smallest suites; it was not released.
 
 **Calibration.** The delta sharpened the raw logits (fitted temperature 2.14 → 2.96; raw out-of-domain Brier 0.327, raw locked Brier 0.278). As served, calibration is unchanged. `KEV_TEMPERATURE=1.0` gives the raw values.
 
-## Previous version: `night2-du` (2026-09-21), kept at tag `night2-du-release`
+## Earlier version: `night2-du` (2026-09-21), kept at tag `night2-du-release`
 
 **At its release, the recommended Kev.** The best accuracy per byte: out of domain 0.797 on the development partition and **0.837 on the locked test**, Brier 0.255 on the test, held-out rule pairs 0.77–0.78. This checkpoint is the `decision-v7` recipe (trial `q35-4b-s23/00-trial-0`, seed 2, selected on development accuracy) followed by a 9-minute **delta fine-tune** (`--init_from`, lr 2e-5, one epoch) on 1,425 additional records — date-bearing policy cases rendered with explicit day counts, and evidence-free cases with uniform targets — mixed with 2,000 replayed training records. Against the pre-delta checkpoint on the locked test: +1.0 pp [−0.1, +2.1], Brier 0.266 → 0.255, `deadline` 0.65 → 0.75.
 

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2c99077b58cd4a604304b743b7db5391187ef1d58e575f90c554c744aa23d65
+oid sha256:90e817356246e7f18bfa7ca3d31794cd4fbeb3332a66a84cb51d9ceae925f2b2
 size 129924032

head.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9a3051c3676f0be29d885935b6f50c287f7acf04e3cb29a6fed82ca3e5540b7e
-size 5249727
+oid sha256:dd633435998ecc751ac538717a3742e32149500fabf7d7276287dbf0693f347c
+size 5249791

provenance.json

@@ -1,7 +1,7 @@
 {
   "config": {
     "epochs": 1,
-    "seed": 2,
+    "seed": 1,
     "lr": 2e-05,
     "lora": 16,
     "accum": 4,
@@ -23,57 +23,58 @@
     "focal_gamma": 0.0,
     "dtype": "bf16",
     "checkpointing": 1,
-    "replay": 2000,
     "max_state": 7552,
-    "data": "evals/documents-v1/train.jsonl",
     "base": "Qwen/Qwen3.5-4B-Base",
     "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
-    "init_from": "jaredpalmer/kev-4b"
+    "init_from": "jaredpalmer/kev-4b@957b91e762e883935830246eeb02381f9d2694b6",
+    "data": "evals/round10/skills/train.jsonl",
+    "replay": 4000
   },
-  "config_sha256": "6fb0132214a793f590cce025ce669b85433bab47a63522c369c226f0fb34ff3c",
+  "config_sha256": "08aa7634eef0409340edd9df72b4b347bdda02e87b25c920e205dc727911525d",
   "suite_sha256": "a8f50e481b7d90b97da049e0ff6a01cee2f1ed204aed61a8265af0edbb5514d2",
   "source_hashes": {
     "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
     "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
     "kev/api.py": "7bffacfb762c626b8bc2f670f350295af5ccb0883dbe90239c7d2f8e5ef56582",
     "kev/autoresearch.py": "f7d7fabb9c0df7065bee3fec4aa4bee7028c5d1565847aefe2d74e3d7d40ed8a",
-    "kev/benchmark.py": "9c442efd8b6b852c574a646192c61ee91afa1306e3b2978bb6391ae1b91a3fa0",
+    "kev/benchmark.py": "10a892d76cb4c200e94d0a68ee74054aa1870d7f484c0530a531cacfa75032ed",
     "kev/calibrate.py": "c1eff4744fabd349e8abca86777a7aa0cbea44904c8223d3b61fca3d43741519",
-    "kev/checkpoint.py": "7cd4fbedfcfd11c2997490df2bd88eaa12af60e5ad60da2e5390e6df1ac1f626",
+    "kev/checkpoint.py": "f3edb4d159c0aa12d7006fc599fe6ebc3f2c288b3b69890c289f7dac451dd03d",
     "kev/compare.py": "3606dbf98bb305d66420158498cb837304d2edb8dab01cdaf2ec42964bef3435",
     "kev/composition.py": "f335ed17e18e0a544893db5e22b9a059e6ce1b2e14dbb863ac7d7bbf8f3e0536",
     "kev/contrastive.py": "cbb979aa5d40265ad0e64695f94b281d91751fa405811ddfa8212fede111edcf",
-    "kev/data.py": "ef1be396e7218ba5c94f0f67de2ceff23a39656b57199cb3aa3192cde53e8320",
+    "kev/cuda_graphs.py": "12f1953fea8f77c3c9dff29468202fbb37545145f93d3618194565a89949fd55",
+    "kev/data.py": "e77a2fffeb8ee7e05b118893be8e38b8358aef97c3ca2e143cdce208201b92f9",
     "kev/device.py": "d1677fd98ec0979c7284546306e34e0d09298ca4042fc39eb2b32a74f4e975c4",
     "kev/evaluate.py": "999264f2837dcfbf2ec93601aa4745e701698ab850a43ad89324672a6604965c",
     "kev/experiment.py": "644adedf43fbd10566830fa6094e16dd001b8cad29d9b9da82f0e0f245ca06b1",
     "kev/jev.py": "acd4cc3f1844e438cc83a8d409c15ef78a5ef64d39ea10f583d75a6c7646b243",
     "kev/metrics.py": "dba8d90550999edcd642581fb4da0367bd4b39d08471815184fa732d11e137b9",
     "kev/mlx_model.py": "f582428796faf6bf962b772a69a6227ac3259ccbd3215c0c8b55ce28dbbd0909",
-    "kev/model.py": "387f281bc0b72da1620a8d2fcf508996fc30b9d8e342211ecd67c6d930f9cbaa",
+    "kev/model.py": "2634ffe7747d69473fb6596942c248e5df2586df3610bd1adb47e7e9acd99f96",
     "kev/plot.py": "0874bcff2885d8155a1cceade0de8a275a7163d3c3fd6aa0c294e8ccf5ff7e02",
-    "kev/predictors.py": "b2a66dd9f0f0a8026bc94603383bacfdd622e75d1c935173997175606626a9b3",
+    "kev/predictors.py": "4d02906c285d78bcbd91908f4fc86577ccc95c320d704817f6444aabbc4fcd4f",
     "kev/publish.py": "8abc9bcf4a01365697b05cf3c5ad0013462bd92d005f5616954e40ab1100c7aa",
-    "kev/serve.py": "c210b1f4598e64b49c399f73603ed372f4314c165ceda50e1340c0466ec1c735",
+    "kev/serve.py": "57d379bbcdeb3e4d4c3fdde5471e4dddeaf7710b8ed61ecad3466f3b67e6c610",
     "kev/study_v3.py": "7891150aa623e4479c7c789d4dc4662f186132b37b7bac1e1f072b9e08ee13e3",
-    "kev/suite.py": "d524663fe2656d873c548a6216a2ddfebaf7299650f07ba31ae24e4c453398f1",
+    "kev/suite.py": "32b2882d1ea0fe5d03ba4d676a6180d580545f157e36fa715e7e89169ec99130",
     "kev/train.py": "68428ed43b3e362e52d4650dc5296ab88767061f7998a525715e1b306eeef474",
     "kev/transfer_v9.py": "0902409742151250a28af1fd8f42258b70fcc161f3deed3ee35fe3f473f3c763",
-    "modal_app.py": "28318ef72dbeba7855f3c6b05339ee8c0f2e57562d21ede72352fba98d0e6ae3",
+    "modal_app.py": "7c6bc859f7d3e7011539a8f5e9e90390288cc72b5d80f911c8841c068f4d1136",
     "pyproject.toml": "7c17fbe9efcadda3eb488b59adf1b6db5938cd33af4d3bd389ce4359b73fcc7d",
     "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
   },
-  "git_commit": "ea271bf9dd0f2a280b1ef096a7d237388484620c",
+  "git_commit": "6d02f5d066cd34958dfd15ffa5d2f6f0f4c21a63",
   "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
   "torch": "2.8.0+cu128",
   "device": "cuda",
   "gpu": "NVIDIA H200",
   "legacy_checkpoint": false,
   "measured_checkpoint": {
-    "requested": "/runs/r8-small/00-trial-0/checkpoint",
-    "resolved": "/runs/r8-small/00-trial-0/checkpoint",
-    "head_sha256": "19fdcdfb65380b51e90458edeb7e8dd7554532f357379c4cda44152cbbcd4195",
-    "adapter_sha256": "c2c99077b58cd4a604304b743b7db5391187ef1d58e575f90c554c744aa23d65",
+    "requested": "/runs/r10-skills/00-trial-0/checkpoint",
+    "resolved": "/runs/r10-skills/00-trial-0/checkpoint",
+    "head_sha256": "979aaa60135274973a6dc57fe24dfc4f1ce88e65df46c73d86b2047c94d897df",
+    "adapter_sha256": "90e817356246e7f18bfa7ca3d31794cd4fbeb3332a66a84cb51d9ceae925f2b2",
     "inference_temperature": 1.0
   }
 }

train.log

@@ -1,97 +1,198 @@
 Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
-delta: warm start from /__modal/volumes/vo-kEMu8BkBAIrorAQI6V8f2D/hub/models--jaredpalmer--kev-4b/snapshots/485ace8703592fcf405488b262449990824cfed1: 496 adapter tensors and the pointer head loaded
+delta: warm start from /__modal/volumes/vo-kEMu8BkBAIrorAQI6V8f2D/hub/models--jaredpalmer--kev-4b/snapshots/957b91e762e883935830246eeb02381f9d2694b6: 496 adapter tensors and the pointer head loaded
 device=cuda trainable params=33.8M
-replay: 2000 of 12576 suite training records mixed with 5219 from evals/documents-v1/train.jsonl
-7219 training requests (holdout=[]), questions by type {'choice': 8557, 'score': 580, 'noul': 837}
+replay: 4000 of 12576 suite training records mixed with 11320 from evals/round10/skills/train.jsonl
+15320 training requests (holdout=[]), questions by type {'choice': 10917, 'noul': 8981, 'score': 1377}
 [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
-ep0 step 10/903 loss 0.574 kl 0.000 anchor 0.000 2.168s/rec
-ep0 step 20/903 loss 0.461 kl 0.000 anchor 0.000 1.289s/rec
-ep0 step 30/903 loss 0.478 kl 0.000 anchor 0.000 1.091s/rec
-ep0 step 40/903 loss 0.427 kl 0.000 anchor 0.000 0.865s/rec
-ep0 step 50/903 loss 0.417 kl 0.000 anchor 0.000 0.726s/rec
-ep0 step 60/903 loss 0.334 kl 0.000 anchor 0.000 0.640s/rec
-ep0 step 70/903 loss 0.464 kl 0.000 anchor 0.000 0.570s/rec
-ep0 step 80/903 loss 0.365 kl 0.000 anchor 0.000 0.550s/rec
-ep0 step 90/903 loss 0.221 kl 0.000 anchor 0.000 0.510s/rec
-ep0 step 100/903 loss 0.299 kl 0.000 anchor 0.000 0.480s/rec
-ep0 step 110/903 loss 0.673 kl 0.000 anchor 0.000 0.453s/rec
-ep0 step 120/903 loss 0.232 kl 0.000 anchor 0.000 0.435s/rec
-ep0 step 130/903 loss 0.207 kl 0.000 anchor 0.000 0.417s/rec
-ep0 step 140/903 loss 0.257 kl 0.000 anchor 0.000 0.406s/rec
-ep0 step 150/903 loss 0.363 kl 0.000 anchor 0.000 0.393s/rec
-ep0 step 160/903 loss 0.418 kl 0.000 anchor 0.000 0.381s/rec
-ep0 step 170/903 loss 0.426 kl 0.000 anchor 0.000 0.371s/rec
-ep0 step 180/903 loss 0.244 kl 0.000 anchor 0.000 0.365s/rec
-ep0 step 190/903 loss 0.142 kl 0.000 anchor 0.000 0.357s/rec
-ep0 step 200/903 loss 0.196 kl 0.000 anchor 0.000 0.349s/rec
-ep0 step 210/903 loss 0.224 kl 0.000 anchor 0.000 0.343s/rec
-ep0 step 220/903 loss 0.290 kl 0.000 anchor 0.000 0.337s/rec
-ep0 step 230/903 loss 0.248 kl 0.000 anchor 0.000 0.331s/rec
-ep0 step 240/903 loss 0.176 kl 0.000 anchor 0.000 0.325s/rec
-ep0 step 250/903 loss 0.135 kl 0.000 anchor 0.000 0.321s/rec
-ep0 step 260/903 loss 0.328 kl 0.000 anchor 0.000 0.317s/rec
-ep0 step 270/903 loss 0.398 kl 0.000 anchor 0.000 0.314s/rec
-ep0 step 280/903 loss 0.311 kl 0.000 anchor 0.000 0.310s/rec
-ep0 step 290/903 loss 0.075 kl 0.000 anchor 0.000 0.308s/rec
-ep0 step 300/903 loss 0.155 kl 0.000 anchor 0.000 0.304s/rec
-ep0 step 310/903 loss 0.367 kl 0.000 anchor 0.000 0.303s/rec
-ep0 step 320/903 loss 0.358 kl 0.000 anchor 0.000 0.300s/rec
-ep0 step 330/903 loss 0.253 kl 0.000 anchor 0.000 0.298s/rec
-ep0 step 340/903 loss 0.400 kl 0.000 anchor 0.000 0.295s/rec
-ep0 step 350/903 loss 0.087 kl 0.000 anchor 0.000 0.293s/rec
-ep0 step 360/903 loss 0.247 kl 0.000 anchor 0.000 0.290s/rec
-ep0 step 370/903 loss 0.225 kl 0.000 anchor 0.000 0.289s/rec
-ep0 step 380/903 loss 0.171 kl 0.000 anchor 0.000 0.286s/rec
-ep0 step 390/903 loss 0.402 kl 0.000 anchor 0.000 0.284s/rec
-ep0 step 400/903 loss 0.151 kl 0.000 anchor 0.000 0.283s/rec
-ep0 step 410/903 loss 0.175 kl 0.000 anchor 0.000 0.281s/rec
-ep0 step 420/903 loss 0.200 kl 0.000 anchor 0.000 0.278s/rec
-ep0 step 430/903 loss 0.460 kl 0.000 anchor 0.000 0.276s/rec
-ep0 step 440/903 loss 0.176 kl 0.000 anchor 0.000 0.274s/rec
-ep0 step 450/903 loss 0.124 kl 0.000 anchor 0.000 0.273s/rec
-ep0 step 460/903 loss 0.139 kl 0.000 anchor 0.000 0.272s/rec
-ep0 step 470/903 loss 0.285 kl 0.000 anchor 0.000 0.271s/rec
-ep0 step 480/903 loss 0.172 kl 0.000 anchor 0.000 0.270s/rec
-ep0 step 490/903 loss 0.296 kl 0.000 anchor 0.000 0.269s/rec
-ep0 step 500/903 loss 0.254 kl 0.000 anchor 0.000 0.268s/rec
-ep0 step 510/903 loss 0.147 kl 0.000 anchor 0.000 0.266s/rec
-ep0 step 520/903 loss 0.168 kl 0.000 anchor 0.000 0.266s/rec
-ep0 step 530/903 loss 0.166 kl 0.000 anchor 0.000 0.265s/rec
-ep0 step 540/903 loss 0.222 kl 0.000 anchor 0.000 0.264s/rec
-ep0 step 550/903 loss 0.178 kl 0.000 anchor 0.000 0.267s/rec
-ep0 step 560/903 loss 0.183 kl 0.000 anchor 0.000 0.266s/rec
-ep0 step 570/903 loss 0.383 kl 0.000 anchor 0.000 0.266s/rec
-ep0 step 580/903 loss 0.316 kl 0.000 anchor 0.000 0.265s/rec
-ep0 step 590/903 loss 0.082 kl 0.000 anchor 0.000 0.264s/rec
-ep0 step 600/903 loss 0.176 kl 0.000 anchor 0.000 0.263s/rec
-ep0 step 610/903 loss 0.064 kl 0.000 anchor 0.000 0.263s/rec
-ep0 step 620/903 loss 0.108 kl 0.000 anchor 0.000 0.263s/rec
-ep0 step 630/903 loss 0.181 kl 0.000 anchor 0.000 0.263s/rec
-ep0 step 640/903 loss 0.073 kl 0.000 anchor 0.000 0.262s/rec
-ep0 step 650/903 loss 0.152 kl 0.000 anchor 0.000 0.261s/rec
-ep0 step 660/903 loss 0.142 kl 0.000 anchor 0.000 0.260s/rec
-ep0 step 670/903 loss 0.452 kl 0.000 anchor 0.000 0.259s/rec
-ep0 step 680/903 loss 0.165 kl 0.000 anchor 0.000 0.259s/rec
-ep0 step 690/903 loss 0.126 kl 0.000 anchor 0.000 0.258s/rec
-ep0 step 700/903 loss 0.177 kl 0.000 anchor 0.000 0.257s/rec
-ep0 step 710/903 loss 0.200 kl 0.000 anchor 0.000 0.259s/rec
-ep0 step 720/903 loss 0.218 kl 0.000 anchor 0.000 0.258s/rec
-ep0 step 730/903 loss 0.173 kl 0.000 anchor 0.000 0.257s/rec
-ep0 step 740/903 loss 0.233 kl 0.000 anchor 0.000 0.257s/rec
-ep0 step 750/903 loss 0.123 kl 0.000 anchor 0.000 0.256s/rec
-ep0 step 760/903 loss 0.070 kl 0.000 anchor 0.000 0.256s/rec
-ep0 step 770/903 loss 0.339 kl 0.000 anchor 0.000 0.255s/rec
-ep0 step 780/903 loss 0.338 kl 0.000 anchor 0.000 0.255s/rec
-ep0 step 790/903 loss 0.131 kl 0.000 anchor 0.000 0.255s/rec
-ep0 step 800/903 loss 0.110 kl 0.000 anchor 0.000 0.254s/rec
-ep0 step 810/903 loss 0.110 kl 0.000 anchor 0.000 0.256s/rec
-ep0 step 820/903 loss 0.100 kl 0.000 anchor 0.000 0.255s/rec
-ep0 step 830/903 loss 0.390 kl 0.000 anchor 0.000 0.255s/rec
-ep0 step 840/903 loss 0.119 kl 0.000 anchor 0.000 0.255s/rec
-ep0 step 850/903 loss 0.233 kl 0.000 anchor 0.000 0.255s/rec
-ep0 step 860/903 loss 0.226 kl 0.000 anchor 0.000 0.254s/rec
-ep0 step 870/903 loss 0.121 kl 0.000 anchor 0.000 0.253s/rec
-ep0 step 880/903 loss 0.125 kl 0.000 anchor 0.000 0.254s/rec
-ep0 step 890/903 loss 0.604 kl 0.000 anchor 0.000 0.253s/rec
-ep0 step 900/903 loss 0.379 kl 0.000 anchor 0.000 0.253s/rec
-saved /runs/r8-small/00-trial-0/checkpoint
+ep0 step 10/1915 loss 1.451 kl 0.000 anchor 0.000 0.441s/rec
+ep0 step 20/1915 loss 1.678 kl 0.000 anchor 0.000 0.340s/rec
+ep0 step 30/1915 loss 1.520 kl 0.000 anchor 0.000 0.308s/rec
+ep0 step 40/1915 loss 1.340 kl 0.000 anchor 0.000 0.343s/rec
+ep0 step 50/1915 loss 1.612 kl 0.000 anchor 0.000 0.312s/rec
+ep0 step 60/1915 loss 0.875 kl 0.000 anchor 0.000 0.304s/rec
+ep0 step 70/1915 loss 0.809 kl 0.000 anchor 0.000 0.295s/rec
+ep0 step 80/1915 loss 1.151 kl 0.000 anchor 0.000 0.295s/rec
+ep0 step 90/1915 loss 0.824 kl 0.000 anchor 0.000 0.292s/rec
+ep0 step 100/1915 loss 0.846 kl 0.000 anchor 0.000 0.311s/rec
+ep0 step 110/1915 loss 0.823 kl 0.000 anchor 0.000 0.300s/rec
+ep0 step 120/1915 loss 0.807 kl 0.000 anchor 0.000 0.299s/rec
+ep0 step 130/1915 loss 0.850 kl 0.000 anchor 0.000 0.296s/rec
+ep0 step 140/1915 loss 0.655 kl 0.000 anchor 0.000 0.293s/rec
+ep0 step 150/1915 loss 0.886 kl 0.000 anchor 0.000 0.291s/rec
+ep0 step 160/1915 loss 0.604 kl 0.000 anchor 0.000 0.288s/rec
+ep0 step 170/1915 loss 0.811 kl 0.000 anchor 0.000 0.290s/rec
+ep0 step 180/1915 loss 0.616 kl 0.000 anchor 0.000 0.287s/rec
+ep0 step 190/1915 loss 0.784 kl 0.000 anchor 0.000 0.284s/rec
+ep0 step 200/1915 loss 0.731 kl 0.000 anchor 0.000 0.293s/rec
+ep0 step 210/1915 loss 0.741 kl 0.000 anchor 0.000 0.288s/rec
+ep0 step 220/1915 loss 0.638 kl 0.000 anchor 0.000 0.285s/rec
+ep0 step 230/1915 loss 0.950 kl 0.000 anchor 0.000 0.284s/rec
+ep0 step 240/1915 loss 0.637 kl 0.000 anchor 0.000 0.283s/rec
+ep0 step 250/1915 loss 0.727 kl 0.000 anchor 0.000 0.280s/rec
+ep0 step 260/1915 loss 0.724 kl 0.000 anchor 0.000 0.279s/rec
+ep0 step 270/1915 loss 0.598 kl 0.000 anchor 0.000 0.281s/rec
+ep0 step 280/1915 loss 0.507 kl 0.000 anchor 0.000 0.279s/rec
+ep0 step 290/1915 loss 0.581 kl 0.000 anchor 0.000 0.276s/rec
+ep0 step 300/1915 loss 0.649 kl 0.000 anchor 0.000 0.277s/rec
+ep0 step 310/1915 loss 0.608 kl 0.000 anchor 0.000 0.280s/rec
+ep0 step 320/1915 loss 0.773 kl 0.000 anchor 0.000 0.279s/rec
+ep0 step 330/1915 loss 0.695 kl 0.000 anchor 0.000 0.277s/rec
+ep0 step 340/1915 loss 0.689 kl 0.000 anchor 0.000 0.277s/rec
+ep0 step 350/1915 loss 0.615 kl 0.000 anchor 0.000 0.276s/rec
+ep0 step 360/1915 loss 0.618 kl 0.000 anchor 0.000 0.274s/rec
+ep0 step 370/1915 loss 0.754 kl 0.000 anchor 0.000 0.274s/rec
+ep0 step 380/1915 loss 0.514 kl 0.000 anchor 0.000 0.275s/rec
+ep0 step 390/1915 loss 0.569 kl 0.000 anchor 0.000 0.273s/rec
+ep0 step 400/1915 loss 0.574 kl 0.000 anchor 0.000 0.273s/rec
+ep0 step 410/1915 loss 0.591 kl 0.000 anchor 0.000 0.274s/rec
+ep0 step 420/1915 loss 0.690 kl 0.000 anchor 0.000 0.272s/rec
+ep0 step 430/1915 loss 0.643 kl 0.000 anchor 0.000 0.272s/rec
+ep0 step 440/1915 loss 0.616 kl 0.000 anchor 0.000 0.272s/rec
+ep0 step 450/1915 loss 0.595 kl 0.000 anchor 0.000 0.270s/rec
+ep0 step 460/1915 loss 0.643 kl 0.000 anchor 0.000 0.270s/rec
+ep0 step 470/1915 loss 0.646 kl 0.000 anchor 0.000 0.270s/rec
+ep0 step 480/1915 loss 0.608 kl 0.000 anchor 0.000 0.269s/rec
+ep0 step 490/1915 loss 0.488 kl 0.000 anchor 0.000 0.268s/rec
+ep0 step 500/1915 loss 0.586 kl 0.000 anchor 0.000 0.267s/rec
+ep0 step 510/1915 loss 0.407 kl 0.000 anchor 0.000 0.265s/rec
+ep0 step 520/1915 loss 0.483 kl 0.000 anchor 0.000 0.264s/rec
+ep0 step 530/1915 loss 0.484 kl 0.000 anchor 0.000 0.264s/rec
+ep0 step 540/1915 loss 0.771 kl 0.000 anchor 0.000 0.265s/rec
+ep0 step 550/1915 loss 0.588 kl 0.000 anchor 0.000 0.264s/rec
+ep0 step 560/1915 loss 0.443 kl 0.000 anchor 0.000 0.264s/rec
+ep0 step 570/1915 loss 0.489 kl 0.000 anchor 0.000 0.263s/rec
+ep0 step 580/1915 loss 0.473 kl 0.000 anchor 0.000 0.262s/rec
+ep0 step 590/1915 loss 0.718 kl 0.000 anchor 0.000 0.262s/rec
+ep0 step 600/1915 loss 0.634 kl 0.000 anchor 0.000 0.263s/rec
+ep0 step 610/1915 loss 0.480 kl 0.000 anchor 0.000 0.262s/rec
+ep0 step 620/1915 loss 0.579 kl 0.000 anchor 0.000 0.261s/rec
+ep0 step 630/1915 loss 0.386 kl 0.000 anchor 0.000 0.261s/rec
+ep0 step 640/1915 loss 0.660 kl 0.000 anchor 0.000 0.260s/rec
+ep0 step 650/1915 loss 0.761 kl 0.000 anchor 0.000 0.259s/rec
+ep0 step 660/1915 loss 0.703 kl 0.000 anchor 0.000 0.259s/rec
+ep0 step 670/1915 loss 0.393 kl 0.000 anchor 0.000 0.259s/rec
+ep0 step 680/1915 loss 0.502 kl 0.000 anchor 0.000 0.258s/rec
+ep0 step 690/1915 loss 0.718 kl 0.000 anchor 0.000 0.258s/rec
+ep0 step 700/1915 loss 0.582 kl 0.000 anchor 0.000 0.258s/rec
+ep0 step 710/1915 loss 0.458 kl 0.000 anchor 0.000 0.258s/rec
+ep0 step 720/1915 loss 0.496 kl 0.000 anchor 0.000 0.257s/rec
+ep0 step 730/1915 loss 0.458 kl 0.000 anchor 0.000 0.257s/rec
+ep0 step 740/1915 loss 0.533 kl 0.000 anchor 0.000 0.257s/rec
+ep0 step 750/1915 loss 0.559 kl 0.000 anchor 0.000 0.257s/rec
+ep0 step 760/1915 loss 0.540 kl 0.000 anchor 0.000 0.256s/rec
+ep0 step 770/1915 loss 0.455 kl 0.000 anchor 0.000 0.256s/rec
+ep0 step 780/1915 loss 0.595 kl 0.000 anchor 0.000 0.255s/rec
+ep0 step 790/1915 loss 0.337 kl 0.000 anchor 0.000 0.255s/rec
+ep0 step 800/1915 loss 0.603 kl 0.000 anchor 0.000 0.255s/rec
+ep0 step 810/1915 loss 0.524 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 820/1915 loss 0.654 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 830/1915 loss 0.543 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 840/1915 loss 0.531 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 850/1915 loss 0.569 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 860/1915 loss 0.371 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 870/1915 loss 0.442 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 880/1915 loss 0.486 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 890/1915 loss 0.513 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 900/1915 loss 0.537 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 910/1915 loss 0.571 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 920/1915 loss 0.574 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 930/1915 loss 0.512 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 940/1915 loss 0.634 kl 0.000 anchor 0.000 0.251s/rec
+ep0 step 950/1915 loss 0.476 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 960/1915 loss 0.529 kl 0.000 anchor 0.000 0.251s/rec
+ep0 step 970/1915 loss 0.621 kl 0.000 anchor 0.000 0.251s/rec
+ep0 step 980/1915 loss 0.529 kl 0.000 anchor 0.000 0.251s/rec
+ep0 step 990/1915 loss 0.565 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1000/1915 loss 0.638 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1010/1915 loss 0.600 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1020/1915 loss 0.581 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1030/1915 loss 0.594 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1040/1915 loss 0.539 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1050/1915 loss 0.547 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1060/1915 loss 0.714 kl 0.000 anchor 0.000 0.250s/rec
+ep0 step 1070/1915 loss 0.378 kl 0.000 anchor 0.000 0.249s/rec
+ep0 step 1080/1915 loss 0.657 kl 0.000 anchor 0.000 0.251s/rec
+ep0 step 1090/1915 loss 0.463 kl 0.000 anchor 0.000 0.251s/rec
+ep0 step 1100/1915 loss 0.585 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1110/1915 loss 0.312 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1120/1915 loss 0.539 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1130/1915 loss 0.541 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1140/1915 loss 0.784 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1150/1915 loss 0.558 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1160/1915 loss 0.600 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1170/1915 loss 0.562 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1180/1915 loss 0.723 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1190/1915 loss 0.394 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1200/1915 loss 0.394 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1210/1915 loss 0.331 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1220/1915 loss 0.334 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1230/1915 loss 0.447 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1240/1915 loss 0.551 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1250/1915 loss 0.509 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1260/1915 loss 0.694 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1270/1915 loss 0.532 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1280/1915 loss 0.496 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1290/1915 loss 0.556 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1300/1915 loss 0.403 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1310/1915 loss 0.537 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1320/1915 loss 0.517 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1330/1915 loss 0.633 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1340/1915 loss 0.491 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1350/1915 loss 0.512 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1360/1915 loss 0.398 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1370/1915 loss 0.396 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1380/1915 loss 0.484 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1390/1915 loss 0.492 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1400/1915 loss 0.487 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1410/1915 loss 0.410 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1420/1915 loss 0.405 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1430/1915 loss 0.391 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1440/1915 loss 0.408 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1450/1915 loss 0.394 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1460/1915 loss 0.461 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1470/1915 loss 0.485 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1480/1915 loss 0.480 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1490/1915 loss 0.539 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1500/1915 loss 0.541 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1510/1915 loss 0.554 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1520/1915 loss 0.513 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1530/1915 loss 0.423 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1540/1915 loss 0.471 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1550/1915 loss 0.342 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 1560/1915 loss 0.547 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1570/1915 loss 0.472 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1580/1915 loss 0.477 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1590/1915 loss 0.593 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1600/1915 loss 0.995 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1610/1915 loss 0.459 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1620/1915 loss 0.609 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1630/1915 loss 0.386 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1640/1915 loss 0.555 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1650/1915 loss 0.517 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1660/1915 loss 0.390 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1670/1915 loss 0.503 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1680/1915 loss 0.870 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1690/1915 loss 0.419 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1700/1915 loss 0.429 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1710/1915 loss 0.552 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1720/1915 loss 0.504 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1730/1915 loss 0.402 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1740/1915 loss 0.383 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1750/1915 loss 0.631 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1760/1915 loss 0.618 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1770/1915 loss 0.335 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1780/1915 loss 0.250 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1790/1915 loss 0.499 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1800/1915 loss 0.351 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1810/1915 loss 0.613 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1820/1915 loss 0.355 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1830/1915 loss 0.556 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1840/1915 loss 0.328 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1850/1915 loss 0.863 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1860/1915 loss 0.441 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1870/1915 loss 0.385 kl 0.000 anchor 0.000 0.253s/rec
+ep0 step 1880/1915 loss 0.398 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1890/1915 loss 0.507 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1900/1915 loss 0.602 kl 0.000 anchor 0.000 0.252s/rec
+ep0 step 1910/1915 loss 0.570 kl 0.000 anchor 0.000 0.251s/rec
+saved /runs/r10-skills/00-trial-0/checkpoint

training_config.json

@@ -37,20 +37,20 @@
     "anchor": "",
     "anchor_w": 0.0,
     "anchor_sources": "",
-    "out": "/runs/r8-small/00-trial-0/checkpoint",
-    "data": "evals/documents-v1/train.jsonl",
+    "out": "/runs/r10-skills/00-trial-0/checkpoint",
+    "data": "evals/round10/skills/train.jsonl",
     "max_state": 7552,
-    "replay": 2000,
-    "init_from": "jaredpalmer/kev-4b",
-    "seed": 2
+    "replay": 4000,
+    "init_from": "jaredpalmer/kev-4b@957b91e762e883935830246eeb02381f9d2694b6",
+    "seed": 1
   },
   "suite_sha256": "a8f50e481b7d90b97da049e0ff6a01cee2f1ed204aed61a8265af0edbb5514d2",
   "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
   "init_source": {
-    "init_from": "jaredpalmer/kev-4b",
-    "resolved": "/__modal/volumes/vo-kEMu8BkBAIrorAQI6V8f2D/hub/models--jaredpalmer--kev-4b/snapshots/485ace8703592fcf405488b262449990824cfed1",
-    "adapter_sha256": "9797de69a42188e411b17b7b4fcb66a23374dcebc21d71a7a66f836b5d34df2b",
-    "head_sha256": "d8f796da36ff7bd7c0fb9496b452139bb7851af4fc82b07b500b682d3f721d6a",
+    "init_from": "jaredpalmer/kev-4b@957b91e762e883935830246eeb02381f9d2694b6",
+    "resolved": "/__modal/volumes/vo-kEMu8BkBAIrorAQI6V8f2D/hub/models--jaredpalmer--kev-4b/snapshots/957b91e762e883935830246eeb02381f9d2694b6",
+    "adapter_sha256": "c2c99077b58cd4a604304b743b7db5391187ef1d58e575f90c554c744aa23d65",
+    "head_sha256": "9a3051c3676f0be29d885935b6f50c287f7acf04e3cb29a6fed82ca3e5540b7e",
     "adapter_tensors": 496
   },
   "ordinal_objective": "ranked_probability_score",

training_metrics.json

@@ -1,14 +1,14 @@
 {
-  "wall_seconds": 2574.7955305576324,
-  "records_seen": 10179,
-  "requested_records": 7219,
+  "wall_seconds": 5031.164765119553,
+  "records_seen": 19994,
+  "requested_records": 15320,
   "truncated_records": 0,
   "rejected_records": 0,
-  "optimizer_steps": 903,
-  "forward_tokens": 6749150,
-  "peak_device_bytes": 39839947776,
+  "optimizer_steps": 1915,
+  "forward_tokens": 8738926,
+  "peak_device_bytes": 47737451520,
   "device": "cuda",
   "dtype": "bf16",
   "batch": 2,
-  "peak_rss_bytes": 31563161600
+  "peak_rss_bytes": 31552528384
 }