v0.2.1 benchmark/architecture atlas; preserve model weights

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+benchmarks/v0.2.1/architecture.png filter=lfs diff=lfs merge=lfs -text
+benchmarks/v0.2.1/architecture_control.png filter=lfs diff=lfs merge=lfs -text
+benchmarks/v0.2.1/context_size_performance.png filter=lfs diff=lfs merge=lfs -text
+benchmarks/v0.2.1/matched_size_performance.png filter=lfs diff=lfs merge=lfs -text
+benchmarks/v0.2.1/public_comparison.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,71 +1,84 @@
----
-license: apache-2.0
-base_model: Qwen/Qwen3.5-9B-Base
-library_name: peft
-pipeline_tag: text-classification
-tags: [midm, decision-model, typed-decisions, pointer-head, lora, qlora]
----
-
-# MiDM-9B-q35-e1-bx
-
-Version **v0.2.0**. This repository contains the LoRA adapter, a linear pointer head and a self-contained Python loader. It scores the provided options jointly in one forward pass; it does not generate SQL, Python or explanations. The API accepts `choice`, yes/no `noul`, and ordinal `score` questions. Outputs are option probabilities, not guaranteed calibrated confidence.
-
-## Use
-
-Install `requirements.txt`, download this repository at revision `v0.2.0`, and run from that directory:
-
-```python
-from midm import MiDM
-model = MiDM.from_pretrained(".")  # downloads the base separately; NF4 by default
-result = model.predict(
-    state="A receipt is required for a refund. The customer has no receipt.",
-    questions={"action": {"type": "choice", "instructions": "Choose the next action.",
-        "criteria": {"approve": "Approve the refund.", "review": "Request a receipt or review."}}},
-)
-print(result)
-```
-
-A compatible NVIDIA GPU and the listed dependencies are required. The configured maximum input is 4096 tokens; individual options are capped at 192 tokens. Long states are truncated while retaining the head and tail. All inference can run offline once the package and base are cached. Apache-2.0 applies to this adapter and code; base model and training data retain their own terms. This repository includes no training data.
-
-# MiDM v0.2.0 — 9B checkpoint and completed evaluation results
-
-Adds the Qwen3.5-9B-Base LoRA adapter and learned pointer head, self-contained loader and verified 4096-token inference configuration. The original 4B release remains available. This model scores supplied options; it does not generate SQL, Python or report prose. The report writer remains Qwen3 8B.
-
-## Matched primary evaluation
-
-NF4/BF16 on RTX 3090, max input 4096, batch token budget 4096, TTA=1; same items and ordering.
-
-|Suite|N|4B|9B|
-|---|---:|---:|---:|
-|Typed Decisions test|2000|78.85%|79.30%|
-|Kev transfer test|764|79.84%|82.59%|
-|JevBench original|72|91.67%|97.22%|
-|JevBench easy|48|100.00%|100.00%|
-|JevBench hard|111|45.95%|54.05%|
-
-These are retrospective hard-label accuracies, not official JevBench full/sealed composite scores. The separate 600-question Typed Decisions holdout is a development/model-selection split; it is not the public 2000-question test. Earlier development results used their recorded configurations and must not be mixed into this matched table.
-
-## Additional completed tests and limitations
-
-- CLM released DeepSWE verifier replay: 31/38 (81.58%) reproduced. The 13 mixed-outcome tasks give exact one-sided p=0.062 against random selection. Numerical reproduction succeeded; significance at 5% was not established.
-- MiDM DeepSWE and local Terminal-Bench remain unmeasured: matching raw candidate traces/evaluation artifacts were not found. No official benchmark score is substituted.
-- SQL/Python best-of-N: 4B → 9B Spider 78.34% → 77.27%, SQL holdout 81.66% → 81.77%, Python hard80 28.75% → 25.00%. Stored pools, dev-selected blend weights; no consistent gain.
-- Learned routing: JevBench public 231 accuracy 77.06% MiDM, 79.65% router, 81.82% local Qwen3-4B reasoning alone. Router not promoted; no non-inferiority or latency-saving claim.
-- Concurrency: two short-input 4B processes nearly doubled throughput; 2048-token 4B and 4096-token 9B lost throughput. Small synthetic single-trial pilots, not endurance or minimum-memory certification.
-- Same-date report integration: 4B/9B candidate-order agreement 1/4 versus 3/4, 19/19 schema-valid sections each, one unknown fact ID each. No semantic-quality or financial-outcome superiority established; only aggregate diagnostics are released.
-
-Training: QLoRA NF4, rank16/alpha32, one epoch, seed0, learning rate2e-4, training context1024; td_train, kev_train, breadth_v1, pp6_new. Inference context4096 matches evaluation. The earlier 1024-token release candidate was not published; package verification uses the final loader. Base weights are separate and retain their own license, as do training datasets. No raw questions, answers, financial data, reports or credentials are included.
-
-Model: https://huggingface.co/yunicro/MiDM-9B-q35-e1-bx/tree/v0.2.0
-
-Code and full evidence: https://github.com/YeoHoonYun/midm-decision-models/releases/tag/v0.2.0
-
-Zenodo versioning retains concept DOI https://doi.org/10.5281/zenodo.23084583 . The v0.1.0 version DOI does not archive this new release.
-
-Version DOI: https://doi.org/10.5281/zenodo.23162744
-
-## Packaged-model verification
-
-The final package reproduces the original development evaluation exactly: **513/600 (85.50%)**, identical choices on 600/600 questions and maximum absolute probability difference 0.0. The original protocol uses NF4/BF16, 4096-token context, 4096 padded tokens per batch, attention masks and TTA=1. Adapter weights, pointer head and encoded inputs match the source exactly. This is package reproducibility, not a new independent test.
-
-A separate single-question, unmasked check gave **509/600 (84.83%)** and failed its original tolerance gate. It is retained rather than hidden. That verification bypassed the public API batching/mask path. The matched check changes batching and mask together, so their individual effects were not isolated. Do not assume identical predictions across serving configurations.
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.5-9B-Base
+library_name: peft
+pipeline_tag: text-classification
+tags: [midm, decision-model, typed-decisions, pointer-head, lora, qlora]
+---
+
+# MiDM-9B-q35-e1-bx
+
+## v0.2.1 benchmark and architecture analysis
+
+Documentation update; this repository's existing weights and inference code are unchanged. [Full benchmark atlas](benchmarks/v0.2.1/README.md) · [GitHub release](https://github.com/YeoHoonYun/midm-decision-models/releases/tag/v0.2.1) · [Version DOI](https://doi.org/10.5281/zenodo.23164651).
+
+![Matched size and performance](benchmarks/v0.2.1/matched_size_performance.png)
+
+![Architecture comparison](benchmarks/v0.2.1/architecture.png)
+
+The atlas separates matched measurements from historical/published/API comparisons. JevBench public accuracy is not its official composite. Missing MiDM DeepSWE and local Terminal-Bench measurements remain explicit; their candidate pools were not found. Unknown proprietary parameter counts are not guessed. SQL/Python regressions remain documented.
+
+![Public cohort comparison](benchmarks/v0.2.1/public_comparison.png)
+
+
+Version **v0.2.0**. This repository contains the LoRA adapter, a linear pointer head and a self-contained Python loader. It scores the provided options jointly in one forward pass; it does not generate SQL, Python or explanations. The API accepts `choice`, yes/no `noul`, and ordinal `score` questions. Outputs are option probabilities, not guaranteed calibrated confidence.
+
+## Use
+
+Install `requirements.txt`, download this repository at revision `v0.2.0`, and run from that directory:
+
+```python
+from midm import MiDM
+model = MiDM.from_pretrained(".")  # downloads the base separately; NF4 by default
+result = model.predict(
+    state="A receipt is required for a refund. The customer has no receipt.",
+    questions={"action": {"type": "choice", "instructions": "Choose the next action.",
+        "criteria": {"approve": "Approve the refund.", "review": "Request a receipt or review."}}},
+)
+print(result)
+```
+
+A compatible NVIDIA GPU and the listed dependencies are required. The configured maximum input is 4096 tokens; individual options are capped at 192 tokens. Long states are truncated while retaining the head and tail. All inference can run offline once the package and base are cached. Apache-2.0 applies to this adapter and code; base model and training data retain their own terms. This repository includes no training data.
+
+# MiDM v0.2.0 — 9B checkpoint and completed evaluation results
+
+Adds the Qwen3.5-9B-Base LoRA adapter and learned pointer head, self-contained loader and verified 4096-token inference configuration. The original 4B release remains available. This model scores supplied options; it does not generate SQL, Python or report prose. The report writer remains Qwen3 8B.
+
+## Matched primary evaluation
+
+NF4/BF16 on RTX 3090, max input 4096, batch token budget 4096, TTA=1; same items and ordering.
+
+|Suite|N|4B|9B|
+|---|---:|---:|---:|
+|Typed Decisions test|2000|78.85%|79.30%|
+|Kev transfer test|764|79.84%|82.59%|
+|JevBench original|72|91.67%|97.22%|
+|JevBench easy|48|100.00%|100.00%|
+|JevBench hard|111|45.95%|54.05%|
+
+These are retrospective hard-label accuracies, not official JevBench full/sealed composite scores. The separate 600-question Typed Decisions holdout is a development/model-selection split; it is not the public 2000-question test. Earlier development results used their recorded configurations and must not be mixed into this matched table.
+
+## Additional completed tests and limitations
+
+- CLM released DeepSWE verifier replay: 31/38 (81.58%) reproduced. The 13 mixed-outcome tasks give exact one-sided p=0.062 against random selection. Numerical reproduction succeeded; significance at 5% was not established.
+- MiDM DeepSWE and local Terminal-Bench remain unmeasured: matching raw candidate traces/evaluation artifacts were not found. No official benchmark score is substituted.
+- SQL/Python best-of-N: 4B → 9B Spider 78.34% → 77.27%, SQL holdout 81.66% → 81.77%, Python hard80 28.75% → 25.00%. Stored pools, dev-selected blend weights; no consistent gain.
+- Learned routing: JevBench public 231 accuracy 77.06% MiDM, 79.65% router, 81.82% local Qwen3-4B reasoning alone. Router not promoted; no non-inferiority or latency-saving claim.
+- Concurrency: two short-input 4B processes nearly doubled throughput; 2048-token 4B and 4096-token 9B lost throughput. Small synthetic single-trial pilots, not endurance or minimum-memory certification.
+- Same-date report integration: 4B/9B candidate-order agreement 1/4 versus 3/4, 19/19 schema-valid sections each, one unknown fact ID each. No semantic-quality or financial-outcome superiority established; only aggregate diagnostics are released.
+
+Training: QLoRA NF4, rank16/alpha32, one epoch, seed0, learning rate2e-4, training context1024; td_train, kev_train, breadth_v1, pp6_new. Inference context4096 matches evaluation. The earlier 1024-token release candidate was not published; package verification uses the final loader. Base weights are separate and retain their own license, as do training datasets. No raw questions, answers, financial data, reports or credentials are included.
+
+Model: https://huggingface.co/yunicro/MiDM-9B-q35-e1-bx/tree/v0.2.0
+
+Code and full evidence: https://github.com/YeoHoonYun/midm-decision-models/releases/tag/v0.2.0
+
+Zenodo versioning retains concept DOI https://doi.org/10.5281/zenodo.23084583 . The v0.1.0 version DOI does not archive this new release.
+
+Version DOI: https://doi.org/10.5281/zenodo.23162744
+
+## Packaged-model verification
+
+The final package reproduces the original development evaluation exactly: **513/600 (85.50%)**, identical choices on 600/600 questions and maximum absolute probability difference 0.0. The original protocol uses NF4/BF16, 4096-token context, 4096 padded tokens per batch, attention masks and TTA=1. Adapter weights, pointer head and encoded inputs match the source exactly. This is package reproducibility, not a new independent test.
+
+A separate single-question, unmasked check gave **509/600 (84.83%)** and failed its original tolerance gate. It is retained rather than hidden. That verification bypassed the public API batching/mask path. The matched check changes batching and mask together, so their individual effects were not isolated. Do not assume identical predictions across serving configurations.

benchmark_manifest_v0.2.1.json

@@ -0,0 +1,25 @@
+{
+  "documentation_version": "0.2.1",
+  "weights_unchanged": {
+    "adapter_model.safetensors": "7231ef52a482760601bd6e9038ec9aae40034d0f63c6fb5de677b5f81e6dd5c2",
+    "pointer_head.safetensors": "f534d840c1b521c04292e763db200e6ab10df12fb4ebdabc78de734c07d88e4a"
+  },
+  "files": {
+    "manifest.json": "71ca79f112c3d0a3e26b213dda6465a2be8aa6f77fc795c93a8bac377c83d605",
+    "README.md": "b392455d328c9af67f162394beeef4d1c230d5d5652fda4e94023d51af15f75d",
+    "benchmarks/v0.2.1/architecture.png": "cc5bfb7fc225b1f974837e36ef43717af271fe1d56281f6faa8790f4d1762280",
+    "benchmarks/v0.2.1/architecture.svg": "0d77746badfff06e4b07f686753830f565568e98ad6b88dfa3e6e179c413bba7",
+    "benchmarks/v0.2.1/architecture_control.png": "18e6afac826bda17ed80a7f0252f63c69109aeeca41026efbc50d01845b52746",
+    "benchmarks/v0.2.1/architecture_control.svg": "d837c9733a0e3af6c4b7c2078a940af18f3d74c6bb5871ff2231f4525c89b661",
+    "benchmarks/v0.2.1/atlas.json": "3d82078bac0024ac3fc4a814580a7fd11528c6f659acda99099e8a3703e70c4a",
+    "benchmarks/v0.2.1/context_size_performance.png": "62364f4bdfd3a00358bf2631a03d550b86d75b62b0682e936a7782d30eda5588",
+    "benchmarks/v0.2.1/context_size_performance.svg": "ef50c39dc7e19f0f289064ae41baf77889ca8fa48f656a4f92f4b9d8cd06c7c3",
+    "benchmarks/v0.2.1/matched_size_performance.png": "6e5b233ccfca43b71bd4f2884d38881751628db14369a035eac97ac5995b8729",
+    "benchmarks/v0.2.1/matched_size_performance.svg": "ea0fbc1d8c4658254c8be3a4e37094262877c404b4357e4d1e65fb2e4c07d83b",
+    "benchmarks/v0.2.1/plot_atlas.py": "2081011cfd8cef64f0f16887f5bf5df4fb1ba71c9a2da31e640e141ee7b03254",
+    "benchmarks/v0.2.1/public_comparison.png": "2ce2d9502232d5683e840afa3e97a3b203665b31b9dec6fa63d4fc3b1576aa87",
+    "benchmarks/v0.2.1/public_comparison.svg": "921ef3941ca10f90acce98bed502e95caaa143a6eb6ad3a706765cd36d3dcbfc",
+    "benchmarks/v0.2.1/README.md": "0768c6ea91f2ce2e218e268d4e7c3fb9a943e6070464976f0be57bd4555fe2b7",
+    "benchmarks/v0.2.1/upstream_inventory.json": "2515bc93354d78242e0df50aef7130ed67d19190d53384daf1fcd0746e3c1186"
+  }
+}

benchmarks/v0.2.1/README.md

@@ -0,0 +1,102 @@
+# Benchmark coverage, architecture and parameter-performance atlas
+
+**v0.2.1 documentation and analysis update. Model weights are unchanged from v0.2.0.** All figures below are derived from completed runs or identified published outcomes. No new model inference, external paid model calls or financial results are implied.
+
+## What fills the previous blanks
+
+|Benchmark / metric|MiDM 4B|MiDM 9B|Comparison and status|
+|---|---:|---:|---|
+|Typed Decisions test (N=2000)|1577/2000 (78.85%)|1586/2000 (79.30%)|Matched local measurement|
+|Kev transfer test (N=764)|610/764 (79.84%)|631/764 (82.59%)|Matched local measurement|
+|JevBench public accuracy (N=231)|165/231 (71.43%)|178/231 (77.06%)|Matched local measurement|
+|JevBench hard accuracy (N=111)|51/111 (45.95%)|60/111 (54.05%)|Matched local measurement|
+|DeepSWE held-out 38, best-of-4|Not measured|Not measured|Released CLM head replay: **31/38 = 81.58%**. MiDM needs the same raw state/action texts; cached CLM embeddings cannot be used as MiDM inputs.|
+|Terminal-Bench verifier, reported 30 tasks|Not measured|Not measured|CLM author reports 87.6%; matching candidate pool, head and full scoring protocol not found. Not a local result and not necessarily task-level integer accuracy.|
+|Official JevBench composite|Not measured|Not measured|Public accuracy cannot replace sealed decisions, distribution calibration, controlled latency and cost axes.|
+
+A blank score is not zero. Missing measurements remain explicit and are not plotted as zero or copied from another model. The [scoped upstream inventory](upstream_inventory.json) rechecks official GitHub and Hugging Face releases; it does not claim an exhaustive search of the internet. A fresh Terminal-Bench agent rollout would be a different experiment from the released best-of-N verifier comparison.
+
+## Architecture: what is being compared
+
+![Architecture comparison](architecture.png)
+
+|System|Input interaction and output|Learned components|Parameter disclosure|
+|---|---|---|---|
+|Released CLM 8B|State and each candidate encoded independently; two projection MLPs; scaled cosine similarity|Frozen Qwen3-8B encoder plus learned state/action heads|Nominal 8B encoder; task-specific head differs from reference head|
+|Local CLM-style 4B control|Independent encoded state/action features; projection heads|Local reimplementation, Qwen3-4B frozen|Nominal 4B; **not** released CLM 8B|
+|MiDM 4B / 9B|State, question and option lines in one causal sequence; shared linear pointer at each option end; softmax over options|Frozen NF4 base + rank-16 LoRA + pointer|Nominal 4B / 9B base; exact adapter/head counts below|
+|Qwen3 4B reasoning|Generates a token sequence and parses a selected answer|Existing instruction/reasoning model|Nominal 4B; more inference work than one-pass selection|
+|Jev / Kev|Typed decision interface; benchmark-author outcomes available|Exact architecture not established by this local audit|Kev nominal 4B/8B labels; Jev size unknown here|
+|Solar Pro4|Generative API answer parsed to a choice|Proprietary service; internals not audited|Unknown here; omitted from parameter-axis plots|
+
+**Causal attention matters:** a later option can see preceding options, but an earlier option-end state cannot see later options. MiDM is not a bidirectional all-options cross-encoder. Option-order sensitivity and context truncation therefore remain relevant. CLM can cache state/action embeddings independently; MiDM option scores generally require the joint sequence to be recomputed. Neither parameter count alone nor adapter file size measures serving latency.
+
+|Package|Nominal base|LoRA parameters (exact)|Pointer parameters (exact)|Trainable components total|
+|---|---:|---:|---:|---:|
+|MiDM 4B|4B|30,474,240|2,561|30,476,801|
+|MiDM 9B|9B|40,108,032|4,097|40,112,129|
+
+Counts are computed from shipped safetensors shapes, not estimated from file bytes. Base labels are nominal and are not exact instantiated text/vision/LM-head parameter counts. The base model is downloaded separately; a small adapter does not make the full model equally small.
+
+## Parameters versus performance
+
+![Matched parameter performance](matched_size_performance.png)
+
+The primary figure contains only the matched 4B/9B run. With 2.25 times the nominal base parameters, 9B improves Typed Decisions by **0.45 pp**, Kev transfer test by **2.75 pp**, JevBench public by **5.63 pp**, and its hard tier by **8.11 pp**. These retrospective point differences do not establish a general scaling law or statistical superiority.
+
+![Contextual parameter comparison](context_size_performance.png)
+
+The contextual plot keeps historical local and benchmark-published rows visibly separate. Historical bases, precision, epoch count, training mix and execution paths differ. It is not a matched architecture or size ablation. Systems with unknown parameter counts are excluded from this x-axis, rather than assigned a guessed size.
+
+## Same public items, different inference protocols
+
+![Public cohort comparison](public_comparison.png)
+
+|System|Nominal B|Public correct / 231|Public accuracy|Hard accuracy|Evidence class|
+|---|---:|---:|---:|---:|---|
+|MiDM 4B (matched)|4|165/231|71.43%|45.95%|local matched BF16/NF4|
+|MiDM 9B (matched)|9|178/231|77.06%|54.05%|local matched BF16/NF4|
+|CLM-style 4B control|4|95/231|41.13%|36.04%|historical local; configuration differs|
+|Frozen joint pointer 4B|4|109/231|47.19%|36.94%|historical local; configuration differs|
+|MiDM Qwen3 4B|4|159/231|68.83%|39.64%|historical local; configuration differs|
+|MiDM Qwen3 8B (2ep)|8|163/231|70.56%|42.34%|historical local; configuration differs|
+|MiDM Qwen3.5 2B|2|147/231|63.64%|37.84%|historical local; configuration differs|
+|Kev 4B (published)|4|153/231|66.23%|36.94%|benchmark-author per-item outcomes, JevBench v1.2 snapshot|
+|Kev 8B (published)|8|165/231|71.43%|45.05%|benchmark-author per-item outcomes, JevBench v1.2 snapshot|
+|Jev 1.13 (published)|Unknown|200/231|86.58%|72.97%|benchmark-author per-item outcomes, JevBench v1.2 snapshot|
+|Jev 1.13 Free (local API run)|Unknown|194/231|83.98%|67.57%|local API responses; different prompting/compute from pointer models|
+|Solar Pro4 (local API run)|Unknown|220/231|95.24%|90.99%|local API responses; different prompting/compute from pointer models|
+|Qwen3 4B reasoning|4|189/231|81.82%|65.77%|local reasoning run; different token budget and output path|
+
+Jev published 200/231 and the local Jev Free run 194/231 are distinct service/protocol observations; neither overwrites the other. Solar 220/231 uses a generative API. They are useful capability references, not equal-compute or equal-cost experiments. The incomplete Space Bunny cohort (162/231 in the inspected snapshot) is excluded from the full-cohort graph; its 159/162 must not be ranked against complete 231-item results. This update makes no fresh calls to these model APIs.
+
+## Architecture control and limits
+
+![Historical architecture control](architecture_control.png)
+
+The Qwen3-4B control series holds the nominal base family/size and original task mix, but training recipes and precision details still differ. Kev transfer accuracy rises from 41.10% (independent frozen heads) to 62.96% (frozen causal pointer) to 71.60% (QLoRA pointer). This supports candidate-context interaction as a useful design direction, but does not isolate every causal contribution. It must not be confused with a direct comparison against the released CLM 8B head.
+
+## SQL/Python and deployment tradeoffs
+
+|Same stored candidate pool|4B|9B|Interpretation|
+|---|---:|---:|---|
+|Spider test, RAG5 + dev-selected blend|78.34%|77.27%|Regression|
+|SQL holdout, RAG5 + dev-selected blend|81.66%|81.77%|Small point increase|
+|Python hard80, RAG0|28.75%|25.00%|Regression|
+
+These are best-of-N selection results, not direct code generation. [Full SQL/Python protocol](https://github.com/YeoHoonYun/midm-decision-models/blob/v0.2.1/docs/evaluations/20261005/SQL_PYTHON_9B.md). The learned router reaches 79.65% public accuracy, below local reasoning alone at 81.82%; it is not promoted as the default. Long-context two-process pilots reduce throughput despite fitting in VRAM. [Serving and routing measurements](https://github.com/YeoHoonYun/midm-decision-models/blob/v0.2.1/docs/evaluations/20261005/README.md).
+
+## Interpretation
+
+9B improves several bounded-choice benchmarks and especially the public hard tier, but it does not dominate the generative comparison models or improve every application. The 4B model remains a smaller one-pass option scorer; the 9B model trades a larger base for higher decision accuracy on these suites. Unknown proprietary sizes, different inference budgets, seen public data and historical recipe differences prevent a universal performance-per-parameter ranking.
+
+All outcomes are retrospective. Public-item accuracy is not an official composite score; trainable parameters are not full-model parameters; nominal size is not VRAM; batched throughput is not single-request latency. No financial inputs, probabilities, generated reports, raw benchmark examples or credentials are included.
+
+## Sources and reproduction
+
+[Machine-readable aggregates and source hashes](atlas.json) · [Plotting script](plot_atlas.py) · [Upstream asset inventory](upstream_inventory.json). Run `python plot_atlas.py` with Python, NumPy and Matplotlib to reproduce PNG/SVG plots from `atlas.json`.
+
+- [CLM source and verifier protocol](https://github.com/Contrastive-LM/CLM); [projection implementation](https://github.com/Contrastive-LM/CLM/blob/main/src/clm/heads.py).
+- [Released CLM DeepSWE heads](https://huggingface.co/Contrastive-LM/deepswe-clm-heads-8k).
+- [JevBench v1.2 per-task source](https://github.com/fstandhartinger/jevbench/blob/main/results/v1.2/jevbench-v1.2-per-task.json). The archived local snapshot, not an assumed current leaderboard, underlies these figures.
+- [MiDM primary evaluation](https://github.com/YeoHoonYun/midm-decision-models/blob/v0.2.1/docs/evaluations/20261005/README.md), [four-benchmark rescore](https://github.com/YeoHoonYun/midm-decision-models/blob/v0.2.1/docs/evaluations/20261005/four-benchmarks/README.md), and previously published local aggregate controls in this repository.

benchmarks/v0.2.1/atlas.json

@@ -0,0 +1,587 @@
+{
+  "created": "2026-10-05T23:16:39.058397",
+  "metric": "hard-label argmax accuracy; public JevBench accuracy is NOT official composite",
+  "parameter_axis": "nominal advertised base-model billions, not exact active/total count, trainable count or memory",
+  "models": [
+    {
+      "id": "midm_4b",
+      "name": "MiDM 4B (matched)",
+      "parameters_b_nominal": 4,
+      "family": "MiDM pointer + QLoRA",
+      "base": "Qwen3.5-4B-Base",
+      "provenance": "local matched BF16/NF4",
+      "metrics": {
+        "td_test": {
+          "n": 2000,
+          "correct": 1577,
+          "accuracy": 0.7885
+        },
+        "kevT_dev": {
+          "n": 764,
+          "correct": 604,
+          "accuracy": 0.7905759162303665
+        },
+        "kevT_test": {
+          "n": 764,
+          "correct": 610,
+          "accuracy": 0.7984293193717278
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 66,
+          "accuracy": 0.9166666666666666
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 51,
+          "accuracy": 0.4594594594594595
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 165,
+          "accuracy": 0.7142857142857143
+        }
+      }
+    },
+    {
+      "id": "midm_9b",
+      "name": "MiDM 9B (matched)",
+      "parameters_b_nominal": 9,
+      "family": "MiDM pointer + QLoRA",
+      "base": "Qwen3.5-9B-Base",
+      "provenance": "local matched BF16/NF4",
+      "metrics": {
+        "td_test": {
+          "n": 2000,
+          "correct": 1586,
+          "accuracy": 0.793
+        },
+        "kevT_dev": {
+          "n": 764,
+          "correct": 604,
+          "accuracy": 0.7905759162303665
+        },
+        "kevT_test": {
+          "n": 764,
+          "correct": 631,
+          "accuracy": 0.8259162303664922
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 70,
+          "accuracy": 0.9722222222222222
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 60,
+          "accuracy": 0.5405405405405406
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 178,
+          "accuracy": 0.7705627705627706
+        }
+      }
+    },
+    {
+      "id": "clm_style4",
+      "name": "CLM-style 4B control",
+      "parameters_b_nominal": 4,
+      "family": "Frozen independent encoders + projection heads",
+      "base": "Qwen/Qwen3-4B",
+      "provenance": "historical local; configuration differs",
+      "metrics": {
+        "td_test": {
+          "n": 2000,
+          "correct": 1518,
+          "accuracy": 0.759
+        },
+        "kevT_dev": {
+          "n": 764,
+          "correct": 280,
+          "accuracy": 0.36649214659685864
+        },
+        "kevT_test": {
+          "n": 764,
+          "correct": 314,
+          "accuracy": 0.4109947643979058
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 29,
+          "accuracy": 0.4027777777777778
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 26,
+          "accuracy": 0.5416666666666666
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 40,
+          "accuracy": 0.36036036036036034
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 95,
+          "accuracy": 0.41125541125541126
+        }
+      }
+    },
+    {
+      "id": "pointer_frozen4",
+      "name": "Frozen joint pointer 4B",
+      "parameters_b_nominal": 4,
+      "family": "Frozen causal encoder + pointer",
+      "base": "Qwen/Qwen3-4B",
+      "provenance": "historical local; configuration differs",
+      "metrics": {
+        "td_test": {
+          "n": 2000,
+          "correct": 1160,
+          "accuracy": 0.58
+        },
+        "kevT_dev": {
+          "n": 764,
+          "correct": 439,
+          "accuracy": 0.574607329842932
+        },
+        "kevT_test": {
+          "n": 764,
+          "correct": 481,
+          "accuracy": 0.6295811518324608
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 26,
+          "accuracy": 0.3611111111111111
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 42,
+          "accuracy": 0.875
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 41,
+          "accuracy": 0.36936936936936937
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 109,
+          "accuracy": 0.47186147186147187
+        }
+      }
+    },
+    {
+      "id": "midm_q3_4",
+      "name": "MiDM Qwen3 4B",
+      "parameters_b_nominal": 4,
+      "family": "MiDM pointer + QLoRA",
+      "base": "Qwen/Qwen3-4B",
+      "provenance": "historical local; configuration differs",
+      "metrics": {
+        "td_test": {
+          "n": 2000,
+          "correct": 1578,
+          "accuracy": 0.789
+        },
+        "kevT_dev": {
+          "n": 764,
+          "correct": 535,
+          "accuracy": 0.7002617801047121
+        },
+        "kevT_test": {
+          "n": 764,
+          "correct": 547,
+          "accuracy": 0.7159685863874345
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 67,
+          "accuracy": 0.9305555555555556
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 44,
+          "accuracy": 0.3963963963963964
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 159,
+          "accuracy": 0.6883116883116883
+        }
+      }
+    },
+    {
+      "id": "midm_q3_8",
+      "name": "MiDM Qwen3 8B (2ep)",
+      "parameters_b_nominal": 8,
+      "family": "MiDM pointer + QLoRA",
+      "base": "Qwen/Qwen3-8B",
+      "provenance": "historical local; configuration differs",
+      "metrics": {
+        "td_test": {
+          "n": 2000,
+          "correct": 1610,
+          "accuracy": 0.805
+        },
+        "kevT_dev": {
+          "n": 764,
+          "correct": 557,
+          "accuracy": 0.7290575916230366
+        },
+        "kevT_test": {
+          "n": 764,
+          "correct": 590,
+          "accuracy": 0.7722513089005235
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 68,
+          "accuracy": 0.9444444444444444
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 47,
+          "accuracy": 0.42342342342342343
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 163,
+          "accuracy": 0.7056277056277056
+        }
+      }
+    },
+    {
+      "id": "midm_q35_2",
+      "name": "MiDM Qwen3.5 2B",
+      "parameters_b_nominal": 2,
+      "family": "MiDM pointer + QLoRA",
+      "base": "Qwen/Qwen3.5-2B-Base",
+      "provenance": "historical local; configuration differs",
+      "metrics": {
+        "td_test": {
+          "n": 2000,
+          "correct": 1562,
+          "accuracy": 0.781
+        },
+        "kevT_dev": {
+          "n": 764,
+          "correct": 488,
+          "accuracy": 0.6387434554973822
+        },
+        "kevT_test": {
+          "n": 764,
+          "correct": 506,
+          "accuracy": 0.662303664921466
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 57,
+          "accuracy": 0.7916666666666666
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 42,
+          "accuracy": 0.3783783783783784
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 147,
+          "accuracy": 0.6363636363636364
+        }
+      }
+    },
+    {
+      "id": "kev-4b",
+      "name": "Kev 4B (published)",
+      "parameters_b_nominal": 4,
+      "family": "External typed-decision model; internals not audited",
+      "base": null,
+      "provenance": "benchmark-author per-item outcomes, JevBench v1.2 snapshot",
+      "metrics": {
+        "jb_public": {
+          "n": 231,
+          "correct": 153,
+          "accuracy": 0.6623376623376623
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 64,
+          "accuracy": 0.8888888888888888
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 41,
+          "accuracy": 0.36936936936936937
+        }
+      }
+    },
+    {
+      "id": "kev-8b",
+      "name": "Kev 8B (published)",
+      "parameters_b_nominal": 8,
+      "family": "External typed-decision model; internals not audited",
+      "base": null,
+      "provenance": "benchmark-author per-item outcomes, JevBench v1.2 snapshot",
+      "metrics": {
+        "jb_public": {
+          "n": 231,
+          "correct": 165,
+          "accuracy": 0.7142857142857143
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 67,
+          "accuracy": 0.9305555555555556
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 50,
+          "accuracy": 0.45045045045045046
+        }
+      }
+    },
+    {
+      "id": "jev-1.13.0",
+      "name": "Jev 1.13 (published)",
+      "parameters_b_nominal": null,
+      "family": "External typed-decision model; internals not audited",
+      "base": null,
+      "provenance": "benchmark-author per-item outcomes, JevBench v1.2 snapshot",
+      "metrics": {
+        "jb_public": {
+          "n": 231,
+          "correct": 200,
+          "accuracy": 0.8658008658008658
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 71,
+          "accuracy": 0.9861111111111112
+        },
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 81,
+          "accuracy": 0.7297297297297297
+        }
+      }
+    },
+    {
+      "id": "jev_free",
+      "name": "Jev 1.13 Free (local API run)",
+      "parameters_b_nominal": null,
+      "family": "Typed API",
+      "base": null,
+      "provenance": "local API responses; different prompting/compute from pointer models",
+      "metrics": {
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 71,
+          "accuracy": 0.9861111111111112
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 75,
+          "accuracy": 0.6756756756756757
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 194,
+          "accuracy": 0.8398268398268398
+        }
+      }
+    },
+    {
+      "id": "solar",
+      "name": "Solar Pro4 (local API run)",
+      "parameters_b_nominal": null,
+      "family": "Generative API",
+      "base": null,
+      "provenance": "local API responses; different prompting/compute from pointer models",
+      "metrics": {
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 71,
+          "accuracy": 0.9861111111111112
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 101,
+          "accuracy": 0.9099099099099099
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 220,
+          "accuracy": 0.9523809523809523
+        }
+      }
+    },
+    {
+      "id": "qwen3_4_reason",
+      "name": "Qwen3 4B reasoning",
+      "parameters_b_nominal": 4,
+      "family": "Generative local reasoning",
+      "base": "Qwen3-4B",
+      "provenance": "local reasoning run; different token budget and output path",
+      "metrics": {
+        "jb_easy": {
+          "n": 48,
+          "correct": 48,
+          "accuracy": 1.0
+        },
+        "jb_original": {
+          "n": 72,
+          "correct": 68,
+          "accuracy": 0.9444444444444444
+        },
+        "jb_hard": {
+          "n": 111,
+          "correct": 73,
+          "accuracy": 0.6576576576576577
+        },
+        "jb_public": {
+          "n": 231,
+          "correct": 189,
+          "accuracy": 0.8181818181818182
+        }
+      }
+    }
+  ],
+  "trainable_parameters": {
+    "4": {
+      "nominal_base_b": 4,
+      "lora_parameters": 30474240,
+      "pointer_parameters": 2561,
+      "rank": 16,
+      "alpha": 32,
+      "base": "Qwen/Qwen3.5-4B-Base",
+      "artifact_hashes": {
+        "adapter_model.safetensors": "7b165d1d0d57a29e9c9ae39f639f77edc1ef8d2084174eb2afcc364eb8f00055",
+        "pointer_head.safetensors": "7e9e1f0f94d00ad5835d7829c22ef74827f4a95dcbf335e9ceac3a6d60bd9d50",
+        "adapter_config.json": "50ba2db8ff05e6d6a1f8bdecd8005a3f76e8cf8e926ec84e3d7e5666d07ffc11"
+      }
+    },
+    "9": {
+      "nominal_base_b": 9,
+      "lora_parameters": 40108032,
+      "pointer_parameters": 4097,
+      "rank": 16,
+      "alpha": 32,
+      "base": "Qwen/Qwen3.5-9B-Base",
+      "artifact_hashes": {
+        "adapter_model.safetensors": "7231ef52a482760601bd6e9038ec9aae40034d0f63c6fb5de677b5f81e6dd5c2",
+        "pointer_head.safetensors": "f534d840c1b521c04292e763db200e6ab10df12fb4ebdabc78de734c07d88e4a",
+        "adapter_config.json": "96f5ae3d77c1c0ef156344c0664cda89bd3a1449ca2f71d398744a2dc073d2d5"
+      }
+    }
+  },
+  "matched_protocol": {
+    "dtype": "NF4/BF16",
+    "gpu": "RTX 3090",
+    "max_input_tokens": 4096,
+    "batch_token_budget": 4096,
+    "tta": 1
+  },
+  "sources": [
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/comparison_20261005/metrics.json",
+      "sha256": "8a7665f6b490259213cf1ea9a82651ec1e470fa71177a93d84a1138dfa00675a"
+    },
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/multi/FINAL_qwen3-4b_e40.json",
+      "sha256": "5b1f9a0fe1cc8ddb606bad84d55bf3e8474d1e77af4b59ffff5d86d7d3f8a967"
+    },
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/pointer/FINAL_H_q3_4b_headonly.json",
+      "sha256": "be7765ceaaf1aa71fb8210832da40ceba48f776e3ed0067df934bae1f81b81a1"
+    },
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/pointer/FINAL_qwen3-4b.json",
+      "sha256": "853d842cbe2d96bfeaad0b3879863e89878655dae41615b0848ef226fd069ab8"
+    },
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/pointer/FINAL_B_qwen3-8b-2ep.json",
+      "sha256": "bc80e97bd672812f442db984852484ba84ba2769b4ccb27a1844c660c659f29f"
+    },
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/pointer/FINAL_P_q35_2b_e1_bf16.json",
+      "sha256": "2ef9456fb45f8cfe73e8d6ce5a8824285d532a33e441f45df59e2ae1157d267a"
+    },
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/jev_paired/jev_paired.json",
+      "sha256": "cc82c78affa57db07373cdc173ddc4630a3d5efc94c3d089c8cecc08ae463a04"
+    },
+    {
+      "source": "experiments/116_api_vs_midm_20261003/free_model_catalog/comparison_latest.json",
+      "sha256": "4fa24e256c0dd910deddca300bfd9d093e0fbbb54b7a1c12033409d51b703c7d"
+    },
+    {
+      "source": "experiments/116_api_vs_midm_20261003/router_pilot_20261005/RESULTS.json",
+      "sha256": "2f78ad47cf8ee4f01e6e88dc346de8792c159af17eb9550a0e4dcf891a0fa044"
+    },
+    {
+      "source": "research_topics/clm_decision_heads_novelty_20260928/huggingface/MiDM-4B-q35-e1-bx/adapter_config.json",
+      "sha256": "50ba2db8ff05e6d6a1f8bdecd8005a3f76e8cf8e926ec84e3d7e5666d07ffc11"
+    },
+    {
+      "source": "research_topics/clm_decision_heads_novelty_20260928/huggingface/MiDM-9B-q35-e1-bx/adapter_config.json",
+      "sha256": "96f5ae3d77c1c0ef156344c0664cda89bd3a1449ca2f71d398744a2dc073d2d5"
+    },
+    {
+      "source": "experiments/113_clm_reproduction_20260928/results/benchmark_atlas_20261005/upstream_inventory.json",
+      "sha256": "516c617f6ee0ddf4253cd816a85a3f9514d7c2e2bb9b46950009506327e8563a"
+    }
+  ]
+}

benchmarks/v0.2.1/plot_atlas.py

@@ -0,0 +1,94 @@
+"""Reproduce the public figures from aggregate atlas.json; no network or model needed."""
+from pathlib import Path
+import json, textwrap
+import numpy as np
+import matplotlib
+matplotlib.use('Agg')
+import matplotlib.pyplot as plt
+from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
+
+ROOT=Path(__file__).resolve().parent
+D=json.loads((ROOT/'atlas.json').read_text(encoding='utf-8'))
+MODELS={r['id']:r for r in D['models']}
+TEAL='#007f86';BLUE='#395a9e';ORANGE='#c56a23';PURPLE='#8667a7';GRAY='#657580'
+plt.rcParams.update({'font.family':'DejaVu Sans','font.size':10,'axes.spines.top':False,'axes.spines.right':False,'axes.labelcolor':'#283a47','text.color':'#283a47','axes.titleweight':'bold','svg.fonttype':'none','savefig.facecolor':'white'})
+
+def save(fig,name):
+ for ext in ['png','svg']:
+  p=ROOT/(name+'.'+ext)
+  fig.savefig(p,dpi=170,bbox_inches='tight',**({'metadata':{'Date':None}} if ext=='svg' else {}))
+  if ext=='svg':p.write_bytes(('\n'.join(line.rstrip() for line in p.read_text(encoding='utf-8').splitlines())+'\n').encode('utf-8'))
+ plt.close(fig)
+
+def matched():
+ fig,axes=plt.subplots(2,2,figsize=(12,8));fig.suptitle('MiDM: nominal base parameters vs matched decision accuracy',fontsize=17,y=.98)
+ for ax,(key,title) in zip(axes.flat,[('td_test','Typed Decisions test · N=2000'),('kevT_test','Kev transfer test · N=764'),('jb_public','JevBench public · N=231'),('jb_hard','JevBench hard · N=111')]):
+  y=[100*MODELS[m]['metrics'][key]['accuracy'] for m in ['midm_4b','midm_9b']]
+  ax.plot([4,9],y,color=TEAL,linewidth=1.5,linestyle='--',zorder=2);ax.scatter([4,9],y,c=[BLUE,TEAL],s=95,zorder=3)
+  for x,v in zip([4,9],y):ax.annotate(f'{v:.2f}%',(x,v),xytext=(0,11),textcoords='offset points',ha='center',weight='bold')
+  ax.set(title=title,xlim=(2.7,10.3),ylim=(0,103),xticks=[4,9],xlabel='Nominal base parameters (billions)',ylabel='Hard-label accuracy (%)');ax.grid(axis='y',alpha=.16)
+  ax.text(.5,.1,f'9B minus 4B: +{y[1]-y[0]:.2f} percentage points',ha='center',transform=ax.transAxes,fontsize=10)
+ fig.subplots_adjust(left=.085,right=.97,bottom=.13,top=.89,wspace=.24,hspace=.45)
+ fig.text(.08,.035,'Same fixed items, NF4/BF16, RTX 3090, context 4096, TTA=1. Retrospective point estimates.\nJevBench public accuracy is not the official composite. Connecting lines are not fitted scaling laws.',fontsize=10,color=GRAY)
+ save(fig,'matched_size_performance')
+
+def context():
+ ids=['midm_q35_2','clm_style4','pointer_frozen4','midm_q3_4','midm_q3_8','midm_4b','midm_9b','kev-4b','kev-8b','qwen3_4_reason']
+ label={'midm_q35_2':'MiDM Q3.5 2B','clm_style4':'CLM-style 4B','pointer_frozen4':'Frozen pointer 4B','midm_q3_4':'MiDM Q3 4B','midm_q3_8':'MiDM Q3 8B','midm_4b':'MiDM Q3.5 4B','midm_9b':'MiDM Q3.5 9B','kev-4b':'Kev 4B [published]','kev-8b':'Kev 8B [published]','qwen3_4_reason':'Qwen3 4B reasoning'}
+ offsets=[{'midm_q35_2':(.8,58),'clm_style4':(4.3,36),'pointer_frozen4':(4.3,48),'midm_q3_4':(2.2,77),'midm_q3_8':(6.2,66),'midm_4b':(4.3,73),'midm_9b':(7.1,85),'kev-4b':(1,67),'kev-8b':(6,79),'qwen3_4_reason':(4.3,88)},
+ {'midm_q35_2':(.8,49),'clm_style4':(2.4,23),'pointer_frozen4':(4.3,30),'midm_q3_4':(4.3,42),'midm_q3_8':(6.4,36),'midm_4b':(4.3,52),'midm_9b':(7,62),'kev-4b':(1,32),'kev-8b':(6.2,49),'qwen3_4_reason':(4.3,71)}]
+ fig,axes=plt.subplots(1,2,figsize=(16,7));fig.suptitle('Parameter context: local measurements and published comparison rows',fontsize=17,y=.97)
+ for j,(ax,key,title) in enumerate(zip(axes,['jb_public','jb_hard'],['Same public 231 items','Public hard tier · 111 items'])):
+  for id in ids:
+   r=MODELS[id];x=r['parameters_b_nominal'];y=r['metrics'][key]['accuracy']*100
+   published=id.startswith('kev-');primary=id in ['midm_4b','midm_9b'];reason=id=='qwen3_4_reason'
+   color=PURPLE if published else ORANGE if reason else TEAL if primary else GRAY
+   marker='s' if published else 'D' if reason else 'o' if primary else '^'
+   ax.scatter(x,y,s=75,marker=marker,c=color,zorder=3)
+   ax.annotate(label[id],(x,y),xytext=offsets[j][id],textcoords='data',ha='left',fontsize=8.5,color=color,arrowprops={'arrowstyle':'-','color':color,'lw':.5})
+  ax.set(title=title,xlabel='Nominal base parameters (billions)',ylabel='Hard-label accuracy (%)',xlim=(.5,10.7),ylim=(0,103),xticks=[2,4,8,9]);ax.grid(alpha=.12)
+ fig.subplots_adjust(left=.06,right=.97,bottom=.2,top=.86,wspace=.20)
+ fig.text(.06,.04,'Teal circles: matched MiDM 4B/9B. Gray triangles: historical local settings. Purple squares: benchmark-published Kev.\nOrange diamonds: local generative reasoning. Bases, precision, data, epochs and compute differ across these groups.\nUnknown-size Jev and Solar are excluded here, not assigned an estimated parameter count. No architecture-only or scaling-law claim.',fontsize=10,color=GRAY)
+ save(fig,'context_size_performance')
+
+def comparisons():
+ rows=sorted(D['models'],key=lambda r:r['metrics']['jb_public']['accuracy']);labels=[r['name'] for r in rows]
+ fig,axes=plt.subplots(1,2,figsize=(15,8.5),sharey=True);fig.suptitle('Public JevBench cohort: capability comparison, not equal compute',fontsize=17,y=.97)
+ for ax,key,title in zip(axes,['jb_public','jb_hard'],['Public accuracy · 231 items','Hard accuracy · 111 items']):
+  vals=[r['metrics'][key]['accuracy']*100 for r in rows]
+  colors=[TEAL if r['id'] in ['midm_4b','midm_9b'] else PURPLE if 'published' in r['name'] else ORANGE if r['id'] in ['solar','jev_free','qwen3_4_reason'] else GRAY for r in rows]
+  ax.barh(range(len(rows)),vals,color=colors,height=.66)
+  for i,v in enumerate(vals):ax.text(v+1,i,f'{v:.2f}%',va='center',fontsize=9)
+  ax.set(title=title,xlim=(0,112),xlabel='Hard-label accuracy (%)',yticks=range(len(rows)),yticklabels=labels);ax.grid(axis='x',alpha=.14);ax.set_axisbelow(True)
+ axes[0].tick_params(axis='y',labelsize=9)
+ fig.subplots_adjust(left=.25,right=.96,bottom=.15,top=.88,wspace=.15)
+ fig.text(.05,.045,'Teal: matched MiDM. Gray: historical local. Purple: benchmark-published outcomes. Orange: local API/reasoning runs.\nAll rows cover the same 231 public items, but service versions, prompts, training exposure and computation differ.\nJev published and Jev Free measured are separate observations. Incomplete cohorts are excluded. Not the official composite.',fontsize=10,color=GRAY)
+ save(fig,'public_comparison')
+
+def architecture():
+ fig,ax=plt.subplots(figsize=(16,7));ax.set(xlim=(0,16),ylim=(0,7));ax.axis('off');fig.suptitle('Architecture comparison: how a bounded decision is produced',fontsize=18,y=.96)
+ lanes=[(5.3,'CLM reference 8B',PURPLE,['State and candidates\nencoded separately','Frozen Qwen3-8B\nlast-token embeddings','Two projection MLPs\nstate / action','Scaled cosine score\nrank / softmax']),
+ (3.35,'MiDM 4B / 9B',TEAL,['State + question +\nordered option lines','Causal base in NF4\n+ rank-16 LoRA','Hidden state at\neach option end','Shared linear pointer\nsoftmax over options']),
+ (1.4,'Generative reasoning',ORANGE,['State + question +\ncandidate prompt','Autoregressive decoder\nreasoning / answer tokens','Parse generated\nanswer or tool output','Typed choice\nwith parser checks'])]
+ for y,title,color,boxes in lanes:
+  ax.text(.15,y+.62,title,fontsize=12,weight='bold',color=color)
+  for j,text in enumerate(boxes):
+   x=.15+j*4.02;box=FancyBboxPatch((x,y-.45),3.55,.82,boxstyle='round,pad=0.08',edgecolor=color,facecolor=color+'12',linewidth=1.4);ax.add_patch(box);ax.text(x+1.775,y-.035,text,ha='center',va='center',fontsize=10.5)
+   if j<3:ax.add_patch(FancyArrowPatch((x+3.65,y-.035),(x+3.9,y-.035),arrowstyle='-|>',mutation_scale=14,color=color))
+  note={'CLM reference 8B':'Independent embedding reuse is possible; released task-specific heads are separate artifacts.','MiDM 4B / 9B':'One causal pass: later options see earlier options, not vice versa. This is not bidirectional attention.','Generative reasoning':'More decoding work; a higher accuracy does not imply equivalent latency, cost or parameter disclosure.'}[title]
+  ax.text(.15,y-.8,note,color=GRAY,fontsize=9.5)
+ fig.subplots_adjust(left=.025,right=.985,bottom=.03,top=.91);save(fig,'architecture')
+
+def control():
+ ids=['clm_style4','pointer_frozen4','midm_q3_4'];names=['Independent frozen\nprojection heads','Frozen causal\npointer head','Causal pointer\n+ QLoRA'];x=np.arange(3);fig,ax=plt.subplots(figsize=(11,6.5))
+ for delta,key,name,color in [(-.18,'td_test','Typed Decisions test (2000)',BLUE),(.18,'kevT_test','Kev transfer test (764)',TEAL)]:
+  vals=[100*MODELS[i]['metrics'][key]['accuracy'] for i in ids];bars=ax.bar(x+delta,vals,.34,label=name,color=color)
+  ax.bar_label(bars,labels=[f'{v:.2f}%' for v in vals],padding=4,fontsize=11)
+ ax.set(title='Historical Qwen3-4B controls: input interaction and adaptation',ylabel='Hard-label accuracy (%)',ylim=(0,104),xticks=x,xticklabels=names);ax.grid(axis='y',alpha=.15);ax.set_axisbelow(True);ax.legend(loc='upper left',frameon=False)
+ fig.subplots_adjust(left=.08,right=.97,bottom=.24,top=.88)
+ fig.text(.08,.045,'Same nominal base family/size and original task mix; training recipes and recorded precision are not fully matched.\nThis is a historical design control, not an architecture-only causal estimate and not the released CLM 8B.\nPrimary Qwen3.5 4B/9B comparisons are shown separately.',fontsize=10,color=GRAY)
+ save(fig,'architecture_control')
+
+if __name__=='__main__':
+ for f in [matched,context,comparisons,architecture,control]:f()
+ print('Wrote five figures as PNG and SVG from atlas.json.')

benchmarks/v0.2.1/upstream_inventory.json

@@ -0,0 +1,80 @@
+{
+  "checked_at": "2026-10-05T23:11:58.401173",
+  "sources": [
+    {
+      "url": "https://huggingface.co/api/models?author=Contrastive-LM&limit=100",
+      "sha": null,
+      "ids": [
+        "Contrastive-LM/CLM-v0.1-8B",
+        "Contrastive-LM/deepswe-clm-heads-8k"
+      ],
+      "file_count": 0,
+      "evaluation_related_files": []
+    },
+    {
+      "url": "https://huggingface.co/api/models/Contrastive-LM/CLM-v0.1-8B",
+      "sha": "e939398d4556fcd9400c76fa8c5a513202f42b0a",
+      "ids": null,
+      "file_count": 6,
+      "evaluation_related_files": []
+    },
+    {
+      "url": "https://huggingface.co/api/models/Contrastive-LM/deepswe-clm-heads-8k",
+      "sha": "c60876f3fdf7a75dc58d33b776e469b7e903d0ee",
+      "ids": null,
+      "file_count": 9,
+      "evaluation_related_files": [
+        "heldout_tasks.json",
+        "split_summary.json",
+        "task_split.json",
+        "verification.json"
+      ]
+    },
+    {
+      "url": "https://huggingface.co/api/datasets?author=Contrastive-LM&limit=100",
+      "sha": null,
+      "ids": [
+        "Contrastive-LM/deepswe-clm-train-embeddings-8k",
+        "Contrastive-LM/CLM-v0.1-Pretrain-Nemotron"
+      ],
+      "file_count": 0,
+      "evaluation_related_files": []
+    },
+    {
+      "url": "https://huggingface.co/api/datasets/Contrastive-LM/deepswe-clm-train-embeddings-8k",
+      "sha": "4e42267114a2b28dc1d4fab85a2fe2790abc16d2",
+      "ids": null,
+      "file_count": 18,
+      "evaluation_related_files": []
+    },
+    {
+      "url": "https://huggingface.co/api/datasets/Contrastive-LM/CLM-v0.1-Pretrain-Nemotron",
+      "sha": "05ca7d05c043350326c5e86c589b9542962747b4",
+      "ids": null,
+      "file_count": 1876,
+      "evaluation_related_files": []
+    },
+    {
+      "url": "https://api.github.com/repos/Contrastive-LM/CLM/git/trees/main?recursive=1",
+      "sha": "bb42c6c5bf914fd449bed2f6ca65be80602cb1f7",
+      "ids": null,
+      "file_count": 46,
+      "evaluation_related_files": [
+        "evaluation/bon_eval.py"
+      ]
+    }
+  ],
+  "local_deepswe_columns": [
+    "trajectory_id",
+    "step_idx",
+    "task_id",
+    "model",
+    "config",
+    "reward",
+    "fold",
+    "prm_score",
+    "state_embedding",
+    "action_embedding"
+  ],
+  "finding": "No matching raw DeepSWE candidate text or Terminal-Bench 30-task verifier pool found in these inspected releases. This is a scoped search, not proof that no such assets exist elsewhere."
+}

manifest.json

@@ -1,18 +1,33 @@
-{
-  "version": "0.2.0",
-  "model": "MiDM-9B-q35-e1-bx",
-  "files": {
-    "adapter_config.json": "96f5ae3d77c1c0ef156344c0664cda89bd3a1449ca2f71d398744a2dc073d2d5",
-    "adapter_model.safetensors": "7231ef52a482760601bd6e9038ec9aae40034d0f63c6fb5de677b5f81e6dd5c2",
-    "evaluation.json": "a3684eef4f8a678a4d821319eff403f83e1dd45db6b432f63e956f8a8d20f746",
-    "LICENSE": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
-    "midm.py": "2a3985c09b38802d240acec138b507358ee4bd30e413c82dd3215ee7ac0681fd",
-    "midm_config.json": "fe75b3e6b3eab1e9f3a9e9ec9475d4c5551ca9d57473320838a51c1affd8b077",
-    "NOTICE": "8c1e780a6af2a8c5402c719d95bd3be4ebd5f05c7fce0f9def1cff665c4ca21d",
-    "package_validation.json": "efb81406f22b909db9a77fbce5c3b3fe80af8201c5cda6abff11d31077957470",
-    "pointer_head.safetensors": "f534d840c1b521c04292e763db200e6ab10df12fb4ebdabc78de734c07d88e4a",
-    "README.md": "072ddecdfb2c60ebf6232f39567e9194e47bed6f7f5cd7cea084abf5b3940cdd",
-    "requirements.txt": "550b986e177232471ed3dee7efacb0d8b0dacf0cbc7044b9b81a58dc38d808c2",
-    "VERIFICATION.md": "4459ba20a9f9447df6ff205b5dfce778a18a1d64ca25614dec500a828f2e7b89"
-  }
-}
+{
+  "version": "0.2.1",
+  "model": "MiDM-9B-q35-e1-bx",
+  "files": {
+    "adapter_config.json": "96f5ae3d77c1c0ef156344c0664cda89bd3a1449ca2f71d398744a2dc073d2d5",
+    "adapter_model.safetensors": "7231ef52a482760601bd6e9038ec9aae40034d0f63c6fb5de677b5f81e6dd5c2",
+    "evaluation.json": "a3684eef4f8a678a4d821319eff403f83e1dd45db6b432f63e956f8a8d20f746",
+    "LICENSE": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
+    "midm.py": "2a3985c09b38802d240acec138b507358ee4bd30e413c82dd3215ee7ac0681fd",
+    "midm_config.json": "fe75b3e6b3eab1e9f3a9e9ec9475d4c5551ca9d57473320838a51c1affd8b077",
+    "NOTICE": "8c1e780a6af2a8c5402c719d95bd3be4ebd5f05c7fce0f9def1cff665c4ca21d",
+    "package_validation.json": "efb81406f22b909db9a77fbce5c3b3fe80af8201c5cda6abff11d31077957470",
+    "pointer_head.safetensors": "f534d840c1b521c04292e763db200e6ab10df12fb4ebdabc78de734c07d88e4a",
+    "README.md": "b392455d328c9af67f162394beeef4d1c230d5d5652fda4e94023d51af15f75d",
+    "requirements.txt": "550b986e177232471ed3dee7efacb0d8b0dacf0cbc7044b9b81a58dc38d808c2",
+    "VERIFICATION.md": "4459ba20a9f9447df6ff205b5dfce778a18a1d64ca25614dec500a828f2e7b89",
+    "benchmarks/v0.2.1/architecture.png": "cc5bfb7fc225b1f974837e36ef43717af271fe1d56281f6faa8790f4d1762280",
+    "benchmarks/v0.2.1/architecture.svg": "0d77746badfff06e4b07f686753830f565568e98ad6b88dfa3e6e179c413bba7",
+    "benchmarks/v0.2.1/architecture_control.png": "18e6afac826bda17ed80a7f0252f63c69109aeeca41026efbc50d01845b52746",
+    "benchmarks/v0.2.1/architecture_control.svg": "d837c9733a0e3af6c4b7c2078a940af18f3d74c6bb5871ff2231f4525c89b661",
+    "benchmarks/v0.2.1/atlas.json": "3d82078bac0024ac3fc4a814580a7fd11528c6f659acda99099e8a3703e70c4a",
+    "benchmarks/v0.2.1/context_size_performance.png": "62364f4bdfd3a00358bf2631a03d550b86d75b62b0682e936a7782d30eda5588",
+    "benchmarks/v0.2.1/context_size_performance.svg": "ef50c39dc7e19f0f289064ae41baf77889ca8fa48f656a4f92f4b9d8cd06c7c3",
+    "benchmarks/v0.2.1/matched_size_performance.png": "6e5b233ccfca43b71bd4f2884d38881751628db14369a035eac97ac5995b8729",
+    "benchmarks/v0.2.1/matched_size_performance.svg": "ea0fbc1d8c4658254c8be3a4e37094262877c404b4357e4d1e65fb2e4c07d83b",
+    "benchmarks/v0.2.1/plot_atlas.py": "2081011cfd8cef64f0f16887f5bf5df4fb1ba71c9a2da31e640e141ee7b03254",
+    "benchmarks/v0.2.1/public_comparison.png": "2ce2d9502232d5683e840afa3e97a3b203665b31b9dec6fa63d4fc3b1576aa87",
+    "benchmarks/v0.2.1/public_comparison.svg": "921ef3941ca10f90acce98bed502e95caaa143a6eb6ad3a706765cd36d3dcbfc",
+    "benchmarks/v0.2.1/README.md": "0768c6ea91f2ce2e218e268d4e7c3fb9a943e6070464976f0be57bd4555fe2b7",
+    "benchmarks/v0.2.1/upstream_inventory.json": "2515bc93354d78242e0df50aef7130ed67d19190d53384daf1fcd0746e3c1186"
+  },
+  "weights_version": "0.2.0"
+}