README: add short-sequence cells, update headline

README.md

@@ -14,7 +14,7 @@ library_name: transformers
 
 **Pretrained encoders for fine-grained file-content-type detection — on any 4 KB byte window.**
 
-A family of 28 small (3.15–37.8 M backbone parameter) BERT-style encoders pretrained MLM-only on 33 GB of heterogeneous binary content for classification under [libmagic](https://github.com/file/file)'s 125-class MIME taxonomy.
+A family of 36 small (3.15–37.8 M backbone parameter) BERT-style encoders pretrained MLM-only on 33 GB of heterogeneous binary content for classification under [libmagic](https://github.com/file/file)'s 125-class MIME taxonomy. 28 parent-cube cells at `seq_len=1024` (4 KB byte windows) plus an 8-cell short-sequence extension at `seq_len=256` (1 KB byte windows) sized for sub-MTU packets, DNS payloads, and small forensic fragments.
 
 Training samples 1024-token windows uniformly at random across files and 64 KB fragments, with no privileged "head-of-file" position. A single checkpoint classifies any 4 KB byte window: a streaming HTTP body before upload completes, a forensic-carved fragment with no recoverable header, a random seek into a multi-gigabyte container, or a packet payload inspected mid-stream.
 
@@ -31,9 +31,13 @@ What's your input?
 │   └─→ Magika is purpose-built for this. Reach for MimeLens only if
 │       libmagic's 125-class taxonomy is required.
 │
-├── A partial / streaming / packet-payload / random-offset chunk
+├── A partial / streaming / packet-payload / random-offset chunk (≥ 1 KB)
 │   └─→ mimelens-001-medium-byte-s1     (saturates from a single 1.4 KB packet)
 │
+├── A sub-KB chunk (sub-MTU packet, DNS payload, small fragment, ≤ 1 KB total)
+│   ├─→ mimelens-001-medium-bpe-64k-s1-seq256  (best short-sequence accuracy: 0.985 4 KB-head / 0.981 256 B-head, ~10× the throughput of the parent cell)
+│   └─→ mimelens-001-medium-bpe-16k-s1-seq256  (ONNX bundled; same family, slightly lower accuracy)
+│
 ├── A clean 4 KB head and you want libmagic-style fine-grained MIME labels
 │   ├─→ mimelens-001-medium-bpe-16k-s1  (recommended default; balances accuracy + adversarial robustness)
 │   └─→ mimelens-001-medium-byte-s1     (essentially tied under clean conditions)
@@ -57,9 +61,9 @@ These are the three cells the paper presents as the deployable system. Each is o
 
 All three load via `AutoModel.from_pretrained(..., trust_remote_code=True)`. See the per-cell READMEs for the copy-pasteable inference snippet.
 
-## All 28 released cells
+## All released cells
 
-Every cell of the pre-registered 3 × 4 × {2,3} factorial cube is published; numbers below are this-cell magic-frags 4 KB-head top-1 / macro-F1 / kNN R@1.
+Every cell of the pre-registered 3 × 4 × {2,3} factorial parent cube is published, plus an 8-cell short-sequence extension at the `medium` tier (seq_len=256) and one matched-tokens ablation. Numbers in the parent-cube tables are this-cell `magic-frags` 4 KB-head top-1 / macro-F1 / kNN R@1 — the within-cube benchmark applied identically to all 28 parent-cube cells (short-sequence numbers use `magic-files` probe-fit; see that subsection). The `medium/bpe-16k/s1` headline calibration numbers against Magika (`0.828` strict / `0.829` aligned / `0.927` top-level on the `magic-files` n=1,024 held-out split) are shown in **Headline findings** below.
 
 ### medium (37.76 M backbone params; the recommended size for deployment)
 
@@ -105,6 +109,23 @@ Every cell of the pre-registered 3 × 4 × {2,3} factorial cube is published; nu
 | `tiny/bpe-64k/s1`   | 0.715   | 0.620   | 0.671   | [link](https://huggingface.co/mjbommar/mimelens-001-tiny-bpe-64k-s1) |
 | `tiny/bpe-64k/s2`   | 0.732   | 0.609   | 0.675   | [link](https://huggingface.co/mjbommar/mimelens-001-tiny-bpe-64k-s2) |
 
+### medium short-sequence (seq_len=256, for sub-MTU packets and small forensic fragments)
+
+Matched-steps to the parent cube (22,888 gradient updates, same architecture, optimizer, schedule). Numbers are `magic-files` 4 KB-head probe-fit top-1 (left) and 256 B-head probe-fit top-1 (right; the design regime). BPE cells preserve or exceed parent accuracy at 4× lower per-step token budget; the byte cell pays ~1 pp.
+
+| Cell | 4 KB head | 256 B head | Repo |
+|---|---|---|---|
+| `medium/byte/s1-seq256`        | 0.947 | 0.947 | [link](https://huggingface.co/mjbommar/mimelens-001-medium-byte-s1-seq256) |
+| `medium/byte/s2-seq256`        | 0.943 | 0.943 | [link](https://huggingface.co/mjbommar/mimelens-001-medium-byte-s2-seq256) |
+| `medium/bpe-4k/s1-seq256`      | 0.971 | 0.967 | [link](https://huggingface.co/mjbommar/mimelens-001-medium-bpe-4k-s1-seq256) |
+| `medium/bpe-4k/s2-seq256`      | 0.972 | 0.967 | [link](https://huggingface.co/mjbommar/mimelens-001-medium-bpe-4k-s2-seq256) |
+| **`medium/bpe-16k/s1-seq256`** | **0.980** | **0.974** | [link](https://huggingface.co/mjbommar/mimelens-001-medium-bpe-16k-s1-seq256) (ONNX bundled) |
+| `medium/bpe-16k/s2-seq256`     | 0.981 | 0.975 | [link](https://huggingface.co/mjbommar/mimelens-001-medium-bpe-16k-s2-seq256) |
+| `medium/bpe-64k/s1-seq256`     | 0.987 | 0.983 | [link](https://huggingface.co/mjbommar/mimelens-001-medium-bpe-64k-s1-seq256) |
+| `medium/bpe-64k/s2-seq256`     | 0.986 | 0.979 | [link](https://huggingface.co/mjbommar/mimelens-001-medium-bpe-64k-s2-seq256) |
+
+Per-vocab seed means at 4 KB head: byte 0.945, bpe-4k 0.971, bpe-16k 0.980, bpe-64k 0.987 (vs parent cube 0.955 / 0.973 / 0.977 / 0.975 at the same probe-fit metric, matched-steps).
+
 ## Headline findings (from the paper)
 
 1. **Calibrated against Magika v1.1** on the same n=1,024 held-out 4,096-file split, libmagic-pinned ground truth: `medium/bpe-16k/s1` exceeds Magika at every level of stringency. Strict top-1: 0.828 vs 0.653 (+17.5 pp). Aligned under a curated 21-class equivalence map applied symmetrically to both systems: 0.829 vs 0.722 (+10.7 pp). Top-level (text vs image vs application vs …): 0.927 vs 0.840 (+8.7 pp). The aligned gap is the residual under this map on this corpus; what would persist under a hypothetically retrained Magika is open.
@@ -115,7 +136,7 @@ Every cell of the pre-registered 3 × 4 × {2,3} factorial cube is published; nu
 
 4. **Truly random-offset disk-block classification** (Section 6): a 1 GB unmounted `ext4` image populated with 3,066 MIME-balanced files; 1,000 random 4 KB block reads. On 980 mid-file blocks, all three medium cells exceed both libmagic and Magika with non-overlapping file-level cluster-bootstrap CIs: `medium/bpe-64k/s1` 0.266, `medium/bpe-16k/s1` 0.220, `medium/byte/s1` 0.219, vs libmagic 0.093 / Magika 0.112. Replicates across 9 matrix cells (ext4 × 4 init-strategies + 2 size-stratified sub-cells; NTFS × 3 init-strategies).
 
-5. **CPU latency.** Idle CPU, single sample, p50: PyTorch fp32 392 ms; ONNX int8 547 ms (int8 is slower than fp32 on this hardware without AVX-VNNI). Magika v1.1 on the same CPU: 1.58 ms/sample. MimeLens occupies a different point on the deployment surface than Magika, not a drop-in replacement.
+5. **CPU latency.** Idle CPU, single sample, p50: parent-cube PyTorch fp32 392 ms; ONNX int8 547 ms (int8 is slower than fp32 on this hardware without AVX-VNNI). Magika v1.1 on the same CPU: 1.58 ms/sample. The short-sequence `medium/bpe-16k/s1-seq256` cell with ONNX int8 closes that gap to **~10× end-to-end** (50 ms vs Magika's 5 ms on the same 500-file bench); ship it where sub-second per-sample inference matters. MimeLens occupies a different point on the deployment surface than Magika, not a drop-in replacement.
 
 Full evaluation (within-cube bootstrap CIs at n=3 medium seeds, calibration, per-class breakdown, network curves, baseline comparisons against libmagic 5.46 and TrID 2.24, byte-coverage matched ablation, pre-registration log) is in the [paper](https://github.com/mjbommar/binary-embedding-paper).
 

eval_summary.json

@@ -1,13 +1,14 @@
 {
   "release": "mimelens-001",
-  "n_cells": 29,
+  "n_cells": 37,
   "cells": {
     "tiny/byte/s1": {
       "cell_id": "tiny/byte/s1",
       "size": "tiny",
       "vocab": "byte",
       "seed": 1,
-      "vocab_size": 261,
+      "seq_len": 1024,
+      "vocab_size": 263,
       "params_m": 3.15,
       "layers": 4,
       "hidden_size": 256,
@@ -24,7 +25,8 @@
       "size": "tiny",
       "vocab": "byte",
       "seed": 2,
-      "vocab_size": 261,
+      "seq_len": 1024,
+      "vocab_size": 263,
       "params_m": 3.15,
       "layers": 4,
       "hidden_size": 256,
@@ -41,7 +43,8 @@
       "size": "tiny",
       "vocab": "bpe-4k",
       "seed": 1,
-      "vocab_size": 4101,
+      "seq_len": 1024,
+      "vocab_size": 4103,
       "params_m": 3.15,
       "layers": 4,
       "hidden_size": 256,
@@ -58,7 +61,8 @@
       "size": "tiny",
       "vocab": "bpe-4k",
       "seed": 2,
-      "vocab_size": 4101,
+      "seq_len": 1024,
+      "vocab_size": 4103,
       "params_m": 3.15,
       "layers": 4,
       "hidden_size": 256,
@@ -75,6 +79,7 @@
       "size": "tiny",
       "vocab": "bpe-16k",
       "seed": 1,
+      "seq_len": 1024,
       "vocab_size": 16391,
       "params_m": 3.15,
       "layers": 4,
@@ -92,6 +97,7 @@
       "size": "tiny",
       "vocab": "bpe-16k",
       "seed": 2,
+      "seq_len": 1024,
       "vocab_size": 16391,
       "params_m": 3.15,
       "layers": 4,
@@ -109,6 +115,7 @@
       "size": "tiny",
       "vocab": "bpe-64k",
       "seed": 1,
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 3.15,
       "layers": 4,
@@ -126,6 +133,7 @@
       "size": "tiny",
       "vocab": "bpe-64k",
       "seed": 2,
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 3.15,
       "layers": 4,
@@ -143,7 +151,8 @@
       "size": "small",
       "vocab": "byte",
       "seed": 1,
-      "vocab_size": 261,
+      "seq_len": 1024,
+      "vocab_size": 263,
       "params_m": 14.16,
       "layers": 8,
       "hidden_size": 384,
@@ -160,7 +169,8 @@
       "size": "small",
       "vocab": "byte",
       "seed": 2,
-      "vocab_size": 261,
+      "seq_len": 1024,
+      "vocab_size": 263,
       "params_m": 14.16,
       "layers": 8,
       "hidden_size": 384,
@@ -177,7 +187,8 @@
       "size": "small",
       "vocab": "bpe-4k",
       "seed": 1,
-      "vocab_size": 4101,
+      "seq_len": 1024,
+      "vocab_size": 4103,
       "params_m": 14.16,
       "layers": 8,
       "hidden_size": 384,
@@ -194,7 +205,8 @@
       "size": "small",
       "vocab": "bpe-4k",
       "seed": 2,
-      "vocab_size": 4101,
+      "seq_len": 1024,
+      "vocab_size": 4103,
       "params_m": 14.16,
       "layers": 8,
       "hidden_size": 384,
@@ -211,6 +223,7 @@
       "size": "small",
       "vocab": "bpe-16k",
       "seed": 1,
+      "seq_len": 1024,
       "vocab_size": 16391,
       "params_m": 14.16,
       "layers": 8,
@@ -228,6 +241,7 @@
       "size": "small",
       "vocab": "bpe-16k",
       "seed": 2,
+      "seq_len": 1024,
       "vocab_size": 16391,
       "params_m": 14.16,
       "layers": 8,
@@ -245,6 +259,7 @@
       "size": "small",
       "vocab": "bpe-64k",
       "seed": 1,
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 14.16,
       "layers": 8,
@@ -262,6 +277,7 @@
       "size": "small",
       "vocab": "bpe-64k",
       "seed": 2,
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 14.16,
       "layers": 8,
@@ -279,7 +295,8 @@
       "size": "medium",
       "vocab": "byte",
       "seed": 1,
-      "vocab_size": 261,
+      "seq_len": 1024,
+      "vocab_size": 263,
       "params_m": 37.76,
       "layers": 12,
       "hidden_size": 512,
@@ -303,7 +320,8 @@
       "size": "medium",
       "vocab": "byte",
       "seed": 2,
-      "vocab_size": 261,
+      "seq_len": 1024,
+      "vocab_size": 263,
       "params_m": 37.76,
       "layers": 12,
       "hidden_size": 512,
@@ -322,7 +340,8 @@
       "size": "medium",
       "vocab": "byte",
       "seed": 3,
-      "vocab_size": 261,
+      "seq_len": 1024,
+      "vocab_size": 263,
       "params_m": 37.76,
       "layers": 12,
       "hidden_size": 512,
@@ -341,7 +360,8 @@
       "size": "medium",
       "vocab": "bpe-4k",
       "seed": 1,
-      "vocab_size": 4101,
+      "seq_len": 1024,
+      "vocab_size": 4103,
       "params_m": 37.76,
       "layers": 12,
       "hidden_size": 512,
@@ -365,7 +385,8 @@
       "size": "medium",
       "vocab": "bpe-4k",
       "seed": 2,
-      "vocab_size": 4101,
+      "seq_len": 1024,
+      "vocab_size": 4103,
       "params_m": 37.76,
       "layers": 12,
       "hidden_size": 512,
@@ -384,7 +405,8 @@
       "size": "medium",
       "vocab": "bpe-4k",
       "seed": 3,
-      "vocab_size": 4101,
+      "seq_len": 1024,
+      "vocab_size": 4103,
       "params_m": 37.76,
       "layers": 12,
       "hidden_size": 512,
@@ -403,6 +425,7 @@
       "size": "medium",
       "vocab": "bpe-16k",
       "seed": 1,
+      "seq_len": 1024,
       "vocab_size": 16391,
       "params_m": 37.76,
       "layers": 12,
@@ -427,6 +450,7 @@
       "size": "medium",
       "vocab": "bpe-16k",
       "seed": 2,
+      "seq_len": 1024,
       "vocab_size": 16391,
       "params_m": 37.76,
       "layers": 12,
@@ -446,6 +470,7 @@
       "size": "medium",
       "vocab": "bpe-16k",
       "seed": 3,
+      "seq_len": 1024,
       "vocab_size": 16391,
       "params_m": 37.76,
       "layers": 12,
@@ -465,6 +490,7 @@
       "size": "medium",
       "vocab": "bpe-64k",
       "seed": 1,
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 37.76,
       "layers": 12,
@@ -489,6 +515,7 @@
       "size": "medium",
       "vocab": "bpe-64k",
       "seed": 2,
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 37.76,
       "layers": 12,
@@ -508,6 +535,7 @@
       "size": "medium",
       "vocab": "bpe-64k",
       "seed": 3,
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 37.76,
       "layers": 12,
@@ -527,6 +555,7 @@
       "size": "medium",
       "vocab": "bpe-64k",
       "seed": "matched-tokens",
+      "seq_len": 1024,
       "vocab_size": 65543,
       "params_m": 37.76,
       "layers": 12,
@@ -540,6 +569,186 @@
       "magicfiles_f1": 0.0,
       "net_top1_k1": 0.7449799196787149,
       "net_top1_kall": 0.7630522088353414
+    },
+    "medium/byte/s1-seq256": {
+      "cell_id": "medium/byte/s1-seq256",
+      "size": "medium",
+      "vocab": "byte",
+      "seed": 1,
+      "seq_len": 256,
+      "vocab_size": 263,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "ece": null,
+      "adv_zero_4": 0.0,
+      "adv_zero_16": 0.0,
+      "adv_zero_64": 0.0,
+      "adv_random_4": 0.0,
+      "net_top1_k1": 0.8554216867469879,
+      "net_top1_kall": 0.8554216867469879
+    },
+    "medium/byte/s2-seq256": {
+      "cell_id": "medium/byte/s2-seq256",
+      "size": "medium",
+      "vocab": "byte",
+      "seed": 2,
+      "seq_len": 256,
+      "vocab_size": 263,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "net_top1_k1": 0.8554216867469879,
+      "net_top1_kall": 0.8554216867469879
+    },
+    "medium/bpe-4k/s1-seq256": {
+      "cell_id": "medium/bpe-4k/s1-seq256",
+      "size": "medium",
+      "vocab": "bpe-4k",
+      "seed": 1,
+      "seq_len": 256,
+      "vocab_size": 4103,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "ece": null,
+      "adv_zero_4": 0.0,
+      "adv_zero_16": 0.0,
+      "adv_zero_64": 0.0,
+      "adv_random_4": 0.0,
+      "net_top1_k1": 0.8152610441767069,
+      "net_top1_kall": 0.821285140562249
+    },
+    "medium/bpe-4k/s2-seq256": {
+      "cell_id": "medium/bpe-4k/s2-seq256",
+      "size": "medium",
+      "vocab": "bpe-4k",
+      "seed": 2,
+      "seq_len": 256,
+      "vocab_size": 4103,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "net_top1_k1": 0.8152610441767069,
+      "net_top1_kall": 0.821285140562249
+    },
+    "medium/bpe-16k/s1-seq256": {
+      "cell_id": "medium/bpe-16k/s1-seq256",
+      "size": "medium",
+      "vocab": "bpe-16k",
+      "seed": 1,
+      "seq_len": 256,
+      "vocab_size": 16391,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "ece": null,
+      "adv_zero_4": 0.0,
+      "adv_zero_16": 0.0,
+      "adv_zero_64": 0.0,
+      "adv_random_4": 0.0,
+      "net_top1_k1": 0.8092369477911646,
+      "net_top1_kall": 0.821285140562249
+    },
+    "medium/bpe-16k/s2-seq256": {
+      "cell_id": "medium/bpe-16k/s2-seq256",
+      "size": "medium",
+      "vocab": "bpe-16k",
+      "seed": 2,
+      "seq_len": 256,
+      "vocab_size": 16391,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "net_top1_k1": 0.8092369477911646,
+      "net_top1_kall": 0.821285140562249
+    },
+    "medium/bpe-64k/s1-seq256": {
+      "cell_id": "medium/bpe-64k/s1-seq256",
+      "size": "medium",
+      "vocab": "bpe-64k",
+      "seed": 1,
+      "seq_len": 256,
+      "vocab_size": 65543,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "ece": null,
+      "adv_zero_4": 0.0,
+      "adv_zero_16": 0.0,
+      "adv_zero_64": 0.0,
+      "adv_random_4": 0.0,
+      "net_top1_k1": 0.7449799196787149,
+      "net_top1_kall": 0.7630522088353414
+    },
+    "medium/bpe-64k/s2-seq256": {
+      "cell_id": "medium/bpe-64k/s2-seq256",
+      "size": "medium",
+      "vocab": "bpe-64k",
+      "seed": 2,
+      "seq_len": 256,
+      "vocab_size": 65543,
+      "params_m": 37.76,
+      "layers": 12,
+      "hidden_size": 512,
+      "heads": 8,
+      "wall_hours": 4.5,
+      "magicfrags_top1": 0.0,
+      "magicfrags_f1": 0.0,
+      "magicfrags_r1": 0.0,
+      "magicfiles_top1": 0.0,
+      "magicfiles_f1": 0.0,
+      "net_top1_k1": 0.7449799196787149,
+      "net_top1_kall": 0.7630522088353414
     }
   }
 }