Add CLEF model guidance and measured Laya, Jev and CLEF latency

README.md

@@ -14,6 +14,19 @@ tags: [prompt-injection, data-exfiltration, llm-security, agent-security, laya,
 
 `main` now contains the **R2a checkpoint**, with matching PyTorch weights, calibration configuration, tokenizer and a newly exported FP32 ONNX graph. This is a complete **321.9M-parameter** Laya decision model: an mmBERT-base encoder plus a two-layer decision head. It can run locally without sending document text to a hosted API.
 
+## Choosing between Laya and CLEF
+
+**Looking for stronger detection? [Get CLEF-Cybersecurity](https://huggingface.co/TextCortex/clef-cybersecurity), our fine-tuned CLEF model.** It has higher AUROC than Laya R2a and Jev on the full English, full German and PDF regression suites below. [Download CLEF's files](https://huggingface.co/TextCortex/clef-cybersecurity/tree/main) and follow its [loading instructions](https://huggingface.co/TextCortex/clef-cybersecurity#usage).
+
+| Priority | Model to consider | Practical difference |
+|---|---|---|
+| Compact local scanning and CPU deployment | **Laya-Cybersec R2a** (this repository) | 321.9M parameters, complete weights and CPU ONNX export; about **30× fewer parameters** than CLEF. |
+| Higher full-suite and PDF detection quality | **[CLEF-Cybersecurity](https://huggingface.co/TextCortex/clef-cybersecurity)** | 9.53B total inference parameters; the 2.20 GB update requires the full CLEF base. The measured batch-one GPU runtime allocated about 19.8 GiB. |
+
+Laya is the lighter deployment option; CLEF is the stronger overall detector in these saved tests. **Speed depends on hardware, runtime and document length.** See the [measured latency results](#latency-and-deployment-tradeoffs) before choosing for a latency target; the parameter ratio is not a measured speedup.
+
+CLEF improves PDF AUROC from Laya's **0.8856 to 0.9856**. At the saved thresholds, it catches **84/107 attacks versus Laya's 81 and Jev's 73**, with **3/623 clean false alarms versus 0 and 2**, respectively. It does not win every subset: **German-skills AUROC is 0.9171**, below Laya's 0.9330 and Jev's 0.9603. The thresholds differ, and these are previously inspected regression sets.
+
 The previous release remains available at [`pre-r2a-20261006`](https://huggingface.co/TextCortex/laya-cybersec/tree/pre-r2a-20261006), commit `a75e214574d9cbbf89f2f5dcc48b2e98dbcc01f6`. Pin that revision to retain its behavior. **R2a changes scores and calibration; it is not an improvement on every benchmark.** The previous card's latency measurements and ONNX parity claims do not describe this release.
 
 ## What it scans
@@ -81,7 +94,28 @@ Each full-language suite contains 510 cases: 256 attacks and 254 clean examples.
 
 These are previously inspected regression sets, not fresh blind tests. AUROC is ranking quality, not the fraction of attacks caught. Thresholds and detector wrappers differ, so the counts are not equal-false-positive-rate comparisons. Zero clean flags on this cohort does not establish a zero false-positive rate on new traffic. R2a's English-skills result is 0.9277 in the saved matched reference and 0.9268 in the batched GPU run, reflecting small backend precision differences. The full-language numbers above are the language-filtered GPU results, not the older mixed-language collection totals.
 
-Jev scores come from saved hosted evaluations; its exact provider-side revision was unavailable. Base CLEF is the unchanged local `Cloudflare/clef-flash` checkpoint; CLEF-Cybersecurity is the separately fine-tuned TextCortex detector. See [benchmark_results.json](benchmark_results.json). Historical numbered charts in `charts/` apply only to the previous release and are documented in [charts/README.md](charts/README.md).
+Jev scores come from saved hosted evaluations; its exact provider-side revision was unavailable. Base CLEF is the unchanged local [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash) checkpoint; [CLEF-Cybersecurity](https://huggingface.co/TextCortex/clef-cybersecurity) is the separately fine-tuned TextCortex detector. Fine-tuning raises CLEF's full English AUROC from **0.9588 to 0.9925**, full German from **0.9391 to 0.9744**, and PDF AUROC from **0.8144 to 0.9856**. See [benchmark_results.json](benchmark_results.json). Historical numbered charts in `charts/` apply only to the previous release and are documented in [charts/README.md](charts/README.md).
+
+## Latency and deployment tradeoffs
+
+**Observed timings from separate saved runs — not a matched speed ranking.** Hardware, input cohorts, window sizes and timing units differ. p50 is the median; p95 is the 95th percentile. Lower times are better within the same setup.
+
+| Model / measurement | Hardware | Timed work and sample | p50 | p95 |
+|---|---|---|---:|---:|
+| **Laya-Cybersec R2a**, short inputs | Local Apple MPS GPU | All 279 single-window inputs from the saved run | 95.4 ms | 148.5 ms |
+| **Laya-Cybersec R2a**, all inputs | Local Apple MPS GPU | All 1,042 inputs, including every document window | 730.3 ms | 4570.1 ms |
+| **Jev**, hosted | Provider infrastructure + network | 10,533 successful PDF chunk requests | 263.6 ms | 348.9 ms |
+| **[CLEF-Cybersecurity](https://huggingface.co/TextCortex/clef-cybersecurity)**, accelerated | NVIDIA B200 | 64 complete inputs × 2 passes, batch one, warm | 40.0 ms | 510.5 ms |
+
+**How to read these results:** Laya's short-input times were lower than Jev's recorded request times, while accelerated CLEF recorded a lower median on its B200 sample than Laya did on the Mac. Different inputs and hardware prevent a controlled speed ranking. Laya offers a much smaller model and CPU/ONNX support; CLEF trades a larger GPU footprint for stronger detection on the main regression suites.
+
+Laya uses 1,500-character windows with 200-character overlap and scores them sequentially; short inputs and long PDFs should not be assigned the same latency. Its 1,042-input timing cohort contains 723 clean PDF texts, 107 attacked excerpts and 212 skills, and differs from the 730-PDF accuracy cohort above. These are saved **PyTorch/MPS** timings, not measurements of the newly exported ONNX graph. Initial model loading and PDF extraction are excluded; this run did not use CLEF's dedicated warmup of every input shape.
+
+Jev's figures include the successful request's network round trip and provider processing. They exclude earlier failed attempts and retry backoff. The harness issued concurrent requests; summing request durations is **not** complete-document wall time. The requested model was `jev-latest`; its resolved provider version was unavailable.
+
+CLEF's 64 inputs were selected by character-length ranks independently of labels or scores, then measured twice with all shapes warmed. Timing includes tokenization and all document windows, excluding model loading, PDF extraction, network and queue time. It used PyTorch 2.9.1+cu128, Transformers 5.17.0, Triton 3.5.1, flash-linear-attention 0.5.2 and causal-conv1d 1.7.0 on a B200. The 40.0 ms median does not describe the earlier H200/reference-kernel runs or hosted Cloudflare API.
+
+See [latency_results.json](latency_results.json) for aggregate timings, measurement scopes and source hashes. No customer content or individual timing records are distributed. A controlled speed comparison would require the same input sample and local hardware/runtime, with hosted network time reported separately.
 
 ## Training and provenance
 

latency_results.json

@@ -0,0 +1,58 @@
+{
+  "type": "aggregate_saved_runtime_measurements",
+  "comparison": "Different hardware, cohorts, windows and timing units; not a matched speed ranking or speedup measurement.",
+  "quantiles": "NumPy median and linear-interpolated 0.95 quantile",
+  "measurements": [
+    {
+      "model": "TextCortex/laya-cybersec",
+      "checkpoint": "R2a",
+      "subset": "single_window",
+      "hardware": "Apple MPS (local Mac GPU)",
+      "unit": "complete single-window input",
+      "scope": "All 279 one-window inputs from the saved 1042-input run; sequential batch-one Laya API scoring, including encoding; no dedicated all-shape warmup. Excludes model loading and PDF extraction.",
+      "source_sha256": "048522ef775ca1402dd37314cb3bb06800ec8e38829d118ab477d7b564b44f9b",
+      "observations": 279,
+      "p50_ms": 95.3577909967862,
+      "p95_ms": 148.5410204855725
+    },
+    {
+      "model": "TextCortex/laya-cybersec",
+      "checkpoint": "R2a",
+      "subset": "all_inputs",
+      "hardware": "Apple MPS (local Mac GPU)",
+      "unit": "complete saved input, including all windows",
+      "scope": "All 1042 saved inputs: 723 clean PDF texts, 107 attack excerpts and 212 skills; 1500-character windows with 200-character overlap, scored sequentially. Excludes model loading and PDF extraction.",
+      "source_sha256": "048522ef775ca1402dd37314cb3bb06800ec8e38829d118ab477d7b564b44f9b",
+      "observations": 1042,
+      "p50_ms": 730.2949790027924,
+      "p95_ms": 4570.080316357779
+    },
+    {
+      "model": "jev-latest",
+      "subset": "pdf_chunk_requests",
+      "hardware": "Hosted API",
+      "unit": "successful chunk request",
+      "scope": "10533 successful requests for 830 saved PDF inputs; includes network and provider time, excludes prior failed attempts and retry backoff. Concurrent request harness; not complete-document wall time. Provider revision unavailable.",
+      "source_sha256": "4ecff2adb4f4bf57aff4cffd8fa316144d1d974d3cbe00ac2584336ca3bb4dcc",
+      "observations": 10533,
+      "p50_ms": 263.58416699804366,
+      "p95_ms": 348.8895498017257
+    },
+    {
+      "model": "TextCortex/clef-cybersecurity",
+      "checkpoint": "validation-selected epoch 3, calibrated",
+      "subset": "length_rank_sample",
+      "hardware": "NVIDIA B200",
+      "unit": "complete saved input, including all windows",
+      "scope": "Complete saved text tokenization and model scoring; excludes loading, PDF extraction, network and queue time. All measured shapes warmed; batch one, accelerated FLA/causal-conv1d runtime. 64 uniformly spaced ranks by character length; independent of labels and scores",
+      "documents": 64,
+      "repetitions": 2,
+      "peak_allocated_gib": 19.843493461608887,
+      "source_sha256": "ced52ccc1ad8ef9c5a6342fa70adf37d3d28bb56151e8054f1125105355d19a8",
+      "observations": 128,
+      "p50_ms": 40.00461706891656,
+      "p95_ms": 510.47315176110715
+    }
+  ],
+  "privacy": "Aggregate values only; no customer text, identifiers or individual measurements."
+}

release_manifest.json

@@ -13,8 +13,8 @@
       "sha256": "2d3ae4362dff323aba51d21cab8c53fd08ec718352dc38f7574feea7b6ec3935"
     },
     "README.md": {
-      "bytes": 8409,
-      "sha256": "c934018600548505ab52845106a01375f3a23e867d378ed8e2fcc77d48aabd18"
+      "bytes": 13381,
+      "sha256": "b260e7b5e4e96e48a5cc77bcaa14f72481be7885d44a7de35f83aedbf8d72266"
     },
     "rl_agent_config.json": {
       "bytes": 676,
@@ -24,6 +24,10 @@
       "bytes": 4932,
       "sha256": "373efea2db16b6901fdf6f62b2e674cdda7027b6cc1b008d277b51b6dcf8412b"
     },
+    "latency_results.json": {
+      "bytes": 3011,
+      "sha256": "3d9c6ff37ff6941679ecff137b1b34167783d01e935dd8d641dc30729900a883"
+    },
     "onnx/laya-cybersec.onnx": {
       "bytes": 1287802788,
       "sha256": "412d4ac021d9f178208d4c2a372891d9c6b70c7034517dccc3f37e45a4f8f7cf"