card: add Long context section (retrieval to 350K, TTFT/decode/memory tables, largest clean max_seq_len)

README.md

@@ -77,6 +77,7 @@ eval/
   bench/local-*.score.json  local-*.meta.json   this quant's per-task scores + run metadata
   bench/ref-*.score.json    ref-*.meta.json     the unquantized FP8 reference, same harness
   eval-quant.json  eval-overflow-64rows.json  dflash-bench-*.json
+  longctx.json                           the long-context sweep, every request (see Long context)
 ```
 
 The shards, index, `quantization_config.json` and the tokenizer files are byte-identical to
@@ -361,6 +362,10 @@ With 86.20 GiB of weights and a 10 GiB reserve, the KV budget is 25.43 GiB:
 and generates coherently on top of these 2.36 bpw weights, but its quality is untested — the
 default stays FP16. The 39 sliding-window rings are always FP16.
 
+Those are the arithmetic limits. What was actually served, loaded and measured — up to
+**350,091 tokens** — is in [Long context](#long-context) below; the cache is **preallocated in
+full at load**, so `max_seq_len` is a memory decision, not a ceiling you pay for on demand.
+
 #### A unified-memory hazard worth knowing about
 
 On GB10 there is **no separate VRAM**: host and GPU share one pool. exllamav3's shard loader
@@ -378,6 +383,169 @@ why the numbers above are flat rather than degrading:
 
 ---
 
+## Long context
+
+Measured on the DGX Spark, batch 1, greedy, thinking off, FP16 KV, needles and haystacks built
+from wikitext-2 paragraphs. **Every retrieval test passed.**
+
+### Retrieval
+
+A six-digit passcode is hidden in a wikitext haystack at 10 / 50 / 90% depth and asked for at
+the end. Exact match on the digits, fresh city and fresh code per request.
+
+| prompt tokens | 10% | 50% | 90% |
+|---|---|---|---|
+| ~8.2K | OK | OK | OK |
+| ~32.8K | OK | OK | OK |
+| ~65.5K | OK | OK | OK |
+| ~131.1K | OK | OK | OK |
+| ~200.1K | OK | OK | OK |
+| ~250.0K | OK | OK | OK |
+| **350,091** | **OK** | — | — |
+
+**18/18** at `max_seq_len 262144`, plus **350,091 tokens** answered correctly in a 393,216
+window — the largest prompt this quant has been given. Nine more cells at `max_seq_len 131072`
+with the drafter on also passed, so retrieval is unaffected by speculative decoding.
+
+That matters more than it looks: **39 of 48 layers are sliding-window with a 128-token
+window** and run on a fixed 768-token ring, so a needle 25,000 tokens into a 250K prompt has
+to survive ~195 ring rebases and reach the question through the 9 global-attention layers
+alone. The ring had previously only been validated to 2,048 tokens.
+
+**Five needles at once, 128K prompt** — all five returned, in order of appearance:
+
+```
+Montevideo: 604403
+Bratislava: 991476
+Kathmandu: 472495
+Ulaanbaatar: 379397
+Ljubljana: 361254
+```
+
+**A real long document, ~100K tokens** — complete wikitext articles concatenated with *Plain
+maskray* buried in the middle, three questions answerable only from that article:
+
+```
+(a) 2008
+(b) ~54 Ma
+(c) 12 and 62 m
+```
+
+All three correct against the source ("In 2008, Last and William White elevated the kuhlii
+group…", "estimated to have occurred ~ 54 Ma", "recorded from between 12 and 62 m"), including
+the source's own "~" hedge.
+
+### Speed and memory vs prompt length
+
+No drafter, 128 generated tokens, `max_seq_len 262144` (last row 393,216):
+
+| prompt | TTFT | prefill | decode | min MemAvailable |
+|---|---|---|---|---|
+| ~8.2K | 9.1–10.6 s | 768–905 tok/s | 31.0 tok/s | 19.85 GiB |
+| ~32.8K | 41.2–42.0 s | 780–799 tok/s | 28.4 tok/s | 19.56 GiB |
+| ~65.5K | 100.1–100.5 s | 652–654 tok/s | 25.3 tok/s | 19.10 GiB |
+| ~131.1K | 270.0–271.5 s | 483–486 tok/s | 21.7 tok/s | 18.98 GiB |
+| ~200.1K | 524.8–525.9 s | 380–381 tok/s | 18.3 tok/s | 18.85 GiB |
+| ~250.0K | 757.6–758.8 s | 329–330 tok/s | 16.7 tok/s | 17.77 GiB |
+| **350,091** | **1,346.8 s** | **260 tok/s** | **13.8 tok/s** | **15.91 GiB** |
+
+**Decode falls only 2.3x from 8K to 250K** — the sliding-window ring again: only 9 of 48
+layers grow their KV with context. Prefill falls 2.7x and TTFT is quadratic, as full
+attention on those 9 layers requires. The curve fits
+
+```
+TTFT(seconds) ≈ 988·L + 8172·L²      (L = prompt tokens in millions)
+```
+
+to better than 1.5% from 64K to 350K — the 350K point was predicted at 1,347 s from a fit to
+the 128K and 250K points alone and came in at 1,346.8 s. Extrapolated: **400K ≈ 28 min,
+500K ≈ 42 min of prefill.** Long context on one GB10 is TTFT-bound, not memory-bound.
+
+With the DFlash drafter at `max_seq_len 131072` (real generations, ~380 tokens, unforced):
+
+| prompt | TTFT | decode drafted | vs no draft |
+|---|---|---|---|
+| 1,983 | 2.97 s | **36.3 tok/s** | 1.17x |
+| 65,544 | 103.1 s | **29.8 tok/s** | 1.18x |
+| 129,876 | 171.9 s | **27.0 tok/s** | 1.24x |
+
+Live acceptance 38–85%. (Drafting speedup on long-context Q&A is smaller than the 1.35–1.78x
+seen on short coding/reasoning prompts, because the answers are short and factual.)
+
+### How much context fits
+
+The paged cache is **preallocated in full at load** — `cache_size` is paid up front whether or
+not anyone sends a long prompt. Cost per slot:
+
+```
+paged KV         27.00 KiB/token      (the 9 global-attention layers only)
+SWA ring        175.5 MiB per slot    (768-token ring x 39 layers, always FP16)
+draft KV         20.00 KiB/token      (5 DFlash layers, sized for the full context)
+drafter weights   2.81 GiB
+```
+
+so a slot costs **27.0 KiB/token** without the drafter and **47.0 KiB/token** with it. On this
+box the whole budget reduces to one line that held to within 0.1 GiB across three configurations:
+
+```
+MemAvailable after load ≈ 29.2 GiB − (paged KV + ring + drafter weights + draft KV)
+```
+
+(121.63 GiB total − 86.15 GiB of weights − ~6 GiB of runtime.)
+
+| `max_seq_len` | drafter | preallocated | load | MemAvailable after load | min under load |
+|---|---|---|---|---|---|
+| 393,216 | off | 10.29 GiB | 21.3 s | 18.65 GiB | **15.91 GiB** @ 350K |
+| 262,144 | off | 6.92 GiB | 21.6 s | 22.30 GiB | **17.77 GiB** @ 250K |
+| **131,072** | **DFlash** | 6.13 + 2.81 GiB | 22.6 s | 20.17 GiB | **17.07 GiB** @ 130K |
+
+**Largest clean settings: `max_seq_len 393216` without the drafter, `131072` with it.**
+
+* **524,288 does not fit at FP16.** Its 13.67 GiB of paged KV leaves ~15.3 GiB after load, and
+  the measured 2.7–4.5 GiB transient of a full-length request would land it at ~11–12.5 GiB —
+  at or through a 12 GiB safety floor. On unified memory that is not an OOM kill, it is a
+  frozen machine. A 500K prefill would also take ~42 minutes.
+* **262,144 with the drafter does not fit either**: 14.73 GiB resident leaves ~14.5 GiB after
+  load and ~10 GiB under a full-length request. The drafter's cache is the problem — it is
+  sized for the whole context although it never looks back more than 1,024 tokens.
+
+So the choice on a 121 GiB box is **3x the window (393K, no drafter)** or **~1.2x the decode
+rate (131K, drafter)**. This repo's reference server runs the latter.
+
+### Recommended serving settings
+
+```yaml
+model:
+  max_seq_len: 131072       # with the drafter; use 393216 and drop draft_model to go longer
+  cache_size: 131072
+  cache_mode: FP16
+  chunk_size: 2048
+  max_batch_size: 1         # every extra slot repeats the ring + paged span + draft cache
+draft_model:
+  draft_mode: model
+  draft_model_name: mimo-dflash-draft
+  draft_cache_mode: FP16
+  dynamic_draft: true
+```
+
+### Caveats
+
+* **`max_seq_len` bounds the prompt, not prompt + generation.** A 131,092-token prompt against
+  `max_seq_len 131072` returns `400 Bad Request` / `Prompt length 131092 exceeds the …` before
+  a token is generated. Budget `max_seq_len ≥ prompt + max_tokens`, and remember the chat
+  template adds ~20 tokens.
+* **Every number here is `max_batch_size: 1`.** A second slot adds another 175.5 MiB ring, its
+  own paged span and its own draft cache; concurrency at 128K is not free.
+* Prefill is chunked at 2,048 tokens. Prefill rates for a prompt sharing a long prefix with the
+  previous request are inflated by page reuse (755 vs 473 tok/s at 130K here), so cold numbers
+  are the ones quoted above.
+* Q8 paged KV halves the 27 KiB/token term and would make ~786K arithmetically fit, but
+  quantized KV quality on top of 2.34 bpw weights is untested and the prefill time would be
+  hours. The 39 sliding-window rings stay FP16 regardless.
+* Raw records: [`eval/longctx.json`](eval/longctx.json).
+
+---
+
 ## How to run
 
 ### Requirements
@@ -430,8 +598,8 @@ models/
 model:
   model_dir: models
   model_name: mimo-2.25bpw-hq
-  max_seq_len: 65536
-  cache_size: 65536
+  max_seq_len: 131072       # validated; 393216 fits if you drop the draft model
+  cache_size: 131072
   cache_mode: FP16          # default; Q8 paged KV works but is unvalidated for quality
   chunk_size: 2048
   max_batch_size: 1         # each extra slot costs another 175.5 MiB ring + paged span + draft cache
@@ -550,8 +718,11 @@ The drafter loads BF16 in 1.3 s / 2.81 GiB — no quantization needed. Its own K
   perfect 2.34 bpw quant — and the benchmark deltas above say the FP8 model is only ~0.6 pp
   better at HumanEval+, so most of that is the base model, not the quantization.
 * **Tensor parallel is unsupported.** The port has only been built and run single-device.
-* **Long context beyond 64K is untested at serving time.** The KV arithmetic says ~980K
-  tokens fit at 1 slot, and 32K was measured, but nothing between 32K and 1M has been served.
+* **Long context is now measured, and it is TTFT-bound rather than memory-bound.** 8K → 350K
+  has been served and retrieval is perfect at every length and depth tested, but a 250K prompt
+  costs ~12.6 minutes of prefill and a 350K prompt ~22.4 minutes on one GB10. `max_seq_len`
+  above 393,216 (or above 131,072 with the drafter) does not fit at FP16 KV. See
+  [Long context](#long-context).
 * **Batch > 1 with DFlash is untested.** Everything above is `max_batch_size: 1`.
 * **No vision, no audio, no MTP.** The MTP heads in the source repo are not ported, so
   MTP-based drafting is unavailable; DFlash is the drafting path.