Publish selected TP4 reference, full speed evidence and matched FP8/NVFP4 cache KLD

README.md

@@ -18,13 +18,175 @@ tags:
 
 # GLM-5.3-Flash TrellisMX-MXFP8
 
-**Weight upload complete; remote inventory and all 168 sidecar SHA-256 identities verified.**
-The HF-layout GPU serving test is still pending. Consult `release-status.json`.
+A compressed GLM-5.3-Flash checkpoint whose routed experts reconstruct directly
+into **FP8 Tensor Core operands**. The September 9 reference stack is the selected
+runtime: **222.100 tokens/s C1 decode at 8K** and **8,457 tokens/s prefill at 32K**
+in separate `llm_decode_bench` runs on four RTX PRO 6000 Blackwell GPUs at 300W each.
 
-This is the measured **17-K5 / 25-K4 coupled checkpoint**, covering all
-42 routed layers (3–44). Historical DCP1 mean true-decode KLD is **0.0341811459** on 32
-already-opened conditional-fit development windows. The routed tensors store
-**4.6587417643 bits/weight**, including metadata; they are not uniform 4.25 bpw.
+## Recommended reference stack — September 9, 2026
+
+[Docker Hub](https://hub.docker.com/r/verdictai/trellismx) image:
+
+```text
+verdictai/trellismx:glm53-flash-p8-r27-reference-20260909
+```
+
+Immutable image reference:
+
+```text
+verdictai/trellismx@sha256:ca6b80188dce154b91f49108b7d87792d2ba6328935afc71b44d1c0e6f6a1adf
+```
+
+Use the supplied [Docker Compose configuration](compose.yaml) and
+[serving script](serve.sh). The [reference runtime recipe](runtime-reference-20260909/Dockerfile)
+and [source manifest](runtime-reference-20260909/source-manifest.json) record the
+inference overlays. The checkpoint has not been re-encoded for this update.
+
+| Setting | Selected reference |
+| --- | --- |
+| GPUs / measured power | 4 × RTX PRO 6000 Blackwell 96GB (SM120), 300W per GPU |
+| Parallelism | TP4 / DCP4 |
+| Routed-expert math | E4M3 FP8, native UE8M0 scales per 32 elements |
+| MLA KV cache | NVFP4 (`nvfp4_ds_mla`) |
+| Speculative decoding | Probabilistic MTP3, standard rejection |
+| Execution | CUDA graphs, B12X MLA attention and PCIe collectives |
+| Scheduler | 24 sequence slots, 4,096-token batch |
+| Memory / maximum request length | 0.97 GPU memory utilization / 1,000,000 tokens |
+
+**FP8 math and NVFP4 KV cache describe different parts of the model.** Compressed
+K4/K5 expert weights decode to FP8 operands for multiplication. NVFP4 stores the
+MLA attention cache. Choosing FP8 KV instead changes the cache representation;
+it does not turn this into an FP4-math checkpoint. The P4 design is separate.
+
+Weight upload is complete: the remote inventory and all **168 sidecar SHA-256
+identities** were verified. Local serving and measurements use the same checkpoint.
+A separate clean download-to-GPU test of the HF directory layout has not been run;
+see [release status](release-status.json).
+
+## Measured speed and KV capacity
+
+Rates below are **tokens/s from `llm_decode_bench`**. C1 is one request;
+C4 and C8 are aggregate throughput across concurrent requests, not per user.
+Context labels are nominal benchmark cells; raw logs retain actual prompt sizes.
+
+| Context | Prefill | C1 decode | C4 decode, aggregate | C8 decode, aggregate |
+| --- | ---: | ---: | ---: | ---: |
+| 0K | — | 204.611 | 327.670 | — |
+| 8K | — | 222.100 | 318.560 | 594.639 |
+| 16K | — | 216.636 | 319.157 | 602.965 |
+| 32K | 8,457 | 204.930 | 329.343 | — |
+| 64K | 8,443 | 199.446 | 330.096 | — |
+| 128K | 8,323 | 204.445 | 334.086 | — |
+
+This table combines the short-context screen with the separate expanded-context
+pass. The earlier comparison screen measured **8,407 tokens/s prefill at both
+32K and 64K**. All runs are retained rather than selecting one figure as a repeated-run average.
+The reference was selected for balanced use: it had the highest observed C1
+throughput at 0K and 8K among the three finalists. Task-count performed better
+at 16K C4 (**353.532 tokens/s**), but neither alternative was measured at C1
+32K–128K. Reference is not established as the fastest candidate for every workload.
+
+The engine reported **23,562,091 aggregate effective KV tokens** in the expanded
+reference run; separate short-context sessions reported **23,568,627**. This is
+allocated capacity across requests, not a tested single-request context length
+or a full-capacity stress result. The configured request limit is 1,000,000 tokens.
+Use the engine's split-cache metric, not the benchmark client's generic
+block-count-times-DCP estimate.
+
+These are exploratory measurements, with one server preparation per configuration.
+Later screens used a minimum 90-second idle period and required all GPUs at or below
+55°C for 30 continuous seconds before each cell; earlier runs retain their original
+protocols. MTP acceptance, generated output and clocks can vary. No independently
+repeated speed qualification or long-context accuracy claim is implied.
+
+[All September 9 results, candidate comparison and protocol](results/speed-20260909/README.md)
+include the full benchmark index, JSON, logs, negative results and diagnostic runs.
+Profiled diagnostics are identified separately from serving speed measurements.
+
+## Reference-stack KLD: matched FP8 and NVFP4 MLA cache
+
+Lower KLD means the student's next-token distribution is closer to the BF16
+teacher on the measured inputs. It is not a direct score for general answer quality.
+
+| Cache on the September 9 reference | Mean true-decode KLD | Status |
+| --- | ---: | --- |
+| FP8 KV | **0.0319451732** | 32/32; receipt audit passed |
+| NVFP4 MLA KV | **0.0354562238** | 32/32; receipt audit passed |
+
+FP8 has lower observed KLD in **22/32 windows**. The paired mean difference
+(FP8 minus NVFP4) is **−0.0035110506**, with a window BCa 95% interval
+[−0.0081718772, −0.0013675941]. [Full results and audit](results/kld-reference-20260909/README.md).
+
+Both arms use the **same 32 previously opened conditional-fit development windows**
+as the prior measurement, with the same token arrays and BF16 teacher. Each window
+contains 2,048 input tokens and 2,047 prediction rows; row zero is excluded, leaving
+**2,046 true-decode rows per window**. Scores use CPU FP64 KL(teacher || student)
+and the equal mean of the 32 window means.
+
+The matched setup is TP4/DCP4, **MTP off**, one sequence, a 4,096-token batch and
+zero prefix-cache hits. FP8 runs first, then NVFP4. The logits-capture image is
+`sha256:0405a1c0dc128b51069798a5d00b346257bbd006c0deb7e6530a3d005d75de71`;
+it adds the capture/warmup seam to the selected serving image and is not the public
+serving tag. This quality measurement therefore does not measure the MTP3 speed
+configuration's complete output behavior.
+
+The paired result and confidence interval passed the receipt/score audit. These
+already-opened windows provide a development comparison, **not untouched final qualification**.
+No matched claim against EXL3, TR3 4bpw or another weight format is established by
+this cache comparison.
+
+The archived historical `.0341811459` cache label is under provenance review:
+the prior card labels it NVFP4, while the owner's historical account identifies
+FP8. The prior text and linked receipts are retained below without silently
+changing either number. Do not use that disputed historical label as a matched
+cache comparison. New reference results are recorded separately above.
+
+## Run the model
+
+Requires the custom runtime, four supported GPUs and NVIDIA Container Toolkit.
+This is a sidecar checkpoint used with a separate carrier model; stock Transformers
+or unmodified vLLM cannot load it as a standalone model.
+
+```bash
+hf download brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8 --local-dir ./trellismx-p8
+hf download local-inference-lab/GLM-5.3-Flash-NVFP4 \
+  --revision 520de24eabf507659eaef7c70f14fd584527facc --local-dir ./glm53-carrier
+cd trellismx-p8
+export MODEL_ROOT=/absolute/path/to/glm53-carrier
+docker compose -f compose.yaml config --quiet
+docker compose -f compose.yaml up -d
+```
+
+The default endpoint is port 8000. The scripts do not set GPU power limits or
+memory clocks. The measured 300W per GPU setting is part of the benchmark setup,
+not a guarantee that a new host will reproduce its speed. The 1,000,000-token
+launch limit is within the model's declared 1,048,576 positions; it is not a
+measured 1M-context accuracy result. API credentials are not baked into the image.
+
+## Required runtime and model layout
+
+This repository contains **168 TP4 routed-expert sidecars (177,269,057,440
+bytes)** plus serving metadata, scripts, license and evaluation receipts.
+They replace the carrier's routed experts at load time. They are not an
+additional expert ensemble, and this is **not a standalone Transformers
+checkpoint**. Do not point stock Transformers or unmodified vLLM at it.
+
+The working runtime additionally requires the stock carrier:
+[`local-inference-lab/GLM-5.3-Flash-NVFP4`](https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4/tree/520de24eabf507659eaef7c70f14fd584527facc),
+revision `520de24eabf507659eaef7c70f14fd584527facc`. Its attention, shared
+experts, embeddings, output head and MTP components remain part of the served
+model. Keeping this exact dependency preserves the existing loader contract;
+the reported logical model payload is not a claim that the two download
+directories together occupy only that many bytes.
+
+## Checkpoint and implementation details
+
+This is the **17-K5 / 25-K4 coupled checkpoint**, covering all 42 routed layers
+(3–44). Routed tensors occupy **4.6587417643 bits/weight including metadata**;
+this is not a uniform 4.25 bpw model.
+
+<details>
+<summary>How the compressed weights, FP8 math and prior work fit together</summary>
 
 ## What TrellisMX does
 
@@ -111,61 +273,20 @@ Existing prior art makes that broad wording inappropriate. The distinctive
 combination above describes this implementation without asserting an
 unverified first-in-history claim or general superiority over its sources.
 
-## Required runtime and model layout
+</details>
 
-This repository contains **168 TP4 routed-expert sidecars (177,269,057,440
-bytes)** plus serving metadata, scripts, license and evaluation receipts.
-They replace the carrier's routed experts at load time. They are not an
-additional expert ensemble, and this is **not a standalone Transformers
-checkpoint**. Do not point stock Transformers or unmodified vLLM at it.
+## Earlier measurements and provenance
 
-The working runtime additionally requires the stock carrier:
-[`local-inference-lab/GLM-5.3-Flash-NVFP4`](https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4/tree/520de24eabf507659eaef7c70f14fd584527facc),
-revision `520de24eabf507659eaef7c70f14fd584527facc`. Its attention, shared
-experts, embeddings, output head and MTP components remain part of the served
-model. Keeping this exact dependency preserves the existing loader contract;
-the reported logical model payload is not a claim that the two download
-directories together occupy only that many bytes.
+<details>
+<summary>September 8, RC5 and DCP1 results — archived runtime settings and receipts</summary>
 
-The Docker image is already published on
-[Docker Hub](https://hub.docker.com/r/verdictai/trellismx):
+The text below preserves the prior release's results and links. References to
+“current” or “production” inside this archive describe the earlier release,
+not the September 9 reference above. In particular, the old 16-sequence recipe
+and old image digest are superseded. The historical `.0341811459` cache label
+is disputed as described above; retaining the original text is not a resolution.
 
-```text
-verdictai/trellismx:glm53-flash-p8-r27-dcp4-20260908@sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f
-```
-
-The image stays in the container registry; this card links its immutable
-digest and includes `compose.yaml` and `serve.sh`. No encoder, calibration
-corpus or teacher logits are included in this HF release.
-
-## Serving
-
-Requires four RTX PRO 6000 Blackwell 96GB SM120 GPUs and the NVIDIA Container
-Toolkit. The current r27 recipe uses TP4/DCP4, probabilistic MTP3 with standard rejection, B12X_MLA_SPARSE attention,
-NVFP4 MLA KV (`nvfp4_ds_mla`), native P8 MoE with E4M3 activations, CUDA graphs,
-B12X PCIe collectives, prefix caching, a 4096-token scheduler batch and at most 16 sequences. GPU memory utilization is 0.97.
-It does not alter GPU power limits, memory clocks or host services.
-
-After `release-status.json` reports upload completion:
-
-```bash
-hf download brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8 --local-dir ./trellismx-p8
-hf download local-inference-lab/GLM-5.3-Flash-NVFP4 \
-  --revision 520de24eabf507659eaef7c70f14fd584527facc --local-dir ./glm53-carrier
-cd trellismx-p8
-export MODEL_ROOT=/absolute/path/to/glm53-carrier
-docker compose -f compose.yaml config --quiet
-docker compose -f compose.yaml up -d
-```
-
-Default port: 8000. Default maximum model length: **1,000,000 tokens**,
-within the model's declared 1,048,576 positions. That new launch limit is
-**not a tested 1M-context accuracy claim**. The HF-layout launch has CPU
-configuration checks only; a clean GPU download-to-serving test is pending.
-Use a firewall or authenticated gateway before exposing the endpoint beyond
-a trusted network. API credentials are not baked into the image.
-
-## Current r27 DCP4 results
+## September 8 r27 DCP4 results (superseded runtime)
 
 | Measurement | Result | Conditions |
 | --- | --- | --- |
@@ -190,7 +311,7 @@ client snapshot and redaction hashes](results/r27-20260908/README.md). The
 benchmark client's larger generic capacity estimate is not valid for this
 split-cache layout.
 
-### Current r27 KV-cache KLD comparison
+### September 8 r27 KV-cache KLD comparison
 
 | Runtime / measurement image | KV dtype | Attention backend | Mean true-decode KLD | Window BCa 95% interval |
 | --- | --- | --- | ---: | --- |
@@ -301,6 +422,8 @@ not reported by this release; no values are inferred for missing metrics.
 See the [full GitHub results and receipts](https://github.com/brandonmmusic-max/glm53-hadamard-shapleymcg-kld/blob/03527b1f092cfbf274ae3ca78cabb23077c37057/results/RC5_RELEASE_RESULTS_20260907.md),
 and the included `results/` and `evidence/` files.
 
+</details>
+
 ## Codec and limitations
 
 Procedural MCG trellis streams decode in registers to E4M3 for native

compose.yaml

@@ -1,12 +1,12 @@
 services:
   trellismx:
-    image: verdictai/trellismx:glm53-flash-p8-r27-dcp4-20260908@sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f
+    image: verdictai/trellismx:glm53-flash-p8-r27-reference-20260909@sha256:ca6b80188dce154b91f49108b7d87792d2ba6328935afc71b44d1c0e6f6a1adf
     restart: "no"
     network_mode: host
     ipc: host
     gpus: all
     working_dir: /
-    entrypoint: ["/bin/bash", "/opt/trellismx/serve.sh"]
+    entrypoint: ["/bin/bash", "/release/serve-r27-production.sh"]
     environment:
       PORT: "8000"
       SERVED_MODEL_NAME: "glm53-flash-trellismx-p8-k45"
@@ -19,11 +19,16 @@ services:
       NUM_SPECULATIVE_TOKENS: "3"
       KV_CACHE_DTYPE: "nvfp4_ds_mla"
       MAX_MODEL_LEN: "1000000"
-      MAX_NUM_SEQS: "16"
+      MAX_NUM_SEQS: "24"
       MAX_NUM_BATCHED_TOKENS: "4096"
       GPU_MEMORY_UTILIZATION: "0.97"
       CP_KV_CACHE_INTERLEAVE_SIZE: "4"
       DCP_CKV_GATHER: "auto"
+      NCCL_MIN_NCHANNELS: "8"
+      NCCL_MAX_NCHANNELS: "8"
+      VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE: "131072"
+      VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE: "86016"
+      VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD: "4096"
       VLLM_NO_USAGE_STATS: "1"
     volumes:
       - ${MODEL_ROOT:?Set MODEL_ROOT to the pinned carrier directory}:/model:ro

image-record.json

@@ -1,8 +1,8 @@
 {
   "schema_version": 4,
-  "record_id": "trellismx-r27-dcp4-20260908",
-  "title": "TrellisMX P8 GLM-5.3-Flash r27 DCP4",
-  "summary": "Reproducible r27 inference overlay with source parity, baked production launcher, measured speed receipts and audited paired NVFP4/FP8 KV development KLD.",
+  "record_id": "trellismx-reference-20260909",
+  "title": "TrellisMX GLM-5.3 Flash selected TP4 reference",
+  "summary": "Selected September9 FP8-math reference image with measured speed, exact source overlays and audited matched FP8/NVFP4 MLA KLD.",
   "model_family": "GLM-5.3-Flash",
   "release_class": "experimental",
   "distribution_role": "custom",
@@ -10,9 +10,9 @@
   "maintenance_status": "ephemeral",
   "image": {
     "repository": "verdictai/trellismx",
-    "tag": "glm53-flash-p8-r27-dcp4-20260908",
-    "digest": "sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f",
-    "reference": "verdictai/trellismx:glm53-flash-p8-r27-dcp4-20260908@sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f"
+    "tag": "glm53-flash-p8-r27-reference-20260909",
+    "digest": "sha256:ca6b80188dce154b91f49108b7d87792d2ba6328935afc71b44d1c0e6f6a1adf",
+    "reference": "verdictai/trellismx:glm53-flash-p8-r27-reference-20260909@sha256:ca6b80188dce154b91f49108b7d87792d2ba6328935afc71b44d1c0e6f6a1adf"
   },
   "base_image": {
     "repository": "voipmonitor/vllm",
@@ -33,69 +33,29 @@
     "relationship": "Community source contract; does not qualify this custom P8 runtime."
   },
   "build": {
-    "recipe_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/tree/53d93dfbc9002df7b73178327dfe99773efb680e/runtime",
-    "recipe_commit": "53d93dfbc9002df7b73178327dfe99773efb680e",
-    "build_command": "docker build --pull=false -t trellismx-r27-rebuild runtime",
-    "components": [
-      {
-        "name": "vllm",
-        "repository": "https://github.com/local-inference-lab/vllm",
-        "commit": "c88fb8847dc8bc18bca56640759f137198750085",
-        "release_reference": "Local integration commit; exact complete overlay files are published in the pinned HF recipe. Inherited binaries remain r27.",
-        "pull_requests": [],
-        "patches": [
-          {
-            "path_or_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/53d93dfbc9002df7b73178327dfe99773efb680e/runtime-files.json",
-            "sha256": "sha256:e1dfd232f438f17fbfceb2c534e60666e7b9f5c2104808826c12dd18b5f92de1",
-            "purpose": "Complete SHA256 inventory for inference overlay files; pins local integration output beyond upstream source labels.",
-            "authors": [
-              "Brandon M. Music; Local Inference Lab, ExLlamaV3, KQuant, QSRT and w4a8 lineage credited in CITATION.cff"
-            ]
-          }
-        ],
-        "overlays": []
-      },
-      {
-        "name": "b12x",
-        "repository": "https://github.com/local-inference-lab/b12x",
-        "commit": "7093ad77849cf181bcf0c30b897c54fd32dac40e",
-        "release_reference": "Local integration commit; exact complete overlay files are published in the pinned HF recipe. Inherited binaries remain r27.",
-        "pull_requests": [],
-        "patches": [
-          {
-            "path_or_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/53d93dfbc9002df7b73178327dfe99773efb680e/runtime-files.json",
-            "sha256": "sha256:e1dfd232f438f17fbfceb2c534e60666e7b9f5c2104808826c12dd18b5f92de1",
-            "purpose": "Complete SHA256 inventory for inference overlay files; pins local integration output beyond upstream source labels.",
-            "authors": [
-              "Brandon M. Music; Local Inference Lab, ExLlamaV3, KQuant, QSRT and w4a8 lineage credited in CITATION.cff"
-            ]
-          }
-        ],
-        "overlays": []
-      }
-    ],
-    "package_changes": [
-      "No dependency installs; r27 native extension binaries and packages inherited."
-    ],
-    "build_arguments": [
-      "JOVIAN_IMAGE=voipmonitor/vllm:jovian-judgement-community-20260906-r27@sha256:a298fe1cd207eaf97bd2ff2686716ed25b7009c09b36650eba732a4a7dc51512"
-    ],
+    "recipe_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/tree/main/runtime-reference-20260909",
+    "recipe_commit": "See containing HF commit",
+    "build_command": "docker build --pull=false -t trellismx-reference-rebuild runtime-reference-20260909",
+    "source_manifest": "runtime-reference-20260909/source-manifest.json",
+    "source_manifest_sha256": "2629fdb6094b54ce1cc1cc59b7dc0fe2617e0fe2ed39ff49a4e92ed72a28b252",
+    "base": "verdictai/trellismx@sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f",
+    "qualification": "Published image is the retained measured image, retagged without a rebuild. Recipe reproduces Python overlay; fresh rebuild not GPU tested.",
     "environment_defaults": [
-      "TP=4 DCP=4 KV_CACHE_DTYPE=nvfp4_ds_mla NUM_SPECULATIVE_TOKENS=3 GPU_MEMORY_UTILIZATION=0.97 MAX_NUM_SEQS=16 MAX_NUM_BATCHED_TOKENS=4096 MAX_MODEL_LEN=1000000"
+      "TP4 DCP4 MTP3 maxseq24 batch4096 maxlen1000000 GMU0.97 KVnvfp4_ds_mla",
+      "NCCL_MIN_NCHANNELS=8 NCCL_MAX_NCHANNELS=8",
+      "VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE=131072 VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE=86016 VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD=4096"
     ],
     "entrypoint_changes": [
-      "Bakes deployed launcher at /opt/trellismx/serve.sh; executes inherited r27 serve-glm53-flash launcher. PORT=8000."
-    ],
-    "result_tree": "N/A",
-    "integration_patch_sha256": "N/A"
+      "compose explicitly launches /bin/bash /release/serve-r27-production.sh; baked launcher defaults16, public compose/wrapper override24"
+    ]
   },
   "changes": {
     "inherited": [
       "r27 split-cache rebalancing, B12X attention, FlashKDA, probabilistic MTP and fairness scheduler; binaries and dependencies unchanged."
     ],
     "introduced": [
-      "Native TrellisMX P8 checkpoint loader and B12X expert path; DCP4 persistent workspace integration; complete inference source in recipe.",
-      "Production launcher defaults baked into image; licenses included."
+      "Route-hoisted FP8 native expert dispatch, tile policy and grouped FC2 paths; exact changed Python files in source manifest.",
+      "Selected collective settings and24slot serving wrapper."
     ],
     "compatibility_impact": [
       "TP4 checkpoint with 168 sidecars, pinned carrier and four SM120 GPUs; no arbitrary model conversion claim."
@@ -103,7 +63,7 @@
   },
   "tested_configurations": [
     {
-      "name": "Measured source-equivalent r27 DCP4 local deployment; public image CPU checked",
+      "name": "Retained September9 selected reference image",
       "hardware": "4x RTX PRO 6000 Blackwell 96GB, 2 Max-Q and 2 standard",
       "topology": "TP4/DCP4, PCIe gen5 x16; no expert parallelism",
       "power_and_clocks": "300 W cap per GPU; clocks and telemetry retained in benchmark JSON; no clock change by release",
@@ -111,86 +71,34 @@
       "cuda_runtime": "13.3",
       "pytorch": "2.13.0",
       "nccl": "2.31.2",
-      "engine_source": "c88fb8847dc8bc18bca56640759f137198750085 overlay on r27 binaries",
+      "engine_source": "Exact selected image Python source and manifest in runtime-reference-20260909",
       "model_revision": "168 sidecar identities in trellismx-manifest.json; carrier 520de24eabf507659eaef7c70f14fd584527facc",
       "quantization": "TrellisMX coupled K4/K5, 4.6587417643 routed bpw; native E4M3 UE8M0/32 MMA",
       "parallelism": "TP4/DCP4, EP off",
-      "kv_cache": "nvfp4_ds_mla; engine reports 23424836 effective aggregate tokens; hybrid split layout",
+      "kv_cache": "nvfp4_ds_mla; expanded benchmark engine capacity23562091 effective aggregate tokens",
       "speculative_mode": "MTP3 probabilistic draft plus standard rejection",
       "graph_mode": "FULL_AND_PIECEWISE",
-      "scheduler_limits": "16 sequences; 4096 batched tokens; 1000000 model length; prefill share 0.4",
+      "scheduler_limits": "24 sequences;4096 batched tokens;1000000 model length",
       "cache_policy": "chunked prefill and prefix caching; vram; GPU memory utilization0.97",
       "launch_command": "MODEL_ROOT=/absolute/path/to/carrier docker compose -f compose.yaml up -d"
     }
   ],
   "validation": {
-    "commands": [
-      "docker build --pull=false -t trellismx-r27-rebuild runtime",
-      "MODEL_ROOT=/model docker compose -f compose.yaml config --quiet",
-      "docker run --rm --entrypoint python -v /absolute/path/to/checkpoint:/checkpoint:ro verdictai/trellismx:glm53-flash-p8-r27-dcp4-20260908@sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f -c 'from vllm.utils.trellismx import load_overlay; print(len(load_overlay(\"/checkpoint\").records))'",
-      "Pinned llm_decode_bench.py commands and effective arguments in results/r27-20260908/README.md and JSON."
-    ],
-    "results": [
-      {
-        "name": "Deployed inference source parity",
-        "status": "passed",
-        "conditions": "Current local image57a9967b75e6 vs public build context",
-        "measurement": "SHA256 every included deployed inference file",
-        "result": "413 files matched",
-        "conclusion": "Source parity only; rebuilt image is separately identified and is not a new GPU benchmark.",
-        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-20260908/runtime-parity.json",
-        "evidence_sha256": "sha256:033767e89483ccf6290f427decbbef96911511d704358da970cee40b512207fa",
-        "category": "correctness"
-      },
-      {
-        "name": "Public image CPU overlay import and manifest inventory",
-        "status": "passed",
-        "conditions": "Public image, no GPUs passed, local HF-layout checkpoint read-only",
-        "measurement": "Import native overlay validator; verify168 file inventory, sizes, and design identities and launcher SHA",
-        "result": "168 records loaded",
-        "conclusion": "CPU manifest/inventory gate only, no complete sidecar rehash or fresh GPU launch claim.",
-        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-20260908/public-image-cpu-check.txt",
-        "evidence_sha256": "sha256:ea5d2725e24abafef7edfd7266efbde829f61153a62426e93df67274eb454d62",
-        "category": "smoke"
-      },
-      {
-        "name": "Recorded current-server speed observations",
-        "status": "passed",
-        "conditions": "Four SM120 GPUs at300W; TP4/DCP4 MTP3 NVFP4 MLA KV; one run each, no matched baseline",
-        "measurement": "C1/C2 20s streaming cells; separate10s cold-prefill targets; full JSON/logs retained",
-        "result": "C1 zero202.45t/s; prefill7692/7940/8006t/s at8200/16227/32316tokens",
-        "conclusion": "Exploratory absolute observations; no advantage or independent qualification claim; workload and MTP confounders detailed.",
-        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-20260908/README.md",
-        "evidence_sha256": "sha256:16e1e1fedf7b49dcdc1d79c156438ee016d47817861c8abdfe32964d9fe2277b",
-        "category": "performance"
-      },
-      {
-        "name": "Paired current r27 NVFP4 versus FP8 MLA KV development KLD",
-        "status": "passed",
-        "conditions": "Same32 opened conditional-fit windows; TP4/DCP4, MTP off, maxseq1; fixed NVFP4 then FP8 order, one server start per arm.",
-        "measurement": "CPU FP64 teacher-to-student KL over2046 true-decode rows/window;20,000-resample paired window BCa95.",
-        "result": "NVFP4=0.0350078183; FP8=0.0310574767; FP8-minus-NVFP4=-0.0039503416, paired BCa95[-0.0092510457,-0.0018107907].",
-        "conclusion": "Audited development comparison only; no final holdout, independent replication, MTP or long-context qualification.",
-        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-kv-cf32-20260908/comparison.json",
-        "evidence_sha256": "sha256:9376acdce0d66738f5af7bc54188ad98d298dbb47dfd3cd3faece459becb77d6",
-        "category": "correctness"
-      }
-    ],
-    "performance_claims": [
-      {
-        "name": "N/A"
-      }
-    ]
+    "speed_evidence": "results/speed-20260909/README.md",
+    "all_speed_index": "results/speed-20260909/benchmark-index.json",
+    "kld_evidence": "results/kld-reference-20260909/comparison.json",
+    "kld_audit": "results/kld-reference-20260909/audit.json",
+    "kld_level": "Already-opened conditional-fit development comparison; single server per arm; MTPoff; not independent replication or final qualification."
   },
   "limitations": {
     "known": [
-      "Existing absolute speed observations belong to source-equivalent local server image57a9967b75e6, not a newly benchmarked registry rebuild.",
-      "Historical KLD0.0341811459 uses DCP1; current paired DCP4 results use the same opened windows, MTP off and2046 true-decode rows per window. Different historical runtime/topology prevents KV-only attribution.",
-      "Weight upload is still in progress; release-status.json governs remote completeness.",
-      "Engine aggregate cache capacity does not establish1M-context accuracy or full-capacity stress."
+      "Recipe Python-source reconstruction is separate from the measured immutable Docker image.",
+      "Historical comparisons differ in image/topology; new matched KLD holds reference checkpoint/TP4/DCP4 fixed and changes KV specialization.",
+      "TR3/EXL3 matched KLD requested, not measured yet.",
+      "KV aggregate capacity is not1M-context accuracy or full-capacity stress."
     ],
     "untested": [
-      "Clean download-to-GPU serving of public image",
+      "Fresh downloaded deployment GPU test",
       "Independent repeated speed qualification"
     ],
     "unsupported": [

results/image-record-historical-20260908.json

@@ -0,0 +1,217 @@
+{
+  "schema_version": 4,
+  "record_id": "trellismx-r27-dcp4-20260908",
+  "title": "TrellisMX P8 GLM-5.3-Flash r27 DCP4",
+  "summary": "Reproducible r27 inference overlay with source parity, baked production launcher, measured speed receipts and audited paired NVFP4/FP8 KV development KLD.",
+  "model_family": "GLM-5.3-Flash",
+  "release_class": "experimental",
+  "distribution_role": "custom",
+  "qualification_status": "implemented",
+  "maintenance_status": "ephemeral",
+  "image": {
+    "repository": "verdictai/trellismx",
+    "tag": "glm53-flash-p8-r27-dcp4-20260908",
+    "digest": "sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f",
+    "reference": "verdictai/trellismx:glm53-flash-p8-r27-dcp4-20260908@sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f"
+  },
+  "base_image": {
+    "repository": "voipmonitor/vllm",
+    "tag": "jovian-judgement-community-20260906-r27",
+    "digest": "sha256:a298fe1cd207eaf97bd2ff2686716ed25b7009c09b36650eba732a4a7dc51512",
+    "reference": "voipmonitor/vllm:jovian-judgement-community-20260906-r27@sha256:a298fe1cd207eaf97bd2ff2686716ed25b7009c09b36650eba732a4a7dc51512",
+    "credit": "Local Inference Lab Jovian Judgement GLM r27; vLLM, B12X and component license obligations remain applicable."
+  },
+  "recommended_image": {
+    "reference": "voipmonitor/vllm:jovian-judgement-community-20260906-r27@sha256:a298fe1cd207eaf97bd2ff2686716ed25b7009c09b36650eba732a4a7dc51512",
+    "relationship": "Pinned community runtime inherited by this custom overlay; not a matched numerical baseline."
+  },
+  "community_wiki": {
+    "repository": "https://github.com/local-inference-lab/rtx6kpro",
+    "commit": "94b71ac2a5f9c75f6b60dd1b6e6dffda492a4942",
+    "runbook_path": "models/glm-5.3-flash.md",
+    "runbook_url": "https://github.com/local-inference-lab/rtx6kpro/blob/94b71ac2a5f9c75f6b60dd1b6e6dffda492a4942/models/glm-5.3-flash.md",
+    "relationship": "Community source contract; does not qualify this custom P8 runtime."
+  },
+  "build": {
+    "recipe_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/tree/53d93dfbc9002df7b73178327dfe99773efb680e/runtime",
+    "recipe_commit": "53d93dfbc9002df7b73178327dfe99773efb680e",
+    "build_command": "docker build --pull=false -t trellismx-r27-rebuild runtime",
+    "components": [
+      {
+        "name": "vllm",
+        "repository": "https://github.com/local-inference-lab/vllm",
+        "commit": "c88fb8847dc8bc18bca56640759f137198750085",
+        "release_reference": "Local integration commit; exact complete overlay files are published in the pinned HF recipe. Inherited binaries remain r27.",
+        "pull_requests": [],
+        "patches": [
+          {
+            "path_or_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/53d93dfbc9002df7b73178327dfe99773efb680e/runtime-files.json",
+            "sha256": "sha256:e1dfd232f438f17fbfceb2c534e60666e7b9f5c2104808826c12dd18b5f92de1",
+            "purpose": "Complete SHA256 inventory for inference overlay files; pins local integration output beyond upstream source labels.",
+            "authors": [
+              "Brandon M. Music; Local Inference Lab, ExLlamaV3, KQuant, QSRT and w4a8 lineage credited in CITATION.cff"
+            ]
+          }
+        ],
+        "overlays": []
+      },
+      {
+        "name": "b12x",
+        "repository": "https://github.com/local-inference-lab/b12x",
+        "commit": "7093ad77849cf181bcf0c30b897c54fd32dac40e",
+        "release_reference": "Local integration commit; exact complete overlay files are published in the pinned HF recipe. Inherited binaries remain r27.",
+        "pull_requests": [],
+        "patches": [
+          {
+            "path_or_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/53d93dfbc9002df7b73178327dfe99773efb680e/runtime-files.json",
+            "sha256": "sha256:e1dfd232f438f17fbfceb2c534e60666e7b9f5c2104808826c12dd18b5f92de1",
+            "purpose": "Complete SHA256 inventory for inference overlay files; pins local integration output beyond upstream source labels.",
+            "authors": [
+              "Brandon M. Music; Local Inference Lab, ExLlamaV3, KQuant, QSRT and w4a8 lineage credited in CITATION.cff"
+            ]
+          }
+        ],
+        "overlays": []
+      }
+    ],
+    "package_changes": [
+      "No dependency installs; r27 native extension binaries and packages inherited."
+    ],
+    "build_arguments": [
+      "JOVIAN_IMAGE=voipmonitor/vllm:jovian-judgement-community-20260906-r27@sha256:a298fe1cd207eaf97bd2ff2686716ed25b7009c09b36650eba732a4a7dc51512"
+    ],
+    "environment_defaults": [
+      "TP=4 DCP=4 KV_CACHE_DTYPE=nvfp4_ds_mla NUM_SPECULATIVE_TOKENS=3 GPU_MEMORY_UTILIZATION=0.97 MAX_NUM_SEQS=16 MAX_NUM_BATCHED_TOKENS=4096 MAX_MODEL_LEN=1000000"
+    ],
+    "entrypoint_changes": [
+      "Bakes deployed launcher at /opt/trellismx/serve.sh; executes inherited r27 serve-glm53-flash launcher. PORT=8000."
+    ],
+    "result_tree": "N/A",
+    "integration_patch_sha256": "N/A"
+  },
+  "changes": {
+    "inherited": [
+      "r27 split-cache rebalancing, B12X attention, FlashKDA, probabilistic MTP and fairness scheduler; binaries and dependencies unchanged."
+    ],
+    "introduced": [
+      "Native TrellisMX P8 checkpoint loader and B12X expert path; DCP4 persistent workspace integration; complete inference source in recipe.",
+      "Production launcher defaults baked into image; licenses included."
+    ],
+    "compatibility_impact": [
+      "TP4 checkpoint with 168 sidecars, pinned carrier and four SM120 GPUs; no arbitrary model conversion claim."
+    ]
+  },
+  "tested_configurations": [
+    {
+      "name": "Measured source-equivalent r27 DCP4 local deployment; public image CPU checked",
+      "hardware": "4x RTX PRO 6000 Blackwell 96GB, 2 Max-Q and 2 standard",
+      "topology": "TP4/DCP4, PCIe gen5 x16; no expert parallelism",
+      "power_and_clocks": "300 W cap per GPU; clocks and telemetry retained in benchmark JSON; no clock change by release",
+      "driver": "610.57.04",
+      "cuda_runtime": "13.3",
+      "pytorch": "2.13.0",
+      "nccl": "2.31.2",
+      "engine_source": "c88fb8847dc8bc18bca56640759f137198750085 overlay on r27 binaries",
+      "model_revision": "168 sidecar identities in trellismx-manifest.json; carrier 520de24eabf507659eaef7c70f14fd584527facc",
+      "quantization": "TrellisMX coupled K4/K5, 4.6587417643 routed bpw; native E4M3 UE8M0/32 MMA",
+      "parallelism": "TP4/DCP4, EP off",
+      "kv_cache": "nvfp4_ds_mla; engine reports 23424836 effective aggregate tokens; hybrid split layout",
+      "speculative_mode": "MTP3 probabilistic draft plus standard rejection",
+      "graph_mode": "FULL_AND_PIECEWISE",
+      "scheduler_limits": "16 sequences; 4096 batched tokens; 1000000 model length; prefill share 0.4",
+      "cache_policy": "chunked prefill and prefix caching; vram; GPU memory utilization0.97",
+      "launch_command": "MODEL_ROOT=/absolute/path/to/carrier docker compose -f compose.yaml up -d"
+    }
+  ],
+  "validation": {
+    "commands": [
+      "docker build --pull=false -t trellismx-r27-rebuild runtime",
+      "MODEL_ROOT=/model docker compose -f compose.yaml config --quiet",
+      "docker run --rm --entrypoint python -v /absolute/path/to/checkpoint:/checkpoint:ro verdictai/trellismx:glm53-flash-p8-r27-dcp4-20260908@sha256:1c8a10d2b21bd6ed5a7ca4a29bcc3900d29acc3ce42e1d722b9ebaa74357de3f -c 'from vllm.utils.trellismx import load_overlay; print(len(load_overlay(\"/checkpoint\").records))'",
+      "Pinned llm_decode_bench.py commands and effective arguments in results/r27-20260908/README.md and JSON."
+    ],
+    "results": [
+      {
+        "name": "Deployed inference source parity",
+        "status": "passed",
+        "conditions": "Current local image57a9967b75e6 vs public build context",
+        "measurement": "SHA256 every included deployed inference file",
+        "result": "413 files matched",
+        "conclusion": "Source parity only; rebuilt image is separately identified and is not a new GPU benchmark.",
+        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-20260908/runtime-parity.json",
+        "evidence_sha256": "sha256:033767e89483ccf6290f427decbbef96911511d704358da970cee40b512207fa",
+        "category": "correctness"
+      },
+      {
+        "name": "Public image CPU overlay import and manifest inventory",
+        "status": "passed",
+        "conditions": "Public image, no GPUs passed, local HF-layout checkpoint read-only",
+        "measurement": "Import native overlay validator; verify168 file inventory, sizes, and design identities and launcher SHA",
+        "result": "168 records loaded",
+        "conclusion": "CPU manifest/inventory gate only, no complete sidecar rehash or fresh GPU launch claim.",
+        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-20260908/public-image-cpu-check.txt",
+        "evidence_sha256": "sha256:ea5d2725e24abafef7edfd7266efbde829f61153a62426e93df67274eb454d62",
+        "category": "smoke"
+      },
+      {
+        "name": "Recorded current-server speed observations",
+        "status": "passed",
+        "conditions": "Four SM120 GPUs at300W; TP4/DCP4 MTP3 NVFP4 MLA KV; one run each, no matched baseline",
+        "measurement": "C1/C2 20s streaming cells; separate10s cold-prefill targets; full JSON/logs retained",
+        "result": "C1 zero202.45t/s; prefill7692/7940/8006t/s at8200/16227/32316tokens",
+        "conclusion": "Exploratory absolute observations; no advantage or independent qualification claim; workload and MTP confounders detailed.",
+        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-20260908/README.md",
+        "evidence_sha256": "sha256:16e1e1fedf7b49dcdc1d79c156438ee016d47817861c8abdfe32964d9fe2277b",
+        "category": "performance"
+      },
+      {
+        "name": "Paired current r27 NVFP4 versus FP8 MLA KV development KLD",
+        "status": "passed",
+        "conditions": "Same32 opened conditional-fit windows; TP4/DCP4, MTP off, maxseq1; fixed NVFP4 then FP8 order, one server start per arm.",
+        "measurement": "CPU FP64 teacher-to-student KL over2046 true-decode rows/window;20,000-resample paired window BCa95.",
+        "result": "NVFP4=0.0350078183; FP8=0.0310574767; FP8-minus-NVFP4=-0.0039503416, paired BCa95[-0.0092510457,-0.0018107907].",
+        "conclusion": "Audited development comparison only; no final holdout, independent replication, MTP or long-context qualification.",
+        "evidence_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/results/r27-kv-cf32-20260908/comparison.json",
+        "evidence_sha256": "sha256:9376acdce0d66738f5af7bc54188ad98d298dbb47dfd3cd3faece459becb77d6",
+        "category": "correctness"
+      }
+    ],
+    "performance_claims": [
+      {
+        "name": "N/A"
+      }
+    ]
+  },
+  "limitations": {
+    "known": [
+      "Existing absolute speed observations belong to source-equivalent local server image57a9967b75e6, not a newly benchmarked registry rebuild.",
+      "Historical KLD0.0341811459 uses DCP1; current paired DCP4 results use the same opened windows, MTP off and2046 true-decode rows per window. Different historical runtime/topology prevents KV-only attribution.",
+      "Weight upload is still in progress; release-status.json governs remote completeness.",
+      "Engine aggregate cache capacity does not establish1M-context accuracy or full-capacity stress."
+    ],
+    "untested": [
+      "Clean download-to-GPU serving of public image",
+      "Independent repeated speed qualification"
+    ],
+    "unsupported": [
+      "Other GPU architectures, TP sizes, arbitrary model encoders; inference-only release."
+    ]
+  },
+  "support": {
+    "owner": "Brandon Music",
+    "contact": "https://github.com/brandonmmusic-max",
+    "issue_tracker": "https://github.com/brandonmmusic-max/glm53-hadamard-shapleymcg-kld/issues",
+    "support_thread": "https://github.com/brandonmmusic-max/glm53-hadamard-shapleymcg-kld/issues/5",
+    "thread_status": "active",
+    "support_commitment": "ephemeral",
+    "triage_policy": "Keep reports in this thread. Escalate upstream only after reproduction on the recommended image or a minimal reproducer identifies the responsible source change.",
+    "superseded_by": "N/A"
+  },
+  "publication": {
+    "record_url": "https://huggingface.co/brandonmusic/GLM-5.3-Flash-TrellisMX-MXFP8/blob/main/image-record.json",
+    "main_channel_link": "N/A",
+    "main_channel_link_count": 0,
+    "bot_listing": "not-applicable",
+    "maintainer_approval_url": "N/A"
+  }
+}

results/kld-reference-20260909/README.md

@@ -0,0 +1,33 @@
+# Reference-stack KV-cache KLD — September 9, 2026
+
+| Cache | Mean KL(teacher || student) | Window BCa 95% interval |
+| --- | ---: | --- |
+| Current r27 DCP4 / NVFP4 MLA KV | 0.0354562238 | [0.0295558848, 0.0434620368] |
+| Current r27 DCP4 / FP8 MLA KV | 0.0319451732 | [0.0268267482, 0.0387948613] |
+
+FP8 minus NVFP4: -0.0035110506; paired-window BCa 95% interval
+[-0.0081718772, -0.0013675941].
+FP8 has lower observed KLD in 22/32 windows.
+
+Exact same 32 previously opened conditional-fit windows and BF16 teacher as the
+September 8 measurement: 2048 input tokens, 2047 captured predictions, exclude
+row zero, leaving 2046 true-decode rows/window. CPU FP64 KL over vocabulary154880;
+equal mean of window means; BCa20000 resamples with seed20260902. Teacher stored F32.
+TP4/DCP4, MTP off, maxseq1, batch4096, GMU0.97, maxlen1M; prefix caching enabled,
+all prefix-hit counters zero. FP8 first then NVFP4, one server preparation each.
+Window uncertainty does not estimate server-run variability. These are development
+measurements, not untouched-final qualification or independent reproduction.
+
+Selected image: verdictai/trellismx@sha256:ca6b80188dce154b91f49108b7d87792d2ba6328935afc71b44d1c0e6f6a1adf.
+Capture-only derivative image, seal, source identities and all64 window scores are
+in comparison.json. Capture seam preserves pre-mask logits and forces exact histories.
+NCCL8, plain one-shot cutoff131072, fused cutoff86016, shared expert threshold4096.
+Checkpoint and FP8 weight/activation math held fixed; KV dtype and native cache
+specialization vary. Capture timing is not a throughput measurement.
+
+Raw logits retired only after hashes, durable numerical scores and receipts.
+Audit recomputes aggregates from retained FP64 scores and verifies metadata,
+zero prefix hits, row masks and receipt hashes; it is not independent recapture.
+No measured windows excluded or rerun. Prior image/cache values remain historical.
+The separate production recipe uses MTP3 and24slots; its KV capacity must not be
+confused with capacity from this MTP-disabled correctness profile.

results/kld-reference-20260909/audit.json

@@ -0,0 +1,30 @@
+{
+  "status": "passed",
+  "arms": [
+    {
+      "arm": "fp8",
+      "windows": 32,
+      "prediction_rows": 65504,
+      "true_decode_rows": 65472,
+      "all_receipt_hashes_and_masks_pass": true,
+      "all_prefix_hit_counters_zero": true,
+      "raw_logits_retired_after_scoring": true,
+      "source_runtime_audit_sha256": "6a3393053b83b33b41a7d49de2fab1803b1efc4843ec5862c1acd8467b993fa4",
+      "source_startup_sha256": "00ce74cfce281b627289f3322829d357467f5bc1f026677ef27cda93ed43fce3"
+    },
+    {
+      "arm": "nvfp4",
+      "windows": 32,
+      "prediction_rows": 65504,
+      "true_decode_rows": 65472,
+      "all_receipt_hashes_and_masks_pass": true,
+      "all_prefix_hit_counters_zero": true,
+      "raw_logits_retired_after_scoring": true,
+      "source_runtime_audit_sha256": "74d7d1914a55a73b5e9de196f557c658603a9501acd2fafb1e540d496b1c7185",
+      "source_startup_sha256": "cae3cb749c61cc18fd23286f41f79266e32bfdc551e0575b3515d21eb993036a"
+    }
+  ],
+  "producing_analysis_sha256": "0b9a6cb0c0f1d1b4b7784190d38e6f8fdf3dbefe5cb619be6f0dc065a742d8a3",
+  "verifier_sha256": "124494f28c018e0945349fb4479d3be9f9164cdf7ce298989de25367f4106ade",
+  "claim": "Receipt and score audit by the same operator; not independent replication."
+}

results/kld-reference-20260909/comparison.json

@@ -0,0 +1,1053 @@
+{
+  "schema": "trellismx.r27-kv-cf32.v1",
+  "evidence": "controlled comparison on already-opened conditional-fit; one prepared run per KV mode, fixed order; not final qualification",
+  "arms": {
+    "fp8": {
+      "mean_true_decode_kld": 0.03194517316654619,
+      "window_bca95": [
+        0.026826748240418287,
+        0.03879486133125964
+      ],
+      "per_window": [
+        {
+          "window_id": "conditional-fit-0056",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02440979598998064,
+          "including_prefill_mean_kld": 0.02501134181859523,
+          "prefill_row_kld": 1.2557741071640407,
+          "raw_sha256": "72bc5007903d7885f04ff28c03128901bfd15d7a94385772af929125edb650ff",
+          "score_sha256": "3d00fe1e7f7505cb4186e5d67c6e7c5df53b904b6c041230b9e227ad34afcd93",
+          "teacher_sha256": "55a70a330f39c5f23294c23d270a1ddc53031def99a42206e2706d4d0bb4466b",
+          "token_sha256": "219d8cd8873c6cf41f52782297cc5cfa9c4ed9666f2bc30d84cee9caa68ea744",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0008",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.06994665102705569,
+          "including_prefill_mean_kld": 0.07040344776705193,
+          "prefill_row_kld": 1.0050095777993453,
+          "raw_sha256": "f9c863fed0667b09b18edeba791c9ab4018c68e010ff488c0b73c1b0a0a67721",
+          "score_sha256": "a1bfbe0a93478e2939a51800dea0ad26d56c6aca01ad3633f18a3709e1601bad",
+          "teacher_sha256": "040641247b2e060035d89c2e7345e029d7c142983585e7c5d63dc818486f650d",
+          "token_sha256": "581d72e3e78b15c02d9e73f6bc742260cb57cdcd74337f4419c8f90cbc7c9c11",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0100",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.022149540339761863,
+          "including_prefill_mean_kld": 0.023738251318587308,
+          "prefill_row_kld": 3.2742409139954454,
+          "raw_sha256": "7b082ebc4250bb8b032c82bdb02a15019b947dfe54131e776a0af539fab8b2d8",
+          "score_sha256": "cef09f41fd3bc3e395b4721344d0b546796c981917ac64c997c78cbcedebe6b9",
+          "teacher_sha256": "f62991383b3ef5e8db0ed71442b17c428972e58da1c868c6aeb0f3fa143c865b",
+          "token_sha256": "7f008ec99068f373c6e15dd059ee977fb70e6dc832f46e009f0f299d48d19892",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0094",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.016832794344000746,
+          "including_prefill_mean_kld": 0.016934019957995376,
+          "prefill_row_kld": 0.22404162619101314,
+          "raw_sha256": "29d51a000848b3f9a158070703dc4b25be593e4890deb83f572291379b54668d",
+          "score_sha256": "7ade9eefb18e3bdcffc63d64abcc88dd7260bf9b97b4be788da8e8c492710e35",
+          "teacher_sha256": "656911b8fb34d7f42e984dd21311caf9ef56db0b7618b414cad4089562d93cba",
+          "token_sha256": "6b6a9b21c49a9984cc8c5ea90e0bda6db5761d039c7cdd335dc65e30444e017b",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0083",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.03015300663995215,
+          "including_prefill_mean_kld": 0.03048617506449642,
+          "prefill_row_kld": 0.7121487716820719,
+          "raw_sha256": "d71da9253ed6946f0fb5632c845d2ffb2f7c03b7b6f7cc25b959b0e14bfe8484",
+          "score_sha256": "0f1cc2f89d5738573eea4d2b3dc1a6fa7fc02180cd2e7cd8a2a5ba68892007a6",
+          "teacher_sha256": "66a7870a1f13c8b5d647c1681fca005bb0f55a6e502e94f282da673a6bed3f1b",
+          "token_sha256": "3df9d2872815e061ff8d32a9a19da0d11d80ac67070cccd28579b24ec3079552",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0021",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.0390320697408021,
+          "including_prefill_mean_kld": 0.03997988073234378,
+          "prefill_row_kld": 1.9792011694266036,
+          "raw_sha256": "63ef61c53db88b1eaf3bd30a833593d48ea2bd25313a7c500c8d754e8142f04c",
+          "score_sha256": "6e41161d9b1edb5b19299d74682f682d5fd8643794befb9c111ff1557bcadc25",
+          "teacher_sha256": "ebf76910c4db8f6955ad03f2ec7352e2f8aaf15e0c939d28f9a7d78f6bd7451b",
+          "token_sha256": "dbd9f2539cb3e97127ae760a04ca5b3456b0b13d6ce8698f88b215cc68b8f1a9",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0098",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04434523531376396,
+          "including_prefill_mean_kld": 0.04552245091649955,
+          "prefill_row_kld": 2.4541055741135085,
+          "raw_sha256": "f2b99e96d0776ba966989383185fc18a776722a9de83a1d7d7d22f833cb0fe15",
+          "score_sha256": "33363d670f8d1e088ebadd3edbfc00e8aac3ebe3a47aac8c64edc9698f5bc44e",
+          "teacher_sha256": "16e3d9524100836c020535635f7722a4c93ce0fdddc2ad0781f25bda77e8bcc9",
+          "token_sha256": "749bda3291a0323d286b0018b7dc3e0f469de8afd5d194370cde357b7b315c7d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0080",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04201227846375734,
+          "including_prefill_mean_kld": 0.04314755010220913,
+          "prefill_row_kld": 2.365913322374568,
+          "raw_sha256": "1646b7c5c5387e320517272052d9b5f01c168e44dbc8497ab66f3b55fd473db1",
+          "score_sha256": "8a1f8466c13bb37d2e1c8f04463d90905c9acddf054d1271678a2cea58b35bfe",
+          "teacher_sha256": "e9c1a77490a5db63d410297bd8febe06adf4df2a6f4e16552db187738c9289d2",
+          "token_sha256": "a31d43f38710eb305c684501a2b02909a4e82c43101c07e76e4a3ca4edb4d0dd",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0081",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.0255516960804676,
+          "including_prefill_mean_kld": 0.025679992777695365,
+          "prefill_row_kld": 0.28817503530570604,
+          "raw_sha256": "dfbb9d556272241c4e45c7a48302895559e7567d5cf85ddfdc788a6dd616a117",
+          "score_sha256": "601136d9bf9208532067fb0fe807c5200ae263e4dd85d01c21b1df89fbed8095",
+          "teacher_sha256": "5ae07760d91030d1eba925b76ea463309baa49c2e7d5f5e14409a23ced7fd20d",
+          "token_sha256": "350628488b94d59f3e4f3e99a418913544ea1fe7eacee14a642b996d9c614d16",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0123",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.033191169179572766,
+          "including_prefill_mean_kld": 0.03317634067093455,
+          "prefill_row_kld": 0.0028372119971554416,
+          "raw_sha256": "0cfa11435295489e24da83d30b42710f530f65a99e473c64755317a2391f8940",
+          "score_sha256": "0c5504a8e70139d48221451c5569e222ac3f55d13a17de9e65adfe9ae645bfa3",
+          "teacher_sha256": "4a24b5b750ba98c06d9bc69e19334aff89e7bd4f9429592598af5d62de7ecb42",
+          "token_sha256": "aab84ee426f91621d123aa68bd53f75e220fbaf8cb270db1feab1529a62ddbf0",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0090",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.03953329012985432,
+          "including_prefill_mean_kld": 0.04086400060183276,
+          "prefill_row_kld": 2.7634976262697193,
+          "raw_sha256": "8067287b0733598d6f9d01ad790e350dab5bc4bc32eba7430a05d38adbfdf455",
+          "score_sha256": "4ac0add16333368e2109970786f1ab2b8a6c1f0532218bdb3dbe94fb3045551e",
+          "teacher_sha256": "db2a5fe33c9e457a22ebeb31a7073c1dac666b877503e9b4e90a8d6abf7b8a32",
+          "token_sha256": "a06e76993db717afee847fcc4d86f10b52dd23cdce5dae916f32d238d08d886c",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0062",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02626102483081332,
+          "including_prefill_mean_kld": 0.02663675046149981,
+          "prefill_row_kld": 0.7953713908460541,
+          "raw_sha256": "9f2f72b2e7ea4ffb425c094bd8d3df7452743a16ec6bcc22065dd25f613670f3",
+          "score_sha256": "9ecc5d9141126ec6f81b86eb28e9c785088f25d994ee8558b5bc7218ca325940",
+          "teacher_sha256": "d6f7883a533625c7c75119fe7403b48bf81885b1de0b311a1a0571949f7f231c",
+          "token_sha256": "7c6d2ef2f4b7c7091a517f6ccd778009f9ee144165c9aecf2f8f28aea7b3753f",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0047",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02201209190929644,
+          "including_prefill_mean_kld": 0.026359335852221596,
+          "prefill_row_kld": 8.920820443077092,
+          "raw_sha256": "6594c75d2eb3fef5ac2049c9924e2be8119f2e0e9948b5c114f1e9b5ea9233b3",
+          "score_sha256": "92f15873c7c6837cebdb8a1d16767da58ec6a2a13091394ad3028b2e2c0fe716",
+          "teacher_sha256": "8513cdcb9940786c7c42bf14d2f5f2cd452791e4ec6036c06a1f973eb969018d",
+          "token_sha256": "89730ee6826d4302c352156d5df766c51622a8b6681b8052d4ca6e7d481177fe",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0051",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.013267071463714407,
+          "including_prefill_mean_kld": 0.013608489001681363,
+          "prefill_row_kld": 0.7121487716820719,
+          "raw_sha256": "43626b428d36c588448d0d2756b020f63e23237c04d852b958f79ca7f601298c",
+          "score_sha256": "30293b5eabd64e4bdfce7177844412ab5834b7f6c199aba692592cc6206665fa",
+          "teacher_sha256": "177477c19b54dc0b0230e19202d48ab71d52f2b1a37fe383511bb64f0aae2fcf",
+          "token_sha256": "c79269b16560e7301549593443c9394007e7a957ad149c46224e7ad271dc6a18",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0055",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.020627011525195565,
+          "including_prefill_mean_kld": 0.021473521460828236,
+          "prefill_row_kld": 1.7534328497652714,
+          "raw_sha256": "e470e9ded871cbe5d724b0416e973bb98b60bbe4cf76f6348feb871494464fbd",
+          "score_sha256": "2c690f7a591f3e932abfd73138a1185cdb253a37ce48cb85292b712ebd412ff8",
+          "teacher_sha256": "7679c828eae4bf08f17598d044904693fc90c05b9d3f189dddeeb846100e43a7",
+          "token_sha256": "feec9ac508172dd275ec6452a03493acab5ffdff9bb57161317c77f24daf90f2",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0031",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.01655011698258689,
+          "including_prefill_mean_kld": 0.020900029208329205,
+          "prefill_row_kld": 8.920820443077092,
+          "raw_sha256": "e94b496a3cec9e30b6bcbd429f1d4757f38bcd031ce303872442bf994404bec8",
+          "score_sha256": "4486c4b80fe284775cb7e57a99199749eb5b8892e397e56d1904c3d27459e1a7",
+          "teacher_sha256": "e04a6c04a951efb5a178927e91457535cc67745ce5c81e785cb6d963aac6ce83",
+          "token_sha256": "2cff3482b4212537d0c35b88b8a0f479502c8905952430c973d25b1ab0bb5818",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0003",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.019912232579636858,
+          "including_prefill_mean_kld": 0.022605333926454176,
+          "prefill_row_kld": 5.5326906895146735,
+          "raw_sha256": "2f0ef47bdc989be608cabfb611ac42a5710712f17f219bec543b0594ab7dadd6",
+          "score_sha256": "b72e81ff056ef12b3edc2b59ed5425223aef7120e8f8f144409329512b0cdb6c",
+          "teacher_sha256": "7677c89cc3dee8e2a64e3c7947add39475f828921a3a8fe2e92b379b3ed8320b",
+          "token_sha256": "d416537c6e8273747bf025e97287a0bb6e3ff047bfee65cf54e4dfe287569b93",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0009",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04340137042823817,
+          "including_prefill_mean_kld": 0.04350757595049225,
+          "prefill_row_kld": 0.2608040744823556,
+          "raw_sha256": "6e8cc5a72d7df8f166b66eb2e84d1fad1947b65002b6112bddf3e824f4adeb39",
+          "score_sha256": "5274f9fb536c8add3acf2f38dcb460002193f9eb13e9425607330a29a9389876",
+          "teacher_sha256": "00701ad8bf4a4a5eedcedd4eafd956830c9ff1c790b15074ca721a3462abf930",
+          "token_sha256": "fc1166707f5938a914faf364dab717b0de3ea0e8451ed1a3fb399469e659b105",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0010",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.016893295516952813,
+          "including_prefill_mean_kld": 0.016892189177210668,
+          "prefill_row_kld": 0.014628618064772496,
+          "raw_sha256": "30eddaed5438035e494759f7a8e5def1346ff8cc3d1144ff135459781ee8c36d",
+          "score_sha256": "a982b303a18638a696fd13660a23cfedfea7373a61663b08e0438d7834345726",
+          "teacher_sha256": "f9bf4519890b70d972dfdf9760181418eb7c55b6237c0108f01d44df89943278",
+          "token_sha256": "ddd9d769a76eba74bc8e680ff25736a3b9146371801d1d2b29e4dc48f6d256a2",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0011",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.016308449522884096,
+          "including_prefill_mean_kld": 0.02065847980796188,
+          "prefill_row_kld": 8.920820443077092,
+          "raw_sha256": "b6098a0ba73fd75f000bbfd23b395eac80f5c3529f5cc55b592cd44c49d3f272",
+          "score_sha256": "4c9d123aaaaf9eb62e494447f1cd08cb61b905d70df3e4b461ce06c48fd4b15b",
+          "teacher_sha256": "9f37c4ead38bb80a5d85f825b156e5b493b6d3f1ca9dabfad06b0416f37db286",
+          "token_sha256": "c4b96acfa47944be3485b20eab76eb16d59913c9fedb5001a537335a3ee13614",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0030",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02726396346416777,
+          "including_prefill_mean_kld": 0.027262084282667205,
+          "prefill_row_kld": 0.023417278932510124,
+          "raw_sha256": "bc8f5f049ba9fea541f21abd4109e523067d52c9b5ef8cd3df6be759c2d1d954",
+          "score_sha256": "5f7dc8e0eafcb5371f8f2a34cd406787b5c35d15cede4c2b167f685629c2f84f",
+          "teacher_sha256": "90c08d120cfac926855e15c106dba1e242099bcd0ae6da7655bfaa54d5ce2420",
+          "token_sha256": "d644a79da2ba01cefc5f2020eab9e8662d45ba3025377844e4bf4d3f1d36802d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0032",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.033507635709301424,
+          "including_prefill_mean_kld": 0.03461520895529943,
+          "prefill_row_kld": 2.3007100702672156,
+          "raw_sha256": "06846c021ae34fc753d82f7163b2c4fecb1ea4d093d1a3c4e494afbff734a9a0",
+          "score_sha256": "e561f17f20aa32b915f1df05ee4af640cb8d053ad9c8df19c6fffdf27d9f01b0",
+          "teacher_sha256": "1aa70ccf8e224c3a58834e2232a3a2477768db29ffeca39f5b27c17ad8c40c2b",
+          "token_sha256": "88745c333ead140a6fb05929031613429cbd42e25881d92add968a94a5d2d930",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0035",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.015753355570421665,
+          "including_prefill_mean_kld": 0.015747717272730583,
+          "prefill_row_kld": 0.004211760196777308,
+          "raw_sha256": "c6735072659f7f4c6f1c9dc5aa12bd496cd81be2c55f5198023d60e5b2bebcc8",
+          "score_sha256": "0d8da276e156afdb165e71e5982d7980741407a093267641016b7880cd2c0a9d",
+          "teacher_sha256": "4e958dce74e093eb62ba5ac210fe0a4e50f12f1875de678769700cde206b429d",
+          "token_sha256": "49f702136da39db9e762b6815a1952c0eba0a1fb04af22fb28a27bb63e2a41dc",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0040",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04597109826675457,
+          "including_prefill_mean_kld": 0.04706420401414631,
+          "prefill_row_kld": 2.2835585631776425,
+          "raw_sha256": "615796ddf57e73c358a672505908dee823ae3da813ad0e43d69d2e5b15c07d80",
+          "score_sha256": "53206f5c9c253f5406ea7b054cfa816db5112665f5c2be6561e2016d1fa402ed",
+          "teacher_sha256": "f4c96798fc9eb5564d9aec52779cc0ce88bc55312dcd7663240938d711c8a90d",
+          "token_sha256": "50f3bc08ab37c551a05364eb48f0d726742f19331e79114dd5a9fb3c42ce6d4f",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0041",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.07671995932113433,
+          "including_prefill_mean_kld": 0.077568246897156,
+          "prefill_row_kld": 1.8131646274374966,
+          "raw_sha256": "f3486e51a876933be29f0ff6656920a2b778351cbc722f24c0829d6e5735901e",
+          "score_sha256": "f94f09523579b471bd22d57bee5657add70718f81bf13271592ae266f0c401a3",
+          "teacher_sha256": "c2bbe4ea31532baceff966423c6ce6e042aebb5a2eb1de45e382a7d3df6f6827",
+          "token_sha256": "8acf2614f0f787c185af112f21bcdf247c9b6ae76d292ada28f476e2a7b0241d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0046",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.07234383010849382,
+          "including_prefill_mean_kld": 0.07241689437092738,
+          "prefill_row_kld": 0.22190637530999638,
+          "raw_sha256": "04c26659f459b862e0584309731b22e5d21d01a9e4449b726053a16b3796d642",
+          "score_sha256": "fcd1f44a5795aa1c23ff4c21f90db35b93a5419ece26a0553a4518b3e566970b",
+          "teacher_sha256": "3caa5c58fb0b0e80129d551b39867fd52fefd93830060bb5655862fdd0e3b8ce",
+          "token_sha256": "3f2df1c090abf4473163a078c168fec33ab98ed6415985b92449caa861c04444",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0060",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.019628373069938383,
+          "including_prefill_mean_kld": 0.020660688213730436,
+          "prefill_row_kld": 2.1327774724122643,
+          "raw_sha256": "688a90f82f45f3dad38f0a72a3b663e0a71ffc5e9ca80528e53e43879bba4610",
+          "score_sha256": "d96ce7e3e1f13a8b49afb666fc3051cb23760b27f7ede74d4c5405d18de22f44",
+          "teacher_sha256": "d9e6344664b92527ee58d3bca038702676b357e7052ade03ac4bc9c70908cc34",
+          "token_sha256": "04ffe38aafcd9407b2846d59e8d797bee04b97f00683f8bc74d475a2d6b8592b",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0063",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.026129365452294863,
+          "including_prefill_mean_kld": 0.026535091377105725,
+          "prefill_row_kld": 0.8566503335401285,
+          "raw_sha256": "01445f5b0df02c5d68900360da39e4d7c52ffc099dc8f89d58ceecce3d06b305",
+          "score_sha256": "b34a6f8bb4a69bfc147bc76df6b8ca1fae7425c53a02ba4cd570b92df5cefc77",
+          "teacher_sha256": "a9bb59659437c1df5b3f3ef0b54eda61402842a2597b76b0124f58b59ecf3ab4",
+          "token_sha256": "38003e2d26db850dd8ff39f2e95fa3d1be9a34838c70d9f7e87056b7ef50b67d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0074",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04218677374740498,
+          "including_prefill_mean_kld": 0.04276227996673919,
+          "prefill_row_kld": 1.2202480047245254,
+          "raw_sha256": "6b1b3fadfdc741d4dae747cc9bba50570a49e4e33a23221c4693775904ec7b7b",
+          "score_sha256": "67e46c4c9f47e3dcdace4de241760311be0a3d8e50d34e2efa8f801ad0c51ee1",
+          "teacher_sha256": "440341730d063ae27bbc15bcbfc879d9d16b3b25691d97b8cfd60be8f32f2313",
+          "token_sha256": "6f6618094ffce3f44b50f94cc533d338a3bfbbda7432890cd5ad2c13efd50129",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0099",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.014557495693748427,
+          "including_prefill_mean_kld": 0.014584803788056863,
+          "prefill_row_kld": 0.07045716474311164,
+          "raw_sha256": "d17316f3263c27d2276a4eaf0e0c9a0db0ff6442d7414be32326aab7fbeb7152",
+          "score_sha256": "9af87d56dc24e6b616ffd759b55c7c64c3698effc4a5432e1a93ff9ca68725a1",
+          "teacher_sha256": "da8dea840dd57564f0b3675872eaf297631b38a6d8dabcde7f8b54db06623e9a",
+          "token_sha256": "07e37956eb606026bdd7479ac5516275e92c0ee834c3c0439a3f300f9099608d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0113",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.053326012340420864,
+          "including_prefill_mean_kld": 0.05418572832239306,
+          "prefill_row_kld": 1.8131646274374966,
+          "raw_sha256": "c4e382d7ab1b99a2bc226cabd882f2c7c4e0763163aaff6a1e1374372378ee32",
+          "score_sha256": "ca2567079ef41e93fe9ba19a536fa45619090dcf218a0dd1c1b8b40924fee362",
+          "teacher_sha256": "6dc5f38f3436d6833b0ab68ce34e0f7ee5cde80359c26602526d8de512fd3207",
+          "token_sha256": "742ab50951388e712d20dab49b12ad7eb31490cfe5091a2fbc76f31ed8fa1b09",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0118",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.012467486577109195,
+          "including_prefill_mean_kld": 0.012881541710171803,
+          "prefill_row_kld": 0.8600383439562671,
+          "raw_sha256": "6051661893d329024bb0b34ae027b8ec6075f01634cea83d4814916c1440f807",
+          "score_sha256": "a3e717d8dbdf162e89757147d2143be0ca13d6aee8396556d949646361133efc",
+          "teacher_sha256": "e296287593aa0dc89b695a6a082a224a85904757c4a3cefa69729394d1469f89",
+          "token_sha256": "2e03fdfcf4d54fc3174ee4620d5efd9f2e5c10e2e6e0b1217577f08392a09749",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        }
+      ],
+      "domains": {
+        "axis1_general": 0.030614171915487813,
+        "axis2_legal": 0.04639708824943424,
+        "axis3_code_agentic": 0.031949470625508854,
+        "axis4_reasoning_termination": 0.018819961875753848
+      },
+      "correctness_profile_engine_capacity_tokens": [
+        19355555
+      ]
+    },
+    "nvfp4": {
+      "mean_true_decode_kld": 0.03545622377215776,
+      "window_bca95": [
+        0.029555884771305184,
+        0.04346203678097485
+      ],
+      "per_window": [
+        {
+          "window_id": "conditional-fit-0056",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.023563083317839253,
+          "including_prefill_mean_kld": 0.025769605740888088,
+          "prefill_row_kld": 4.540314483298789,
+          "raw_sha256": "435e6c97b6f953bed64d5bc5fe1a66a22a35b5c8a0a5e181b0054577b69c89cd",
+          "score_sha256": "76b63f8ba02de8d5c913bcc6870fcf73890f72f8eeecae981741c148a0886c71",
+          "teacher_sha256": "55a70a330f39c5f23294c23d270a1ddc53031def99a42206e2706d4d0bb4466b",
+          "token_sha256": "219d8cd8873c6cf41f52782297cc5cfa9c4ed9666f2bc30d84cee9caa68ea744",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0008",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.06856101759452098,
+          "including_prefill_mean_kld": 0.06958029764551615,
+          "prefill_row_kld": 2.1550272819816234,
+          "raw_sha256": "0bcd5339efda6b717b15343f5de53e83d0757e1fd96aa9b8e08840b11fd32be2",
+          "score_sha256": "592c290d69ded521dfbceb1bfda56d0a23550bf6bc90c1b33692cecfe34aa2ee",
+          "teacher_sha256": "040641247b2e060035d89c2e7345e029d7c142983585e7c5d63dc818486f650d",
+          "token_sha256": "581d72e3e78b15c02d9e73f6bc742260cb57cdcd74337f4419c8f90cbc7c9c11",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0100",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02667201133477343,
+          "including_prefill_mean_kld": 0.027072764083237236,
+          "prefill_row_kld": 0.8470128874401833,
+          "raw_sha256": "b75fba8ee764641d60da5e9aa16fc0d8331c1f6fc90bf9da54c8fd9d3583dd6b",
+          "score_sha256": "83558033705994a6c4563da73feca28252fde4e18e6fdb0e1a1597fbaf785ebe",
+          "teacher_sha256": "f62991383b3ef5e8db0ed71442b17c428972e58da1c868c6aeb0f3fa143c865b",
+          "token_sha256": "7f008ec99068f373c6e15dd059ee977fb70e6dc832f46e009f0f299d48d19892",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0094",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.01980333069056471,
+          "including_prefill_mean_kld": 0.019910660629571305,
+          "prefill_row_kld": 0.2395077158370706,
+          "raw_sha256": "387038afed9bd884c7430caed10c37159f96b475f782f74c42f39d161289c416",
+          "score_sha256": "4a29ea890ee1c5fb5ef4b38af4faf3a2d0ccbf4b9e039cd4e2fae795ab9adad0",
+          "teacher_sha256": "656911b8fb34d7f42e984dd21311caf9ef56db0b7618b414cad4089562d93cba",
+          "token_sha256": "6b6a9b21c49a9984cc8c5ea90e0bda6db5761d039c7cdd335dc65e30444e017b",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0083",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.029895885238883084,
+          "including_prefill_mean_kld": 0.03010492019686818,
+          "prefill_row_kld": 0.4577904442343803,
+          "raw_sha256": "caeca2ce6c7e3883a51eb1fe7af4f3f945fac4122c763b7ff6770fa3e8fc3532",
+          "score_sha256": "cf6e4f767052c8393f8f7f422237981e13589db63db732ef56478b955b0cfec2",
+          "teacher_sha256": "66a7870a1f13c8b5d647c1681fca005bb0f55a6e502e94f282da673a6bed3f1b",
+          "token_sha256": "3df9d2872815e061ff8d32a9a19da0d11d80ac67070cccd28579b24ec3079552",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0021",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04397075315325536,
+          "including_prefill_mean_kld": 0.044495762997015596,
+          "prefill_row_kld": 1.1186659033304427,
+          "raw_sha256": "ff68e13dedc4ed15d16f1e3b7651f3b030cd406b4e0840480b33dc2fa2dbdfdb",
+          "score_sha256": "677465d5f820174bf8485fb26bc023cc12882f11a493e62ca2ce0b114ac7bcef",
+          "teacher_sha256": "ebf76910c4db8f6955ad03f2ec7352e2f8aaf15e0c939d28f9a7d78f6bd7451b",
+          "token_sha256": "dbd9f2539cb3e97127ae760a04ca5b3456b0b13d6ce8698f88b215cc68b8f1a9",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0098",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.05041230633987613,
+          "including_prefill_mean_kld": 0.051236972335366136,
+          "prefill_row_kld": 1.738503599107944,
+          "raw_sha256": "4b290f9027a497d7af5bc5c06174dae87e2723fb95b93f3d2e02ffcf355d5234",
+          "score_sha256": "1e3fcd31518d465b473c11fbed05d857b572785311b32c88e7fa6ba9a7e75a18",
+          "teacher_sha256": "16e3d9524100836c020535635f7722a4c93ce0fdddc2ad0781f25bda77e8bcc9",
+          "token_sha256": "749bda3291a0323d286b0018b7dc3e0f469de8afd5d194370cde357b7b315c7d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0080",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.047528968125352955,
+          "including_prefill_mean_kld": 0.04910846403209107,
+          "prefill_row_kld": 3.2807570892182745,
+          "raw_sha256": "68f5c356fbf444f5d0022938812e368f1ba97c6cc2f7541072c1609923160030",
+          "score_sha256": "87dda561523fd7657fe08d596e9f14bc21a47d1cba5f5f425e392b4ea4d436dc",
+          "teacher_sha256": "e9c1a77490a5db63d410297bd8febe06adf4df2a6f4e16552db187738c9289d2",
+          "token_sha256": "a31d43f38710eb305c684501a2b02909a4e82c43101c07e76e4a3ca4edb4d0dd",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0081",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.025434826960204975,
+          "including_prefill_mean_kld": 0.026087226927511527,
+          "prefill_row_kld": 1.360897560036721,
+          "raw_sha256": "55a8ed57d62691e0e1252f3f244d14fe16a7382025f276fd851612b4b5d71ffc",
+          "score_sha256": "b3a51927b33af5a34e3a67c19f25e8cb70e335a25e09b270ea8916213a430f55",
+          "teacher_sha256": "5ae07760d91030d1eba925b76ea463309baa49c2e7d5f5e14409a23ced7fd20d",
+          "token_sha256": "350628488b94d59f3e4f3e99a418913544ea1fe7eacee14a642b996d9c614d16",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0123",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.03625410701112817,
+          "including_prefill_mean_kld": 0.03623887373163693,
+          "prefill_row_kld": 0.0050715838925684655,
+          "raw_sha256": "9b464ec3a977d1ee3cebc601ffb2a2169d0cf3a9ce70673ce1ea96d807947b12",
+          "score_sha256": "47f8a43f8c66147cabaedafeea53145f78935ed971f3cc45d5ec7f8870970954",
+          "teacher_sha256": "4a24b5b750ba98c06d9bc69e19334aff89e7bd4f9429592598af5d62de7ecb42",
+          "token_sha256": "aab84ee426f91621d123aa68bd53f75e220fbaf8cb270db1feab1529a62ddbf0",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0090",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04104338344204917,
+          "including_prefill_mean_kld": 0.04195193367340904,
+          "prefill_row_kld": 1.9008457070357228,
+          "raw_sha256": "890278e84ba9ed1f86276a9702da857639ec6542257c80a8c84ccfb5c9485f63",
+          "score_sha256": "5e6e6d5b823b9f005abb50e738519564f2fc407837530abdf906b7cad8348e39",
+          "teacher_sha256": "db2a5fe33c9e457a22ebeb31a7073c1dac666b877503e9b4e90a8d6abf7b8a32",
+          "token_sha256": "a06e76993db717afee847fcc4d86f10b52dd23cdce5dae916f32d238d08d886c",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0062",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02918700579461782,
+          "including_prefill_mean_kld": 0.030030454221016484,
+          "prefill_row_kld": 1.7557259346326815,
+          "raw_sha256": "4ee3b4a4a812c85274c3c836dcd86d0b123e38e4e33bbab437385e1955636035",
+          "score_sha256": "6a54c1c1ebae3e5b81823df2342e1d41c4b77ee7c9bbf26d205163b937fcca75",
+          "teacher_sha256": "d6f7883a533625c7c75119fe7403b48bf81885b1de0b311a1a0571949f7f231c",
+          "token_sha256": "7c6d2ef2f4b7c7091a517f6ccd778009f9ee144165c9aecf2f8f28aea7b3753f",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0047",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.025365434304478303,
+          "including_prefill_mean_kld": 0.03031257573187498,
+          "prefill_row_kld": 10.152163936185477,
+          "raw_sha256": "dd5c4505db35935c58c9f74bb6e74a6edd3c19374f835664d81a6e8de5b33eb9",
+          "score_sha256": "d6e3feb817873e71e0dd76cdfa4a8f638ca2b3b8ee42264f726be54b4d8f0845",
+          "teacher_sha256": "8513cdcb9940786c7c42bf14d2f5f2cd452791e4ec6036c06a1f973eb969018d",
+          "token_sha256": "89730ee6826d4302c352156d5df766c51622a8b6681b8052d4ca6e7d481177fe",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0051",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.012924581711263027,
+          "including_prefill_mean_kld": 0.013141907486799479,
+          "prefill_row_kld": 0.4577904442343803,
+          "raw_sha256": "d3d0a1b6cbfdcecf0842b87193c339608d8d51fb39007b7db0ec03f073db51a7",
+          "score_sha256": "c590acb2bb149f97ecf9be58ea5884de08396d26e18935a548f2d11c56fd8ce8",
+          "teacher_sha256": "177477c19b54dc0b0230e19202d48ab71d52f2b1a37fe383511bb64f0aae2fcf",
+          "token_sha256": "c79269b16560e7301549593443c9394007e7a957ad149c46224e7ad271dc6a18",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0055",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.023650142194727084,
+          "including_prefill_mean_kld": 0.02412364237713099,
+          "prefill_row_kld": 0.9929050155755266,
+          "raw_sha256": "0ab63005f5783e2d377aa953c64abe217cc6d1957ba9d164a3daf62b4a4f3693",
+          "score_sha256": "c7ca6c112700a4f07e16ad2806d6f2b18483a929964476311de92a1594a43746",
+          "teacher_sha256": "7679c828eae4bf08f17598d044904693fc90c05b9d3f189dddeeb846100e43a7",
+          "token_sha256": "feec9ac508172dd275ec6452a03493acab5ffdff9bb57161317c77f24daf90f2",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0031",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02037624180208121,
+          "including_prefill_mean_kld": 0.02532582054872674,
+          "prefill_row_kld": 10.152163936185477,
+          "raw_sha256": "0e31a3893a84a0ec7e1fdb6fb71c2a14d4fb2175229b0019bc57e59728d0380a",
+          "score_sha256": "74ac2939aafb8dbe1f1654aacbb4abd0b3b295da5ce29ee0788188acb38dc105",
+          "teacher_sha256": "e04a6c04a951efb5a178927e91457535cc67745ce5c81e785cb6d963aac6ce83",
+          "token_sha256": "2cff3482b4212537d0c35b88b8a0f479502c8905952430c973d25b1ab0bb5818",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0003",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.02690073030649202,
+          "including_prefill_mean_kld": 0.029946398068044746,
+          "prefill_row_kld": 6.2613826382049185,
+          "raw_sha256": "bc400c83ee2139f177dabf51511a8683e1a73e03f57fc40554c77f563c7da8e0",
+          "score_sha256": "dbfa39c66bc06fa999aaac571e8377131c13782d16e47b119caf725f1c7d28f3",
+          "teacher_sha256": "7677c89cc3dee8e2a64e3c7947add39475f828921a3a8fe2e92b379b3ed8320b",
+          "token_sha256": "d416537c6e8273747bf025e97287a0bb6e3ff047bfee65cf54e4dfe287569b93",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0009",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.04604780270271153,
+          "including_prefill_mean_kld": 0.046153134975366834,
+          "prefill_row_kld": 0.26166296482811324,
+          "raw_sha256": "b425869ce07538ae045c4d809021ae9564be0171c2776a9e7ab6465ffe11c5d7",
+          "score_sha256": "7ce0191e8d42f8aa8b9a45d584ac950a90b92023e131bb2dc60ffc35ee081583",
+          "teacher_sha256": "00701ad8bf4a4a5eedcedd4eafd956830c9ff1c790b15074ca721a3462abf930",
+          "token_sha256": "fc1166707f5938a914faf364dab717b0de3ea0e8451ed1a3fb399469e659b105",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0010",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.01563325095084946,
+          "including_prefill_mean_kld": 0.015634848716246288,
+          "prefill_row_kld": 0.018903876718152742,
+          "raw_sha256": "0020ca7d06c49315301acd377253c6fa721a5e8db71c88efaa33eb4ce96ee31b",
+          "score_sha256": "53fa079c5a28e23de66c8f16e72d56125b2721d5943a1ea387b4d2ac9ffaa5cf",
+          "teacher_sha256": "f9bf4519890b70d972dfdf9760181418eb7c55b6237c0108f01d44df89943278",
+          "token_sha256": "ddd9d769a76eba74bc8e680ff25736a3b9146371801d1d2b29e4dc48f6d256a2",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0011",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.018165048102336066,
+          "including_prefill_mean_kld": 0.023115707060852503,
+          "prefill_row_kld": 10.152163936185477,
+          "raw_sha256": "e5460f80bb681f0001beea1baead1660b76ba8565ec4aa06cbbb13e7f59b256f",
+          "score_sha256": "1870f3a7be94e65bf76df49d9fdf3690cd4c8563146acbc0186efc9e7c6c7cce",
+          "teacher_sha256": "9f37c4ead38bb80a5d85f825b156e5b493b6d3f1ca9dabfad06b0416f37db286",
+          "token_sha256": "c4b96acfa47944be3485b20eab76eb16d59913c9fedb5001a537335a3ee13614",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0030",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.024984801688836707,
+          "including_prefill_mean_kld": 0.024978325612706537,
+          "prefill_row_kld": 0.011728273850378172,
+          "raw_sha256": "9b7a816b646bd6c974b15105d1f7f92d162d594ca262d7c79fa1bcf1afdb980a",
+          "score_sha256": "fd35a10287015d380956b9985ce62b7df231d250fc9dbf2f923d3e974a7d29f6",
+          "teacher_sha256": "90c08d120cfac926855e15c106dba1e242099bcd0ae6da7655bfaa54d5ce2420",
+          "token_sha256": "d644a79da2ba01cefc5f2020eab9e8662d45ba3025377844e4bf4d3f1d36802d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0032",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.021718153455878218,
+          "including_prefill_mean_kld": 0.022319411629929006,
+          "prefill_row_kld": 1.2524936357378378,
+          "raw_sha256": "b364f38f47ad85d8efe4e8426747acb301a9e63117501450139127fc75039721",
+          "score_sha256": "fb18bbe03c546a46151b364d26cf895f7991b8f30fe53489b9447d7cc7e0fa7c",
+          "teacher_sha256": "1aa70ccf8e224c3a58834e2232a3a2477768db29ffeca39f5b27c17ad8c40c2b",
+          "token_sha256": "88745c333ead140a6fb05929031613429cbd42e25881d92add968a94a5d2d930",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0035",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.032422367247282986,
+          "including_prefill_mean_kld": 0.03463729855715423,
+          "prefill_row_kld": 4.566386758553708,
+          "raw_sha256": "a4c51cbc1ac997dd9c3bb9215dadeca429e88747a33aa61d37445974baac480c",
+          "score_sha256": "37eefbf56891e3d8af693c6f8685bbd172337a6f8bf081849e805e78b3148866",
+          "teacher_sha256": "4e958dce74e093eb62ba5ac210fe0a4e50f12f1875de678769700cde206b429d",
+          "token_sha256": "49f702136da39db9e762b6815a1952c0eba0a1fb04af22fb28a27bb63e2a41dc",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0040",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.05400247271460698,
+          "including_prefill_mean_kld": 0.0549407898731226,
+          "prefill_row_kld": 1.9747376961960739,
+          "raw_sha256": "fd04d2487f3904f117be851809799cc8c320f950d589b3c328db83f9cda9dfcd",
+          "score_sha256": "c79be837a60e6ba89adb4b80483b1b10d49aeac719841479ca4eb8980618e9b7",
+          "teacher_sha256": "f4c96798fc9eb5564d9aec52779cc0ce88bc55312dcd7663240938d711c8a90d",
+          "token_sha256": "50f3bc08ab37c551a05364eb48f0d726742f19331e79114dd5a9fb3c42ce6d4f",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0041",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.08657045244168773,
+          "including_prefill_mean_kld": 0.08720775297518737,
+          "prefill_row_kld": 1.3911246445154333,
+          "raw_sha256": "8bbe8f464764b145ae367b5ffdbafd78c30bacf85f8bc90f22bbe2db6e7da597",
+          "score_sha256": "b571a88deab449256a1c881ef8550aad88b9ea274c98f6182267c11da48e79bf",
+          "teacher_sha256": "c2bbe4ea31532baceff966423c6ce6e042aebb5a2eb1de45e382a7d3df6f6827",
+          "token_sha256": "8acf2614f0f787c185af112f21bcdf247c9b6ae76d292ada28f476e2a7b0241d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0046",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.06463482692444868,
+          "including_prefill_mean_kld": 0.0648417166461593,
+          "prefill_row_kld": 0.48813808726609126,
+          "raw_sha256": "daa52d49b4550a1fd0bdf38c70a737b613369f485bd5051fafae07cb8ce6a5eb",
+          "score_sha256": "b25aa33f209bc0e72888e76efe9feef250e2c106ca34142f4c0dbd353a355532",
+          "teacher_sha256": "3caa5c58fb0b0e80129d551b39867fd52fefd93830060bb5655862fdd0e3b8ce",
+          "token_sha256": "3f2df1c090abf4473163a078c168fec33ab98ed6415985b92449caa861c04444",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0060",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.015704907348830933,
+          "including_prefill_mean_kld": 0.01629535873627198,
+          "prefill_row_kld": 1.2243588974406492,
+          "raw_sha256": "e4067368b626a98b028811ad6772aeaeadd85f04dcdeb9f91dab6658078b3a74",
+          "score_sha256": "bab86bd14f05d787511ea71f9bdf76096edf975451528afd499e35b3701a5454",
+          "teacher_sha256": "d9e6344664b92527ee58d3bca038702676b357e7052ade03ac4bc9c70908cc34",
+          "token_sha256": "04ffe38aafcd9407b2846d59e8d797bee04b97f00683f8bc74d475a2d6b8592b",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0063",
+          "domain": "axis4_reasoning_termination",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.03132559147375623,
+          "including_prefill_mean_kld": 0.031814505567999064,
+          "prefill_row_kld": 1.0321327423888638,
+          "raw_sha256": "614a33a2276349c6d3ef14734996f4579e50cc22ef755b409471361e7b24699d",
+          "score_sha256": "17cfb59297e5f31cfa88eae36930a85afbfd41d4edfdbd51e9c612bfe75af43f",
+          "teacher_sha256": "a9bb59659437c1df5b3f3ef0b54eda61402842a2597b76b0124f58b59ecf3ab4",
+          "token_sha256": "38003e2d26db850dd8ff39f2e95fa3d1be9a34838c70d9f7e87056b7ef50b67d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0074",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.08556837423676537,
+          "including_prefill_mean_kld": 0.0867002508399446,
+          "prefill_row_kld": 2.4025197809446555,
+          "raw_sha256": "9abd2d8047eef0aa95de284928fde34187426078a2759b214f339b67fafea057",
+          "score_sha256": "02b7030207fb7736d57c060fbc8208a57254ef4f66b81b6954e3b8b1cb2ca972",
+          "teacher_sha256": "440341730d063ae27bbc15bcbfc879d9d16b3b25691d97b8cfd60be8f32f2313",
+          "token_sha256": "6f6618094ffce3f44b50f94cc533d338a3bfbbda7432890cd5ad2c13efd50129",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0099",
+          "domain": "axis3_code_agentic",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.01659116406314401,
+          "including_prefill_mean_kld": 0.016611742343204117,
+          "prefill_row_kld": 0.058714903346186786,
+          "raw_sha256": "5cb40de2fe89feb4655bd5ab724266f6376e2d8559eb6f14838fff92e0f57eda",
+          "score_sha256": "c52bf6a766f6db58d17972e8cdb67aaec0d255c27eee59affc117e2bf110f423",
+          "teacher_sha256": "da8dea840dd57564f0b3675872eaf297631b38a6d8dabcde7f8b54db06623e9a",
+          "token_sha256": "07e37956eb606026bdd7479ac5516275e92c0ee834c3c0439a3f300f9099608d",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0113",
+          "domain": "axis2_legal",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.0561128193667057,
+          "including_prefill_mean_kld": 0.056764999056568295,
+          "prefill_row_kld": 1.3911246445154333,
+          "raw_sha256": "c232c29068dada31c94b8aba188f91fd007fd4bc2170cec51b51ada2659f0593",
+          "score_sha256": "19dcdfe7a722a3f784e204d6b7126be29a6fd82803522814a9f13efba2d99c8c",
+          "teacher_sha256": "6dc5f38f3436d6833b0ab68ce34e0f7ee5cde80359c26602526d8de512fd3207",
+          "token_sha256": "742ab50951388e712d20dab49b12ad7eb31490cfe5091a2fbc76f31ed8fa1b09",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        },
+        {
+          "window_id": "conditional-fit-0118",
+          "domain": "axis1_general",
+          "prediction_rows": 2047,
+          "true_decode_rows": 2046,
+          "true_decode_mean_kld": 0.013573318669100111,
+          "including_prefill_mean_kld": 0.01402729782126711,
+          "prefill_row_kld": 0.9428686431549456,
+          "raw_sha256": "1dd59d9376e947ed299c24653b27953232374ae9f1b124692f8dfb9534afdb85",
+          "score_sha256": "4f2f5d0ac0658964a5bc7efa3ddb34089a34adc96e31a3e17a428179686992d0",
+          "teacher_sha256": "e296287593aa0dc89b695a6a082a224a85904757c4a3cefa69729394d1469f89",
+          "token_sha256": "2e03fdfcf4d54fc3174ee4620d5efd9f2e5c10e2e6e0b1217577f08392a09749",
+          "raw_retirement": "after durable score and verified raw hash",
+          "raw_bytes": 1268157440
+        }
+      ],
+      "domains": {
+        "axis1_general": 0.030449786890764326,
+        "axis2_legal": 0.05576342020065474,
+        "axis3_code_agentic": 0.031720420854409875,
+        "axis4_reasoning_termination": 0.023891267142802115
+      },
+      "correctness_profile_engine_capacity_tokens": [
+        31557971
+      ]
+    }
+  },
+  "fp8_minus_nvfp4": -0.003511050605611574,
+  "paired_window_bca95": [
+    -0.008171877159734635,
+    -0.001367594090752266
+  ],
+  "fp8_lower_windows": 22,
+  "true_decode_rows_per_window": 2046,
+  "windows": 32,
+  "historical_dcp1_context_only": 0.03418114591027796,
+  "limits": [
+    "DCP4 new versus DCP1 historical; different runtime image.",
+    "MTP off and maxseq1 forced decode; not MTP quality or throughput evidence.",
+    "Fixed FP8 then NVFP4 order; one server preparation per arm, window intervals do not capture server-run variability."
+  ],
+  "receipt_audit": [
+    {
+      "arm": "fp8",
+      "windows": 32,
+      "prediction_rows": 65504,
+      "true_decode_rows": 65472,
+      "all_receipt_hashes_and_masks_pass": true,
+      "all_prefix_hit_counters_zero": true,
+      "raw_logits_retired_after_scoring": true,
+      "source_runtime_audit_sha256": "6a3393053b83b33b41a7d49de2fab1803b1efc4843ec5862c1acd8467b993fa4",
+      "source_startup_sha256": "00ce74cfce281b627289f3322829d357467f5bc1f026677ef27cda93ed43fce3"
+    },
+    {
+      "arm": "nvfp4",
+      "windows": 32,
+      "prediction_rows": 65504,
+      "true_decode_rows": 65472,
+      "all_receipt_hashes_and_masks_pass": true,
+      "all_prefix_hit_counters_zero": true,
+      "raw_logits_retired_after_scoring": true,
+      "source_runtime_audit_sha256": "74d7d1914a55a73b5e9de196f557c658603a9501acd2fafb1e540d496b1c7185",
+      "source_startup_sha256": "cae3cb749c61cc18fd23286f41f79266e32bfdc551e0575b3515d21eb993036a"
+    }
+  ],
+  "public_base_image": "verdictai/trellismx@sha256:ca6b80188dce154b91f49108b7d87792d2ba6328935afc71b44d1c0e6f6a1adf",
+  "capture_image_local_id": "sha256:0405a1c0dc128b51069798a5d00b346257bbd006c0deb7e6530a3d005d75de71",
+  "plan_seal": {
+    "algorithm": "sha256",
+    "created_at": "2026-09-09T20:32:50.484730+00:00",
+    "plan_sha256": "8bd19d67a09ddc6ef6bea1afe67306f1a6859e927cb31969730b51e5d13e24fe"
+  },
+  "metric_sha256": "752af6740595791f300cf47f15ef81f8ca85245d5282fc600ef26ee57eae41e6",
+  "role_manifest_sha256": "b5d7e4524eb98ddfbd230a5d9a44de0dc5dbeb796c03e859898b5838e4463d14",
+  "teacher_revision": "7c378d5f17dba158c4c803eff27c346dd0615660",
+  "teacher_model_revision": "a6c167b62691b2bac901344b65cb651a70f53e43",
+  "capture_cpu_test_sha256": "d470ec0385cb9164d789a3a04c5aa638464b656f1c6a0ac12fdc9c2f8a5f37bc",
+  "prior_attempts": []
+}

results/speed-20260909/README.md

@@ -0,0 +1,40 @@
+# September 9 TrellisMX FP8 speed experiments
+
+Selected runtime: **token-map-hoist reference**, local image identity `sha256:ca6b80188dce154b91f49108b7d87792d2ba6328935afc71b44d1c0e6f6a1adf`. TP4/DCP4, MTP3 probabilistic, CUDA graphs, NVFP4 MLA KV, 24 sequences, batch4096, four RTX PRO6000 GPUs at300W. Expert math remains E4M3 FP8 with UE8M0 scales. Checkpoint unchanged.
+
+All rates are tokens/sec; C4 is aggregate. Single exploratory server runs; MTP acceptance/output trajectories and sustained clocks differ. Later cooled screens require90seconds minimum idle and all GPUs<=55C continuously30seconds before each cell. Earlier runs retain their original protocols; do not silently treat them as cooled replications. No independent replication or superiority claim. No measurement reached300t/s C1 or20000t/s prefill.
+
+| Measurement | Selected reference | Task-count | Selective grid-floor |
+|---|---:|---:|---:|
+| 32K prefill | 8407.000 | 8439.000 | 8423.000 |
+| 64K prefill | 8407.000 | 8451.000 | 8448.000 |
+| 8K C1 decode | 222.100 | 212.098 | 214.588 |
+| 8K C4 decode aggregate | 318.560 | 325.083 | 326.620 |
+| 16K C1 decode | 216.636 | 208.708 | 217.293 |
+| 16K C4 decode aggregate | 319.157 | 353.532 | 343.859 |
+| 0K C1 decode | 204.611 | 195.372 | 185.422 |
+| 0K C4 decode aggregate | 327.670 | 333.729 | 325.106 |
+
+## Selected reference extended context
+
+| Measurement | Tokens/sec |
+|---|---:|
+| 32K prefill | 8457.000 |
+| 64K prefill | 8443.000 |
+| 128K prefill | 8323.000 |
+| 32K C1 decode | 204.930 |
+| 32K C4 decode aggregate | 329.343 |
+| 64K C1 decode | 199.446 |
+| 64K C4 decode aggregate | 330.096 |
+| 128K C1 decode | 204.445 |
+| 128K C4 decode aggregate | 334.086 |
+| 8K C8 decode aggregate | 594.639 |
+| 16K C8 decode aggregate | 602.965 |
+| 0K C1 decode | 204.611 |
+| 0K C4 decode aggregate | 327.670 |
+
+Task-count and grid-floor were not measured at C1 contexts32K/64K/128K. Their shorter-context results cannot rank long-context C1. Reference KV capacity is23,562,091 tokens in the expanded run; separate zero-context reference sessions reported23,568,627. Use the explicit engine KV metric, not the benchmark's generic block-times-DCP estimate.
+
+Reference is retained for balanced use: best observed C1 at0K/8K and effectively tied at16K. Task-count is a concurrent-throughput alternative, especially16K C4. Neither alternative has a meaningful prefill improvement established. Both alternatives passed172 exact representative K4/K5 component comparisons and FP8 compiled-opcode checks; these are not full-model quality evaluations.
+
+`benchmark-index.json` indexes every discovered llm_decode_bench result in today's campaign, including earlier/negative/profile diagnostics. Results under profiling paths are diagnostic, not serving throughput qualification. `evidence/` includes raw benchmark JSON and associated logs/receipts plus candidate summaries/failures. Local host/path prefixes are redacted; `public-file-inventory.json` records both original and public hashes. Historical raw receipt hashes refer to the originals, not the redacted copies. Raw local archives are retained separately. KLD is published separately with cache dtype and capture-image provenance.

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/decode-cap8192-command.json

@@ -0,0 +1,27 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "8192",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/batch16384-speed-window-01/results-01/batch16384/rep-1/decode-cap8192.json",
+  "--contexts",
+  "0,8k,32k",
+  "--concurrency",
+  "1,2,4",
+  "--skip-prefill",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/decode-cap8192-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "cd13e14d3d3f0b8b357a7d0c2f80a963c62ba2aa169ff9bbde9ad08d9d3cd255"
+}

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/decode-cap8192.json

@@ -0,0 +1,1394 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T07:18:01.203111",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "skipped",
+    "standalone_prefill": false,
+    "prefill_only": false,
+    "skip_prefill": true,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 32768,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 17031168,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0,
+      8192,
+      32768
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0,8k,32k",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": false,
+      "prefill_only": false,
+      "skip_prefill": true,
+      "prefill_contexts": "8k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 112,
+    "duration_seconds": 267.276,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 10.31,
+    "cpu_temp_max_c": 77.25,
+    "gpu_util_avg_pct": 91.56,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 39.56,
+    "mem_util_max_pct": 55.0,
+    "temp_avg_c": 67.18,
+    "temp_max_c": 84.0,
+    "power_total_avg_w": 1101.05,
+    "power_total_max_w": 1178.9,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 380174.0,
+    "vram_used_max_mb": 380174.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 97.1,
+    "vram_used_max_pct": 97.1,
+    "pcie_rx_avg_mb_s": 11487.79,
+    "pcie_rx_max_mb_s": 71064.0,
+    "pcie_tx_avg_mb_s": 11408.48,
+    "pcie_tx_max_mb_s": 66265.0
+  },
+  "event_log": [
+    "07:13:31 benchmark start engine=vllm",
+    "07:13:31 startup server=http://127.0.0.1:8001 model=glm53-flash-trellismx-p8-k45",
+    "07:13:31 startup decode concurrency=1,2,4 contexts=0,8k,32k",
+    "07:13:31 startup NVIDIA P2P override: enabled: runtime NVIDIA P2P override matches expected RegistryDwords",
+    "07:13:31 startup engine vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f models=['glm53-flash-trellismx-p8-k45']",
+    "07:13:31 startup KV cache budget from vLLM metrics: 17,031,168 tokens (2079 blocks x 2048; local 4,257,792 \u00d7 CP 4; CP source: local process)",
+    "07:13:31 startup model context length: 1,000,000 tokens",
+    "07:13:31 startup prefill tests: skipped",
+    "07:13:31 startup calibrating padding text run=wwbkdgkvesfu up_to=32k",
+    "07:13:31 startup context 8k: 50,544 chars (8,192 prompt tokens via /tokenize)",
+    "07:13:31 startup context 32k: 205,140 chars (32,768 prompt tokens via /tokenize)",
+    "07:13:31 startup token targeting: /tokenize exact",
+    "07:13:31 startup startup preparation done",
+    "07:13:31 hardware monitor interval=2s",
+    "07:13:31 decode warmup start",
+    "07:13:31 decode warmup start C=1 ctx=32k 3s",
+    "07:13:31 cell start C=1 ctx=32k",
+    "07:13:39 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "07:13:42 cell done C=1 ctx=32k 175.7 tok/s",
+    "07:13:42 decode warmup done C=1 ctx=32k",
+    "07:13:44 cell start C=1 ctx=0",
+    "07:13:50 ready C=1 ctx=0 running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "07:14:10 cell done C=1 ctx=0 167.2 tok/s",
+    "07:14:12 cell start C=1 ctx=8k",
+    "07:14:18 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "07:14:38 cell done C=1 ctx=8k 173.0 tok/s",
+    "07:14:40 cell start C=1 ctx=32k",
+    "07:14:49 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "07:15:09 cell done C=1 ctx=32k 186.4 tok/s",
+    "07:15:11 cell start C=2 ctx=0",
+    "07:15:17 ready C=2 ctx=0 running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "07:15:37 cell done C=2 ctx=0 244.6 tok/s",
+    "07:15:39 cell start C=4 ctx=0",
+    "07:15:44 ready C=4 ctx=0 running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "07:16:04 cell done C=4 ctx=0 292.9 tok/s",
+    "07:16:06 cell start C=2 ctx=8k",
+    "07:16:12 ready C=2 ctx=8k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "07:16:32 cell done C=2 ctx=8k 246.0 tok/s",
+    "07:16:34 cell start C=4 ctx=8k",
+    "07:16:41 ready C=4 ctx=8k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "07:17:01 cell done C=4 ctx=8k 298.4 tok/s",
+    "07:17:04 cell start C=2 ctx=32k",
+    "07:17:09 ready C=2 ctx=32k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "07:17:29 cell done C=2 ctx=32k 252.6 tok/s",
+    "07:17:31 cell start C=4 ctx=32k",
+    "07:17:39 ready C=4 ctx=32k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "07:17:59 cell done C=4 ctx=32k 297.7 tok/s"
+  ],
+  "prefill": {},
+  "results": [
+    {
+      "concurrency": 1,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.988585,
+      "measurement_wall_seconds": 20.00075,
+      "client_output_tokens": 3342,
+      "server_output_tokens": 3342,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 167.19542416324822,
+      "per_request_avg_tps": 167.19542416324822,
+      "ttft_avg": 0.07083372515626252,
+      "ttft_p50": 0.07083372515626252,
+      "ttft_p90": 0.07083372515626252,
+      "ttft_p99": 0.07083372515626252,
+      "time_to_second_token_avg": 0.012936464976519346,
+      "time_to_second_token_p50": 0.012936464976519346,
+      "time_to_second_token_p90": 0.012936464976519346,
+      "time_to_second_token_p99": 0.012936464976519346,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005911081440074571,
+      "inter_token_latency_p50": 0.005911081440074571,
+      "inter_token_latency_p90": 0.005911081440074571,
+      "inter_token_latency_p99": 0.005911081440074571,
+      "output_tps_per_user_avg": 169.17378150475705,
+      "output_tps_per_user_p50": 169.17378150475705,
+      "output_tps_per_user_p90": 169.17378150475705,
+      "output_tps_per_user_p99": 169.17378150475705,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.01407805126159353,
+      "chunk_inter_token_latency_p50": 0.01407805126159353,
+      "chunk_inter_token_latency_p90": 0.01407805126159353,
+      "chunk_inter_token_latency_p99": 0.01407805126159353,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 4307.0,
+      "output_seq_len_p50": 4307.0,
+      "output_seq_len_p90": 4307.0,
+      "output_seq_len_p99": 4307.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.07083372515626252,
+          "time_to_second_token": 0.012936464976519346,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005911081440074571,
+          "chunk_inter_token_latency_avg": 0.01407805126159353,
+          "input_tokens": 78,
+          "output_tokens": 4307,
+          "output_tps_per_user": 169.17378150475705,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3342,
+      "wall_time": 25.542422840138897,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 167.0490340328882,
+      "server_utilization": 0.0101058710298364,
+      "server_spec_accept_rate": 0.49765258215962443,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.535,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.305,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.88,
+        "cpu_temp_max_c": 76.12,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 48.47,
+        "mem_util_max_pct": 53.0,
+        "temp_avg_c": 65.39,
+        "temp_max_c": 80.0,
+        "power_total_avg_w": 1150.89,
+        "power_total_max_w": 1153.46,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 8406.89,
+        "pcie_rx_max_mb_s": 8711.0,
+        "pcie_tx_avg_mb_s": 8352.33,
+        "pcie_tx_max_mb_s": 8558.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.99916,
+      "measurement_wall_seconds": 20.000275,
+      "client_output_tokens": 3459,
+      "server_output_tokens": 3459,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 172.95726809626717,
+      "per_request_avg_tps": 172.95726809626717,
+      "ttft_avg": 0.5792566961608827,
+      "ttft_p50": 0.5792566961608827,
+      "ttft_p90": 0.5792566961608827,
+      "ttft_p99": 0.5792566961608827,
+      "time_to_second_token_avg": 0.0174833619967103,
+      "time_to_second_token_p50": 0.0174833619967103,
+      "time_to_second_token_p90": 0.0174833619967103,
+      "time_to_second_token_p99": 0.0174833619967103,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005649759039614473,
+      "inter_token_latency_p50": 0.005649759039614473,
+      "inter_token_latency_p90": 0.005649759039614473,
+      "inter_token_latency_p99": 0.005649759039614473,
+      "output_tps_per_user_avg": 176.9986990574801,
+      "output_tps_per_user_p50": 176.9986990574801,
+      "output_tps_per_user_p90": 176.9986990574801,
+      "output_tps_per_user_p99": 176.9986990574801,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.014258915671407954,
+      "chunk_inter_token_latency_p50": 0.014258915671407954,
+      "chunk_inter_token_latency_p90": 0.014258915671407954,
+      "chunk_inter_token_latency_p99": 0.014258915671407954,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 4241.0,
+      "output_seq_len_p50": 4241.0,
+      "output_seq_len_p90": 4241.0,
+      "output_seq_len_p99": 4241.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5792566961608827,
+          "time_to_second_token": 0.0174833619967103,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005649759039614473,
+          "chunk_inter_token_latency_avg": 0.014258915671407954,
+          "input_tokens": 8192,
+          "output_tokens": 4241,
+          "output_tps_per_user": 176.9986990574801,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3459,
+      "wall_time": 26.074295089114457,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 172.904696376223,
+      "server_utilization": 0.010587102983638075,
+      "server_spec_accept_rate": 0.539906103286385,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 6.06,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.905,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.94,
+        "cpu_temp_max_c": 75.25,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 49.0,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 66.22,
+        "temp_max_c": 81.0,
+        "power_total_avg_w": 1153.53,
+        "power_total_max_w": 1154.66,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 8274.25,
+        "pcie_rx_max_mb_s": 8601.0,
+        "pcie_tx_avg_mb_s": 8292.38,
+        "pcie_tx_max_mb_s": 8476.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.999218,
+      "measurement_wall_seconds": 20.000322,
+      "client_output_tokens": 3727,
+      "server_output_tokens": 3727,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 186.35728993398365,
+      "per_request_avg_tps": 186.35728993398365,
+      "ttft_avg": 0.5975865018554032,
+      "ttft_p50": 0.5975865018554032,
+      "ttft_p90": 0.5975865018554032,
+      "ttft_p99": 0.5975865018554032,
+      "time_to_second_token_avg": 0.011153812054544687,
+      "time_to_second_token_p50": 0.011153812054544687,
+      "time_to_second_token_p90": 0.011153812054544687,
+      "time_to_second_token_p99": 0.011153812054544687,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005334254001541856,
+      "inter_token_latency_p50": 0.005334254001541856,
+      "inter_token_latency_p90": 0.005334254001541856,
+      "inter_token_latency_p99": 0.005334254001541856,
+      "output_tps_per_user_avg": 187.4676383447342,
+      "output_tps_per_user_p50": 187.4676383447342,
+      "output_tps_per_user_p90": 187.4676383447342,
+      "output_tps_per_user_p99": 187.4676383447342,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.014272389806152839,
+      "chunk_inter_token_latency_p50": 0.014272389806152839,
+      "chunk_inter_token_latency_p90": 0.014272389806152839,
+      "chunk_inter_token_latency_p99": 0.014272389806152839,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 4488.0,
+      "output_seq_len_p50": 4488.0,
+      "output_seq_len_p90": 4488.0,
+      "output_seq_len_p99": 4488.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5975865018554032,
+          "time_to_second_token": 0.011153812054544687,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005334254001541856,
+          "chunk_inter_token_latency_avg": 0.014272389806152839,
+          "input_tokens": 32768,
+          "output_tokens": 4488,
+          "output_tps_per_user": 187.4676383447342,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3727,
+      "wall_time": 29.12444451614283,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 186.30161122127362,
+      "server_utilization": 0.012030798845043322,
+      "server_spec_accept_rate": 0.4225352112676056,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 9.11,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.884,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.89,
+        "cpu_temp_max_c": 76.12,
+        "gpu_util_avg_pct": 99.03,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 48.38,
+        "mem_util_max_pct": 54.0,
+        "temp_avg_c": 67.03,
+        "temp_max_c": 82.0,
+        "power_total_avg_w": 1154.29,
+        "power_total_max_w": 1155.42,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 8367.62,
+        "pcie_rx_max_mb_s": 8719.0,
+        "pcie_tx_avg_mb_s": 8179.25,
+        "pcie_tx_max_mb_s": 8438.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.996612,
+      "measurement_wall_seconds": 20.000729,
+      "client_output_tokens": 4892,
+      "server_output_tokens": 4892,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 244.64143701284553,
+      "per_request_avg_tps": 122.32071850642276,
+      "ttft_avg": 0.11364343203604221,
+      "ttft_p50": 0.11364343203604221,
+      "ttft_p90": 0.14825461693108083,
+      "ttft_p99": 0.1560421335324645,
+      "time_to_second_token_avg": 0.015344579587690532,
+      "time_to_second_token_p50": 0.015344579587690532,
+      "time_to_second_token_p90": 0.016974025568924845,
+      "time_to_second_token_p99": 0.017340650914702563,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.008021675154484334,
+      "inter_token_latency_p50": 0.008021675154484334,
+      "inter_token_latency_p90": 0.008142962393239039,
+      "inter_token_latency_p99": 0.008170252021958846,
+      "output_tps_per_user_avg": 124.70678698569867,
+      "output_tps_per_user_p50": 124.70678698569867,
+      "output_tps_per_user_p90": 126.59234599378327,
+      "output_tps_per_user_p99": 127.0165967706023,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.020790530252109002,
+      "chunk_inter_token_latency_p50": 0.020790530252109002,
+      "chunk_inter_token_latency_p90": 0.02081209152359912,
+      "chunk_inter_token_latency_p99": 0.020816942809684397,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 3170.5,
+      "output_seq_len_p50": 3170.5,
+      "output_seq_len_p90": 3214.1,
+      "output_seq_len_p99": 3223.91,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.07037945091724396,
+          "time_to_second_token": 0.013307772111147642,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008173284202927714,
+          "chunk_inter_token_latency_avg": 0.02081748184147165,
+          "input_tokens": 78,
+          "output_tokens": 3116,
+          "output_tps_per_user": 122.34983822559292,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.15690741315484047,
+          "time_to_second_token": 0.017381387064233422,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007870066106040956,
+          "chunk_inter_token_latency_avg": 0.02076357866274635,
+          "input_tokens": 78,
+          "output_tokens": 3225,
+          "output_tps_per_user": 127.06373574580442,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4892,
+      "wall_time": 25.551810663193464,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 244.53218903662054,
+      "server_utilization": 0.0202117420596728,
+      "server_spec_accept_rate": 0.5034013605442177,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.534,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.285,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.84,
+        "cpu_temp_max_c": 76.12,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 45.14,
+        "mem_util_max_pct": 49.0,
+        "temp_avg_c": 67.78,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1176.88,
+        "power_total_max_w": 1178.9,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 11281.22,
+        "pcie_rx_max_mb_s": 11385.0,
+        "pcie_tx_avg_mb_s": 11144.0,
+        "pcie_tx_max_mb_s": 11241.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.968882,
+      "measurement_wall_seconds": 20.001101,
+      "client_output_tokens": 5848,
+      "server_output_tokens": 5848,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 292.85565306969767,
+      "per_request_avg_tps": 73.21391326742442,
+      "ttft_avg": 0.1554050333215855,
+      "ttft_p50": 0.18278046406339854,
+      "ttft_p90": 0.18279754216782748,
+      "ttft_p99": 0.18280153354629874,
+      "time_to_second_token_avg": 0.026175830571446568,
+      "time_to_second_token_p50": 0.030380802578292787,
+      "time_to_second_token_p90": 0.030411765072494747,
+      "time_to_second_token_p99": 0.030420526592060924,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.013267129396052025,
+      "inter_token_latency_p50": 0.013429554256049785,
+      "inter_token_latency_p90": 0.013534146595532138,
+      "inter_token_latency_p99": 0.013559169974355398,
+      "output_tps_per_user_avg": 75.43238689615268,
+      "output_tps_per_user_p50": 74.46328652304663,
+      "output_tps_per_user_p90": 77.75213899442375,
+      "output_tps_per_user_p99": 78.93575450758087,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.034184321112551014,
+      "chunk_inter_token_latency_p50": 0.03419339492657416,
+      "chunk_inter_token_latency_p90": 0.034255501869857555,
+      "chunk_inter_token_latency_p99": 0.03427054421702008,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 1913.0,
+      "output_seq_len_p50": 1890.5,
+      "output_seq_len_p90": 1969.7,
+      "output_seq_len_p99": 1999.67,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.0732572281267494,
+          "time_to_second_token": 0.013520217034965754,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013469271168953313,
+          "chunk_inter_token_latency_avg": 0.03427221558892703,
+          "input_tokens": 78,
+          "output_tokens": 1889,
+          "output_tps_per_user": 74.24306686355838,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18280197703279555,
+          "time_to_second_token": 0.030421500094234943,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012647458722328322,
+          "chunk_inter_token_latency_avg": 0.034216503192028784,
+          "input_tokens": 78,
+          "output_tokens": 2003,
+          "output_tps_per_user": 79.06726734237611,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18278719414956868,
+          "time_to_second_token": 0.030389050021767616,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.01338983734314626,
+          "chunk_inter_token_latency_avg": 0.034170286661119535,
+          "input_tokens": 78,
+          "output_tokens": 1892,
+          "output_tps_per_user": 74.68350618253487,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.1827737339772284,
+          "time_to_second_token": 0.030372555134817958,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013561950349780205,
+          "chunk_inter_token_latency_avg": 0.03407827900812872,
+          "input_tokens": 78,
+          "output_tokens": 1868,
+          "output_tps_per_user": 73.73570719614136,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5848,
+      "wall_time": 25.541023290948942,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 292.3212847040937,
+      "server_utilization": 0.04042348411934549,
+      "server_spec_accept_rate": 0.49712643678160917,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.535,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.833,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.81,
+        "cpu_temp_max_c": 77.0,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 39.75,
+        "mem_util_max_pct": 43.0,
+        "temp_avg_c": 68.5,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1172.95,
+        "power_total_max_w": 1173.93,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 7881.25,
+        "pcie_rx_max_mb_s": 8056.0,
+        "pcie_tx_avg_mb_s": 7872.12,
+        "pcie_tx_max_mb_s": 8191.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.997371,
+      "measurement_wall_seconds": 20.000506,
+      "client_output_tokens": 4919,
+      "server_output_tokens": 4919,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 245.98233342981337,
+      "per_request_avg_tps": 122.99116671490668,
+      "ttft_avg": 0.9427531745750457,
+      "ttft_p50": 0.9427531745750457,
+      "ttft_p90": 1.225604160549119,
+      "ttft_p99": 1.2892456323932855,
+      "time_to_second_token_avg": 0.022013463429175317,
+      "time_to_second_token_p50": 0.022013463429175317,
+      "time_to_second_token_p90": 0.025811466271989048,
+      "time_to_second_token_p99": 0.026666016911622136,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.008114433301139572,
+      "inter_token_latency_p50": 0.008114433301139572,
+      "inter_token_latency_p90": 0.00823075733708024,
+      "inter_token_latency_p99": 0.008256930245166891,
+      "output_tps_per_user_avg": 123.27677949684329,
+      "output_tps_per_user_p50": 123.27677949684329,
+      "output_tps_per_user_p90": 125.04400734833438,
+      "output_tps_per_user_p99": 125.44163361491988,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.021151300976615273,
+      "chunk_inter_token_latency_p50": 0.021151300976615273,
+      "chunk_inter_token_latency_p90": 0.021420214613202804,
+      "chunk_inter_token_latency_p99": 0.021480720181434997,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 2907.5,
+      "output_seq_len_p50": 2907.5,
+      "output_seq_len_p90": 2914.3,
+      "output_seq_len_p99": 2915.83,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5891894421074539,
+          "time_to_second_token": 0.017265959875658154,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008259838346065407,
+          "chunk_inter_token_latency_avg": 0.021487443022349687,
+          "input_tokens": 8192,
+          "output_tokens": 2899,
+          "output_tps_per_user": 121.06774468247944,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.2963169070426375,
+          "time_to_second_token": 0.02676096698269248,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007969028256213737,
+          "chunk_inter_token_latency_avg": 0.020815158930880862,
+          "input_tokens": 8192,
+          "output_tokens": 2916,
+          "output_tps_per_user": 125.48581431120715,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4919,
+      "wall_time": 25.566069599008188,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 245.8762565579269,
+      "server_utilization": 0.02117420596727626,
+      "server_spec_accept_rate": 0.5510204081632653,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.547,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.882,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.89,
+        "cpu_temp_max_c": 75.62,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 45.12,
+        "mem_util_max_pct": 49.0,
+        "temp_avg_c": 68.28,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1178.0,
+        "power_total_max_w": 1178.89,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 11233.75,
+        "pcie_rx_max_mb_s": 11359.0,
+        "pcie_tx_avg_mb_s": 11151.88,
+        "pcie_tx_max_mb_s": 11299.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.996605,
+      "measurement_wall_seconds": 20.000706,
+      "client_output_tokens": 5967,
+      "server_output_tokens": 5967,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 298.40065698894307,
+      "per_request_avg_tps": 74.60016424723577,
+      "ttft_avg": 2.7296009025303647,
+      "ttft_p50": 3.442606173455715,
+      "ttft_p90": 3.4429665624164043,
+      "ttft_p99": 3.443046331773512,
+      "time_to_second_token_avg": 0.04401593271177262,
+      "time_to_second_token_p50": 0.05272976343985647,
+      "time_to_second_token_p90": 0.05277097581420094,
+      "time_to_second_token_p99": 0.052783893730957064,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.013244349253168272,
+      "inter_token_latency_p50": 0.01323177277428084,
+      "inter_token_latency_p90": 0.013676198067583552,
+      "inter_token_latency_p99": 0.013812555348605936,
+      "output_tps_per_user_avg": 75.57579651538197,
+      "output_tps_per_user_p50": 75.57923042394843,
+      "output_tps_per_user_p90": 78.00785040565367,
+      "output_tps_per_user_p99": 78.74432054765556,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.03463163790764121,
+      "chunk_inter_token_latency_p50": 0.03453151627338626,
+      "chunk_inter_token_latency_p90": 0.03481200221493731,
+      "chunk_inter_token_latency_p99": 0.03492015582317328,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 1798.5,
+      "output_seq_len_p50": 1790.5,
+      "output_seq_len_p90": 1861.2,
+      "output_seq_len_p99": 1876.32,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5901360681746155,
+          "time_to_second_token": 0.01781887491233647,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013827706157608423,
+          "chunk_inter_token_latency_avg": 0.03493217289075506,
+          "input_tokens": 8192,
+          "output_tokens": 1878,
+          "output_tps_per_user": 72.31857465019748,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 3.443055195035413,
+          "time_to_second_token": 0.052722041960805655,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012686145306502984,
+          "chunk_inter_token_latency_avg": 0.03453134619303727,
+          "input_tokens": 8192,
+          "output_tokens": 1822,
+          "output_tps_per_user": 78.82615056343354,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 3.4427597529720515,
+          "time_to_second_token": 0.052737484918907285,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013322679190858855,
+          "chunk_inter_token_latency_avg": 0.034531428575409945,
+          "input_tokens": 8192,
+          "output_tokens": 1735,
+          "output_tps_per_user": 75.05997747706289,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 3.442452593939379,
+          "time_to_second_token": 0.052785329055041075,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013140866357702825,
+          "chunk_inter_token_latency_avg": 0.03453160397136258,
+          "input_tokens": 8192,
+          "output_tokens": 1759,
+          "output_tps_per_user": 76.09848337083397,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5967,
+      "wall_time": 27.59907579794526,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 298.2659723183749,
+      "server_utilization": 0.04234841193455241,
+      "server_spec_accept_rate": 0.4511494252873563,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 7.567,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.211,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.83,
+        "cpu_temp_max_c": 77.0,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 39.47,
+        "mem_util_max_pct": 43.0,
+        "temp_avg_c": 68.94,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1173.61,
+        "power_total_max_w": 1174.17,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 7879.33,
+        "pcie_rx_max_mb_s": 8354.0,
+        "pcie_tx_avg_mb_s": 7779.89,
+        "pcie_tx_max_mb_s": 8190.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.988436,
+      "measurement_wall_seconds": 20.000537,
+      "client_output_tokens": 5050,
+      "server_output_tokens": 5050,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 252.6460844518971,
+      "per_request_avg_tps": 126.32304222594856,
+      "ttft_avg": 0.951587092014961,
+      "ttft_p50": 0.951587092014961,
+      "ttft_p90": 1.232194907194935,
+      "ttft_p99": 1.295331665610429,
+      "time_to_second_token_avg": 0.0161319924518466,
+      "time_to_second_token_p50": 0.0161319924518466,
+      "time_to_second_token_p90": 0.019481185637414456,
+      "time_to_second_token_p99": 0.020234754104167224,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.007939165504351434,
+      "inter_token_latency_p50": 0.007939165504351434,
+      "inter_token_latency_p90": 0.008045622734254517,
+      "inter_token_latency_p99": 0.008069575610982711,
+      "output_tps_per_user_avg": 125.99321968668501,
+      "output_tps_per_user_p50": 125.99321968668501,
+      "output_tps_per_user_p90": 127.68267799903076,
+      "output_tps_per_user_p99": 128.06280611930853,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.02123036904289779,
+      "chunk_inter_token_latency_p50": 0.02123036904289779,
+      "chunk_inter_token_latency_p90": 0.021406878769873895,
+      "chunk_inter_token_latency_p99": 0.02144659345844352,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 2971.0,
+      "output_seq_len_p50": 2971.0,
+      "output_seq_len_p90": 3046.2,
+      "output_seq_len_p99": 3063.12,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.6008273230399936,
+          "time_to_second_token": 0.01194550096988678,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007806093966972579,
+          "chunk_inter_token_latency_avg": 0.021451006201617922,
+          "input_tokens": 32768,
+          "output_tokens": 3065,
+          "output_tps_per_user": 128.1050425771172,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.3023468609899282,
+          "time_to_second_token": 0.02031848393380642,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008072237041730289,
+          "chunk_inter_token_latency_avg": 0.021009731884177655,
+          "input_tokens": 32768,
+          "output_tokens": 2877,
+          "output_tps_per_user": 123.88139679625283,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5050,
+      "wall_time": 25.579677499830723,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 252.43712151026958,
+      "server_utilization": 0.012030798845043322,
+      "server_spec_accept_rate": 0.5173611111111112,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.549,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.873,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.84,
+        "cpu_temp_max_c": 77.25,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 45.06,
+        "mem_util_max_pct": 49.0,
+        "temp_avg_c": 68.59,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1178.44,
+        "power_total_max_w": 1178.81,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 11056.12,
+        "pcie_rx_max_mb_s": 11191.0,
+        "pcie_tx_avg_mb_s": 11029.38,
+        "pcie_tx_max_mb_s": 11153.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.966272,
+      "measurement_wall_seconds": 20.000472,
+      "client_output_tokens": 5944,
+      "server_output_tokens": 5956,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 297.7020479777527,
+      "per_request_avg_tps": 74.42551199443818,
+      "ttft_avg": 2.7296407557441853,
+      "ttft_p50": 3.4367186579620466,
+      "ttft_p90": 3.437307026120834,
+      "ttft_p99": 3.4374052726081574,
+      "time_to_second_token_avg": 0.02786868193652481,
+      "time_to_second_token_p50": 0.03361363697331399,
+      "time_to_second_token_p90": 0.033683925354853275,
+      "time_to_second_token_p99": 0.03370359269436449,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.013227158501728438,
+      "inter_token_latency_p50": 0.013280406418363698,
+      "inter_token_latency_p90": 0.01328424798952288,
+      "inter_token_latency_p99": 0.013284256693709504,
+      "output_tps_per_user_avg": 75.60591881108353,
+      "output_tps_per_user_p50": 75.29890661417849,
+      "output_tps_per_user_p90": 76.18032211148439,
+      "output_tps_per_user_p99": 76.51194460773044,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.03458492849846504,
+      "chunk_inter_token_latency_p50": 0.0345171884634303,
+      "chunk_inter_token_latency_p90": 0.03476874527228148,
+      "chunk_inter_token_latency_p99": 0.034855800227265865,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 1799.75,
+      "output_seq_len_p50": 1738.5,
+      "output_seq_len_p90": 1910.5,
+      "output_seq_len_p99": 1976.6499999999999,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.6077095181681216,
+          "time_to_second_token": 0.01054167584516108,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013063563509345002,
+          "chunk_inter_token_latency_avg": 0.03486547300004191,
+          "input_tokens": 32768,
+          "output_tokens": 1984,
+          "output_tps_per_user": 76.54879155175779,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 3.4374161888845265,
+          "time_to_second_token": 0.03363293595612049,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013284225423113112,
+          "chunk_inter_token_latency_avg": 0.034491329686020145,
+          "input_tokens": 32768,
+          "output_tokens": 1738,
+          "output_tps_per_user": 75.27725314417718,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 3.4370523130055517,
+          "time_to_second_token": 0.033594337990507483,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013276587413614283,
+          "chunk_inter_token_latency_avg": 0.03443986406695765,
+          "input_tokens": 32768,
+          "output_tokens": 1739,
+          "output_tps_per_user": 75.3205600841798,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 3.4363850029185414,
+          "time_to_second_token": 0.03370577795431018,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013284257660841351,
+          "chunk_inter_token_latency_avg": 0.03454304724084046,
+          "input_tokens": 32768,
+          "output_tokens": 1738,
+          "output_tps_per_user": 75.27707046421935,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5944,
+      "wall_time": 27.572524111019447,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 297.6989316734227,
+      "server_utilization": 0.04379210779595766,
+      "server_spec_accept_rate": 0.5028735632183908,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 7.566,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.263,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.06,
+        "cpu_temp_max_c": 76.75,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 38.81,
+        "mem_util_max_pct": 43.0,
+        "temp_avg_c": 69.0,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1173.28,
+        "power_total_max_w": 1174.27,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 7830.56,
+        "pcie_rx_max_mb_s": 8017.0,
+        "pcie_tx_avg_mb_s": 7803.78,
+        "pcie_tx_max_mb_s": 8098.0
+      }
+    }
+  ],
+  "summary_table": {
+    "0": {
+      "1": 167.19542416324822,
+      "2": 244.64143701284553,
+      "4": 292.85565306969767
+    },
+    "8192": {
+      "1": 172.95726809626717,
+      "2": 245.98233342981337,
+      "4": 298.40065698894307
+    },
+    "32768": {
+      "1": 186.35728993398365,
+      "2": 252.6460844518971,
+      "4": 297.7020479777527
+    }
+  },
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": false,
+      "mode": "skipped",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": true,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/decode-cap8192.log

@@ -0,0 +1,123 @@
+
+New version available: v0.6.2 (current: v0.4.29)
+Upgrade and restart? [Y/n]: Skipping update.
+
+╭──────────────────────────── NVIDIA P2P Override ─────────────────────────────╮
+│ Effective: yes                                                               │
+│ Configured file: yes (/etc/modprobe.d/nvidia-p2p-override.conf)              │
+│ Runtime: ForceP2P=0x11; RMForceP2PType=1; RMPcieP2PType=2;                   │
+│ GrdmaPciTopoCheckOverride=1; EnableResizableBar=1; DmaRemapPeerMmio=1        │
+╰──────────────────────────────────────────────────────────────────────────────╯
+╭─────────────────────────────── Configuration ────────────────────────────────╮
+│ LLM Inference Benchmark                                                      │
+│ Model: glm53-flash-trellismx-p8-k45 @ 127.0.0.1:8001                         │
+│ Decode concurrency: [1, 2, 4]                                                │
+│ Decode contexts: ['0', '8k', '32k']                                          │
+│ Duration: 20.0s per decode test | Max tokens: 8192                           │
+│ Pre-decode warmup: C=1 max-runnable context for 3s                           │
+│ Prefill: skipped | Sustained decode: 9 cells                                 │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Engine: vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f  
+Models: ['glm53-flash-trellismx-p8-k45']
+KV cache budget (vLLM metrics): 17,031,168 tokens (2079 blocks × 2048; local 
+4,257,792 × CP 4; CP source: local process)
+Model context length: 1,000,000 tokens
+Prefill tests: skipped
+Calibrating padding text (run=wwbkdgkvesfu, up to 32k)...
+  8k: 50,544 chars (8,192 prompt tokens via /tokenize)
+  32k: 205,140 chars (32,768 prompt tokens via /tokenize)
+  Token targeting: /tokenize exact
+Done.
+
+
+
+llm-decode-bench v0.4.29
+╭────────────────────────────────── Phase 2 ───────────────────────────────────╮
+│ Sustained Decode                                                             │
+│ Steady-state decode throughput after the engine has admitted the requested   │
+│ concurrency and passed warmup. Use this as the main tuning/regression signal │
+│ for kernels, NCCL, DCP, MTP, and scheduler changes.                          │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Aggregate tok/s  + TTFT/ITL                              
+╭────────────┬─────────────┬─────────────┬──────────────╮
+│ ctx \ conc │           1 │           2 │            4 │
+├────────────┼─────────────┼─────────────┼──────────────┤
+│ 0          │  167.2 71/6 │ 244.6 114/8 │ 292.9 183/13 │
+│ 8k         │ 173.0 579/6 │ 246.0 943/8 │  298.4 3k/13 │
+│ 32k        │ 186.4 598/5 │ 252.6 952/8 │  297.7 3k/13 │
+╰────────────┴─────────────┴─────────────┴──────────────╯
+Sustained Decode: aggregate tok/s uses OpenAI stream usage by default 
+(continuous completion_tokens when the server supports it). Prometheus is kept 
+as validation/scheduler data.
+Aggregate source(s): openai_continuous_usage
+Per-Request tok/s                    
+╭────────────┬───────┬───────┬──────╮
+│ ctx \ conc │     1 │     2 │    4 │
+├────────────┼───────┼───────┼──────┤
+│ 0          │ 167.2 │ 122.3 │ 73.2 │
+│ 8k         │ 173.0 │ 123.0 │ 74.6 │
+│ 32k        │ 186.4 │ 126.3 │ 74.4 │
+╰────────────┴───────┴───────┴──────╯
+Client request latency: p50 /   
+p90 ms                          
+╭────────────┬─────┬─────┬─────╮
+│ ctx \ conc │   1 │   2 │   4 │
+├────────────┼─────┼─────┼─────┤
+│ 0          │ —/— │ —/— │ —/— │
+│ 8k         │ —/— │ —/— │ —/— │
+│ 32k        │ —/— │ —/— │ —/— │
+╰────────────┴─────┴─────┴─────╯
+Aggregate cells show dim detail as TTFT ms / ITL ms for the same ctx/conc 
+coordinate. ITL is computed from observed generated tokens, including streams 
+stopped at the measurement boundary; a missing ITL means no stream produced at 
+least two measured output tokens. Per-request tok/s and request latency are 
+shown in separate per-cell matrices. Completion/sample counts and full 
+request-level distributions remain in JSON under request_samples.
+Sustained mode: client latency metrics explain request UX variance; aggregate 
+tok/s remains the primary throughput signal. 
+ITL=(last_token_time-first_token_time)/(output_tokens-1), user tok/s=1/ITL.
+Hardware Summary                                                                
+╭───┬─┬───────┬───────────┬───────┬─────────┬─────┬──────┬─────┬───────────────╮
+│ … │ │ mode  │ GPU avg/… │ Mem … │ W avg/… │ T … │ CPU… │ VR… │ PCIe rx/tx a… │
+├───┼─┼───────┼───────────┼───────┼─────────┼─────┼──────┼─────┼───────────────┤
+│ 0 │ │ sust… │    99/99% │   48% │ 1151/1… │ 80C │  76C │ 97… │     8407/8352 │
+│ … │ │ sust… │    99/99% │   49% │ 1154/1… │ 81C │  75C │ 97… │     8274/8292 │
+│ … │ │ sust… │   99/100% │   48% │ 1154/1… │ 82C │  76C │ 97… │     8368/8179 │
+│ 0 │ │ sust… │  100/100% │   45% │ 1177/1… │ 83C │  76C │ 97… │   11281/11144 │
+│ 0 │ │ sust… │  100/100% │   40% │ 1173/1… │ 84C │  77C │ 97… │     7881/7872 │
+│ … │ │ sust… │  100/100% │   45% │ 1178/1… │ 84C │  76C │ 97… │   11234/11152 │
+│ … │ │ sust… │  100/100% │   39% │ 1174/1… │ 84C │  77C │ 97… │     7879/7780 │
+│ … │ │ sust… │  100/100% │   45% │ 1178/1… │ 84C │  77C │ 97… │   11056/11029 │
+│ … │ │ sust… │  100/100% │   39% │ 1173/1… │ 84C │  77C │ 97… │     7831/7804 │
+╰───┴─┴───────┴───────────┴───────┴─────────┴─────┴──────┴─────┴───────────────╯
+╭───────────────────────── Whole-run GPU Power ─────────────────────────╮
+│ avg 1,101 W | max 1,179 W | limit 1,200 W | over 4m 27s | 112 samples │
+╰───────────────────────────────────────────────────────────────────────╯
+Hardware summary is sampled from nvidia-smi during the measured part of each 
+cell. Whole-run GPU power is the sampled sum of GPU power draw across the 
+complete benchmark run, not wall-outlet system power. PCIe rx/tx is MB/s and is 
+a coarse live diagnostic, not a per-kernel NCCL profiler.
+
+╭────────────────────────────────── Phase 3 ───────────────────────────────────╮
+│ Burst / E2E Decode                                                           │
+│ Not run. Re-run with --run-burst to append a finite client-facing request    │
+│ burst after Sustained Decode. This is intentionally disabled by default      │
+│ because it adds another full decode matrix.                                  │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+╭────────────────────────────── Primary Summary ───────────────────────────────╮
+│ Primary matrices repeated last so the important numbers are visible without  │
+│ scrolling back through diagnostics.                                          │
+╰─────────────────────────────────────��────────────────────────────────────────╯
+Aggregate decode tok/s                
+╭────────────┬───────┬───────┬───────╮
+│ ctx \ conc │     1 │     2 │     4 │
+├────────────┼───────┼───────┼───────┤
+│ 0          │ 167.2 │ 244.6 │ 292.9 │
+│ 8k         │ 173.0 │ 246.0 │ 298.4 │
+│ 32k        │ 186.4 │ 252.6 │ 297.7 │
+╰────────────┴───────┴───────┴───────╯
+
+Results saved to 
+<campaign>/batch16384-speed-w
+indow-01/results-01/batch16384/rep-1/decode-cap8192.json

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/prefill-command.json

@@ -0,0 +1,31 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "8192",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/batch16384-speed-window-01/results-01/batch16384/rep-1/prefill.json",
+  "--contexts",
+  "0",
+  "--concurrency",
+  "1,2,4",
+  "--prefill-only",
+  "--prefill-contexts",
+  "8k,32k,64k,128k",
+  "--prefill-duration",
+  "20",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/prefill-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "0068157e646ee939ef2c7a0c5c860e6ec4582eecfe6284929bdeaa7272046797"
+}

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/prefill.json

@@ -0,0 +1,396 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T07:13:30.053751",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "standalone_cold",
+    "standalone_prefill": true,
+    "prefill_only": true,
+    "skip_prefill": false,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 0,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 17031168,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": true,
+      "prefill_only": true,
+      "skip_prefill": false,
+      "prefill_contexts": "8k,32k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 46,
+    "duration_seconds": 108.093,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 10.0,
+    "cpu_temp_max_c": 76.25,
+    "gpu_util_avg_pct": 84.77,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 22.67,
+    "mem_util_max_pct": 35.0,
+    "temp_avg_c": 59.3,
+    "temp_max_c": 80.0,
+    "power_total_avg_w": 1039.43,
+    "power_total_max_w": 1180.51,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 376719.74,
+    "vram_used_max_mb": 380174.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 96.22,
+    "vram_used_max_pct": 97.1,
+    "pcie_rx_avg_mb_s": 42900.63,
+    "pcie_rx_max_mb_s": 68701.0,
+    "pcie_tx_avg_mb_s": 40511.74,
+    "pcie_tx_max_mb_s": 63363.0
+  },
+  "event_log": [],
+  "prefill": {
+    "8192": {
+      "ttft_seconds": 1.041,
+      "prefill_seconds": 1.041,
+      "tok_per_sec": 7874.0,
+      "client_ttft_seconds": 1.041,
+      "client_tok_per_sec": 7874.0,
+      "prompt_tokens": 8194,
+      "samples": 14,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.127,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.29,
+        "cpu_temp_max_c": 75.25,
+        "gpu_util_avg_pct": 65.11,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 18.42,
+        "mem_util_max_pct": 33.0,
+        "temp_avg_c": 52.0,
+        "temp_max_c": 66.0,
+        "power_total_avg_w": 963.89,
+        "power_total_max_w": 1159.22,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 366586.0,
+        "vram_used_max_mb": 366586.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 93.62,
+        "vram_used_max_pct": 93.62,
+        "pcie_rx_avg_mb_s": 33986.56,
+        "pcie_rx_max_mb_s": 58738.0,
+        "pcie_tx_avg_mb_s": 32431.22,
+        "pcie_tx_max_mb_s": 56269.0
+      }
+    },
+    "32768": {
+      "ttft_seconds": 4.08,
+      "prefill_seconds": 4.08,
+      "tok_per_sec": 8033.0,
+      "client_ttft_seconds": 4.08,
+      "client_tok_per_sec": 8033.0,
+      "prompt_tokens": 32770,
+      "samples": 5,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.245,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.29,
+        "cpu_temp_max_c": 75.75,
+        "gpu_util_avg_pct": 97.5,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 25.89,
+        "mem_util_max_pct": 33.0,
+        "temp_avg_c": 58.39,
+        "temp_max_c": 73.0,
+        "power_total_avg_w": 1058.45,
+        "power_total_max_w": 1148.19,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380045.11,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.07,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 50337.22,
+        "pcie_rx_max_mb_s": 61317.0,
+        "pcie_tx_avg_mb_s": 47808.0,
+        "pcie_tx_max_mb_s": 54706.0
+      }
+    },
+    "65536": {
+      "ttft_seconds": 8.239,
+      "prefill_seconds": 8.239,
+      "tok_per_sec": 7955.0,
+      "client_ttft_seconds": 8.239,
+      "client_tok_per_sec": 7955.0,
+      "prompt_tokens": 65538,
+      "samples": 3,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 11,
+        "duration_seconds": 24.061,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.33,
+        "cpu_temp_max_c": 76.0,
+        "gpu_util_avg_pct": 94.18,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 25.11,
+        "mem_util_max_pct": 35.0,
+        "temp_avg_c": 61.48,
+        "temp_max_c": 77.0,
+        "power_total_avg_w": 1093.56,
+        "power_total_max_w": 1180.51,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 50410.73,
+        "pcie_rx_max_mb_s": 59338.0,
+        "pcie_tx_avg_mb_s": 47142.09,
+        "pcie_tx_max_mb_s": 60407.0
+      }
+    },
+    "131072": {
+      "ttft_seconds": 16.814,
+      "prefill_seconds": 16.814,
+      "tok_per_sec": 7796.0,
+      "client_ttft_seconds": 16.814,
+      "client_tok_per_sec": 7796.0,
+      "prompt_tokens": 131074,
+      "samples": 2,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 15,
+        "duration_seconds": 33.649,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.22,
+        "cpu_temp_max_c": 76.25,
+        "gpu_util_avg_pct": 93.33,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 24.52,
+        "mem_util_max_pct": 31.0,
+        "temp_avg_c": 64.23,
+        "temp_max_c": 80.0,
+        "power_total_avg_w": 1110.04,
+        "power_total_max_w": 1154.26,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 380174.0,
+        "vram_used_max_mb": 380174.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 97.1,
+        "vram_used_max_pct": 97.1,
+        "pcie_rx_avg_mb_s": 43409.47,
+        "pcie_rx_max_mb_s": 68701.0,
+        "pcie_tx_avg_mb_s": 40960.33,
+        "pcie_tx_max_mb_s": 63363.0
+      }
+    }
+  },
+  "results": [],
+  "summary_table": {},
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": true,
+      "mode": "standalone_cold",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": false,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/batch16384/rep-1/prefill.log

@@ -0,0 +1,56 @@
+
+New version available: v0.6.2 (current: v0.4.29)
+Upgrade and restart? [Y/n]: Skipping update.
+
+╭──────────────────────────── NVIDIA P2P Override ─────────────────────────────╮
+│ Effective: yes                                                               │
+│ Configured file: yes (/etc/modprobe.d/nvidia-p2p-override.conf)              │
+│ Runtime: ForceP2P=0x11; RMForceP2PType=1; RMPcieP2PType=2;                   │
+│ GrdmaPciTopoCheckOverride=1; EnableResizableBar=1; DmaRemapPeerMmio=1        │
+╰──────────────────────────────────────────────────────────────────────────────╯
+╭─────────────────────────────── Configuration ────────────────────────────────╮
+│ LLM Inference Benchmark                                                      │
+│ Model: glm53-flash-trellismx-p8-k45 @ 127.0.0.1:8001                         │
+│ Decode concurrency: [1, 2, 4]                                                │
+│ Decode contexts: ['0']                                                       │
+│ Decode: skipped (--prefill-only) | Max tokens: 8192                          │
+│ Pre-decode warmup: C=1 max-runnable context for 3s                           │
+│ Prefill-only: standalone cold profile (client) | Sustained decode: 0 cells   │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Engine: vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f  
+Models: ['glm53-flash-trellismx-p8-k45']
+KV cache budget (vLLM metrics): 17,031,168 tokens (2079 blocks × 2048; local 
+4,257,792 × CP 4; CP source: local process)
+Model context length: 1,000,000 tokens
+Prefill tests: standalone cold profile ['8k', '32k', '64k', '128k']
+Calibrating padding text (run=jowlqgbaqdiz, up to 128k)...
+  8k: 50,558 chars (8,192 prompt tokens via /tokenize)
+  32k: 205,152 chars (32,768 prompt tokens via /tokenize)
+  64k: 411,264 chars (65,536 prompt tokens via /tokenize)
+  128k: 823,408 chars (131,072 prompt tokens via /tokenize)
+  Token targeting: /tokenize exact
+Done.
+
+
+
+llm-decode-bench v0.4.29
+Prefill Speed (C=1, client ISL / TTFT)                                          
+                                                                                
+                                      Client                   PCIe rx/tx       
+  Context    Tokens   TTFT (s)         tok/s   Server tok/s           avg    N  
+ ────────────────────────────────────────────────────────────────────────────── 
+  8k          8,194       1.04         7,874              —   33987/32431   14  
+  32k        32,770       4.08         8,033              —   50337/47808    5  
+  64k        65,538       8.24         7,955              —   50411/47142    3  
+  128k      131,074      16.81         7,796              —   43409/40960    2  
+                                                                                
+Client tok/s = prompt_tokens / TTFT. Integrated scout rows come from the 
+prefix-cache scout request that decode needs anyway. Server tok/s is optional 
+Prometheus validation when the engine exports prefill counters and the exact 
+counter delta is uncontaminated; for vLLM this uses newly computed KV tokens, 
+not request prompt tokens.
+
+
+Results saved to 
+<campaign>/batch16384-speed-w
+indow-01/results-01/batch16384/rep-1/prefill.json

results/speed-20260909/evidence/batch16384-speed-window-01/results-01/failure.json

@@ -0,0 +1,3 @@
+{
+  "error": "KeyboardInterrupt()"
+}

results/speed-20260909/evidence/candidate-graph-analysis-02/REPORT.md

@@ -0,0 +1,20 @@
+# Candidate FP8 serving graph profile
+
+All four planned cells completed on retained image sha256:9bb99e0b47f00c4ccf77f2d4be77bb7c3ce8d6b5636169aa1f1fddda040ab45e. Runtime/source receipts match TP4/DCP4, MTP3 probabilistic, NVFP4 KV, graphs, maxseq24, batch4096, four300W GPUs. Original production and clients stayed stopped. These are instrumented diagnostic traces, not new throughput claims.
+
+| Profile | Graph kernel events mapped | Non-graph kernel events |
+|---|---:|---:|
+| 8K C1 decode | 1020936 | 31624 |
+| 8K C4 decode | 429400 | 16400 |
+| 32K cold prefill plus setup | 214348 | 238940 |
+| 64K cold prefill plus setup | 219208 | 350852 |
+
+All1,883,892graph-associated events map via explicit process-scoped clone/original node IDs to captured topology. All6,368graph receipts succeeded. Both decode clients report zero errors and no underfill/capacity/warmup-timeout flags. Raw capture timings include start/stop RPC latency; nominal1.5seconds is the controller sleep between profiling RPCs, not an exact measured trace duration.
+
+Target P8 direct FC1/FC2 kernels are present in both decode traces. P8 grouped FC1/FC2 kernels are additionally present in both prefill traces. The serving adapter directly dispatches P8NativeTPMoE with use_a16=False; missing target runtime raises an error. Marlin kernels are recorded separately and must retain their layer-attribution boundary. Four representative compiled target regimes were independently checked for QMMA.SF.16832.F32.E4M3.E4M3.E8; no W4A16 target replacement was made.
+
+Do not remove waits from this evidence. Many graph edges have non-default dependency/port types, which the analyzer preserves without interpreting them as full-completion barriers. Some default-edge kernel timestamps overlap by approximately1.0-1.44microseconds in C1; keep these anomalies instead of treating them as dependency violations or deleting edges. Cross-rank, eager-prefill and external-stream dependencies remain incomplete. Summed durations are not wall time or attainable speedup fractions.
+
+Next bounded candidate comes from actual grouped FC2 launch geometry:256CTAs,128threads, registers199(K4)/216(K5), reported dynamic shared34816/38912bytes and executed partition102400bytes where available. These resources admit2blocks per188-SM device, or376CTAs, but current grid exposes only256. Test376CTAs with unchanged task stride/ownership and unchanged FP8/Hadamard/orderedK512 math. More CTAs might reduce imbalance or hurt locality; CPU task-coverage/source-scope proof passed and incremental component test is running. This does not assume the300C1/20000prefill targets are reachable from this change.
+
+All raw Nsight reports and graph records are indexed by graph-profile-preservation-02/INDEX.json. Exported SQLite and these analyses are additional artifacts, hashed separately here. Earlier failed profiling attempts remain preserved.

results/speed-20260909/evidence/candidate-graph-analysis-02/SUMMARY.json

@@ -0,0 +1,242 @@
+{
+  "status": "all-four-profiles-completed-and-graph-associated-kernels-mapped",
+  "profiles": [
+    {
+      "profile": "8K C1 decode",
+      "coverage": {
+        "total_kernel_events": 1052560,
+        "matched_graph_kernel_events": 1020936,
+        "non_graph_kernel_events": 31624
+      },
+      "kernel_categories": {
+        "other": {
+          "calls": 1010080,
+          "summed_duration_ns": 5759231895
+        },
+        "Marlin-needs-layer-attribution": {
+          "calls": 2832,
+          "summed_duration_ns": 64929376
+        },
+        "target-P8-direct-FC1": {
+          "calls": 19824,
+          "summed_duration_ns": 1332961683
+        },
+        "target-P8-direct-FC2": {
+          "calls": 19824,
+          "summed_duration_ns": 476408029
+        }
+      },
+      "edge_checks": {
+        "both_endpoints_timed": 1122416,
+        "nondefault_edges_not_interpreted_as_completion": 433768,
+        "default_completion_edges": 688648
+      },
+      "timing_anomaly_count": 299,
+      "grouped_fc2_resources": [],
+      "analysis_sha256": "8f6e2864cbd68a184a6f8b33b74adaa31da50612ce33e99e230b1205a54e0a27"
+    },
+    {
+      "profile": "8K C4 decode",
+      "coverage": {
+        "total_kernel_events": 445800,
+        "matched_graph_kernel_events": 429400,
+        "non_graph_kernel_events": 16400
+      },
+      "kernel_categories": {
+        "other": {
+          "calls": 427800,
+          "summed_duration_ns": 3763933505
+        },
+        "Marlin-needs-layer-attribution": {
+          "calls": 1200,
+          "summed_duration_ns": 66829405
+        },
+        "target-P8-direct-FC1": {
+          "calls": 8400,
+          "summed_duration_ns": 2249130432
+        },
+        "target-P8-direct-FC2": {
+          "calls": 8400,
+          "summed_duration_ns": 846519484
+        }
+      },
+      "edge_checks": {
+        "both_endpoints_timed": 473200,
+        "default_completion_edges": 279800,
+        "nondefault_edges_not_interpreted_as_completion": 193400
+      },
+      "timing_anomaly_count": 0,
+      "grouped_fc2_resources": [],
+      "analysis_sha256": "ecdbc033af4eae4984b62014ccc2e7f586a2b4e98992b3edba945b887f56fa57"
+    },
+    {
+      "profile": "32K cold prefill plus setup",
+      "coverage": {
+        "total_kernel_events": 453288,
+        "matched_graph_kernel_events": 214348,
+        "non_graph_kernel_events": 238940
+      },
+      "kernel_categories": {
+        "other": {
+          "calls": 438168,
+          "summed_duration_ns": 25529631362
+        },
+        "Marlin-needs-layer-attribution": {
+          "calls": 1008,
+          "summed_duration_ns": 21740194
+        },
+        "target-P8-direct-FC1": {
+          "calls": 3864,
+          "summed_duration_ns": 246439082
+        },
+        "target-P8-direct-FC2": {
+          "calls": 3864,
+          "summed_duration_ns": 87860594
+        },
+        "target-P8-grouped-FC1": {
+          "calls": 3192,
+          "summed_duration_ns": 4867068677
+        },
+        "target-P8-grouped-FC2": {
+          "calls": 3192,
+          "summed_duration_ns": 3764819288
+        }
+      },
+      "edge_checks": {
+        "both_endpoints_timed": 232380,
+        "nondefault_edges_not_interpreted_as_completion": 87884,
+        "default_completion_edges": 144496
+      },
+      "timing_anomaly_count": 69,
+      "grouped_fc2_resources": [
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o754974721_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 199,
+          "dynamic_shared": 34816,
+          "shared_partition": 0,
+          "requested_percent": null,
+          "calls": 1800
+        },
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o754974721_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 199,
+          "dynamic_shared": 34816,
+          "shared_partition": 102400,
+          "requested_percent": 68,
+          "calls": 100
+        },
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o943718401_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 216,
+          "dynamic_shared": 38912,
+          "shared_partition": 0,
+          "requested_percent": null,
+          "calls": 1224
+        },
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o943718401_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 216,
+          "dynamic_shared": 38912,
+          "shared_partition": 102400,
+          "requested_percent": 76,
+          "calls": 68
+        }
+      ],
+      "analysis_sha256": "dbc87a37bea8fdc6ce706e723c40a34d0596c40a075e8db8d9c506ead76f76cd"
+    },
+    {
+      "profile": "64K cold prefill plus setup",
+      "coverage": {
+        "total_kernel_events": 570060,
+        "matched_graph_kernel_events": 219208,
+        "non_graph_kernel_events": 350852
+      },
+      "kernel_categories": {
+        "other": {
+          "calls": 551700,
+          "summed_duration_ns": 39640395478
+        },
+        "target-P8-grouped-FC1": {
+          "calls": 4704,
+          "summed_duration_ns": 7145088363
+        },
+        "target-P8-grouped-FC2": {
+          "calls": 4704,
+          "summed_duration_ns": 5633648267
+        },
+        "Marlin-needs-layer-attribution": {
+          "calls": 1224,
+          "summed_duration_ns": 25492544
+        },
+        "target-P8-direct-FC1": {
+          "calls": 3864,
+          "summed_duration_ns": 247245351
+        },
+        "target-P8-direct-FC2": {
+          "calls": 3864,
+          "summed_duration_ns": 87240073
+        }
+      },
+      "edge_checks": {
+        "both_endpoints_timed": 237024,
+        "default_completion_edges": 148096,
+        "nondefault_edges_not_interpreted_as_completion": 88928
+      },
+      "timing_anomaly_count": 61,
+      "grouped_fc2_resources": [
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o754974721_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 199,
+          "dynamic_shared": 34816,
+          "shared_partition": 0,
+          "requested_percent": null,
+          "calls": 2700
+        },
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o754974721_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 199,
+          "dynamic_shared": 34816,
+          "shared_partition": 102400,
+          "requested_percent": 68,
+          "calls": 100
+        },
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o943718401_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 216,
+          "dynamic_shared": 38912,
+          "shared_partition": 0,
+          "requested_percent": null,
+          "calls": 1836
+        },
+        {
+          "kernel": "kernel_cutlass_kernel_b12xmoe_sharedkernelsp8_coupled_prefill_fc2P8CoupledPrefillFC2Kernel_object_at__tensorptri32gmemalign16o1_tensorptri32gmemalign16o943718401_tensorptri32gmemalign16o1_2",
+          "grid_z": 256,
+          "block_x": 128,
+          "registers": 216,
+          "dynamic_shared": 38912,
+          "shared_partition": 102400,
+          "requested_percent": 76,
+          "calls": 68
+        }
+      ],
+      "analysis_sha256": "a5bb8b2ec42a9930a512254c254e23c163d31b3bfd8afc9edcaed2aa1c33c55d"
+    }
+  ],
+  "throughput_claim": false,
+  "next_candidate": "grouped-fc2-grid376-component-01",
+  "scope": "All graph-associated kernel events matched; eager/non-graph prefill kernels remain outside graph-topology coverage."
+}

results/speed-20260909/evidence/candidate-graph-profile-01/results-01/failure.json

@@ -0,0 +1,3 @@
+{
+  "error": "RuntimeError(\"trellismx-candidate-graph-profile-01 exited: {'Status': 'exited', 'Running': False, 'Paused': False, 'Restarting': False, 'OOMKilled': False, 'Dead': False, 'Pid': 0, 'ExitCode': 1, 'Error': '', 'StartedAt': '2026-09-09T14:00:23.959294351Z', 'FinishedAt': '2026-09-09T14:03:17.551677807Z'}\")"
+}

results/speed-20260909/evidence/candidate-graph-profile-02/results-01/decode-c1-8k/result.json

@@ -0,0 +1,345 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T10:10:50.723841",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "skipped",
+    "standalone_prefill": false,
+    "prefill_only": false,
+    "skip_prefill": true,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 8192,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": 0.0,
+    "ignore_eos": true,
+    "max_total_tokens": 29188096,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1
+    ],
+    "context_lengths": [
+      8192
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1",
+      "contexts": "8k",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": false,
+      "prefill_only": false,
+      "skip_prefill": true,
+      "prefill_contexts": "8k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 19,
+    "duration_seconds": 43.426,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 8.54,
+    "cpu_temp_max_c": 75.12,
+    "gpu_util_avg_pct": 64.61,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 35.28,
+    "mem_util_max_pct": 60.0,
+    "temp_avg_c": 48.41,
+    "temp_max_c": 67.0,
+    "power_total_avg_w": 874.72,
+    "power_total_max_w": 1154.84,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 386175.47,
+    "vram_used_max_mb": 386578.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.63,
+    "vram_used_max_pct": 98.73,
+    "pcie_rx_avg_mb_s": 8453.16,
+    "pcie_rx_max_mb_s": 48004.0,
+    "pcie_tx_avg_mb_s": 8921.84,
+    "pcie_tx_max_mb_s": 47296.0
+  },
+  "event_log": [
+    "10:10:04 benchmark start engine=vllm",
+    "10:10:04 startup server=http://127.0.0.1:8001 model=glm53-flash-trellismx-p8-k45",
+    "10:10:04 startup decode concurrency=1 contexts=8k",
+    "10:10:04 startup NVIDIA P2P override: enabled: runtime NVIDIA P2P override matches expected RegistryDwords",
+    "10:10:04 startup engine vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f models=['glm53-flash-trellismx-p8-k45']",
+    "10:10:04 startup KV cache budget from vLLM metrics: 29,188,096 tokens (3563 blocks x 2048; local 7,297,024 \u00d7 CP 4; CP source: local process)",
+    "10:10:04 startup model context length: 1,000,000 tokens",
+    "10:10:04 startup prefill tests: skipped",
+    "10:10:04 startup calibrating padding text run=tmxrepeatabc up_to=8k",
+    "10:10:04 startup context 8k: 50,558 chars (8,192 prompt tokens via /tokenize)",
+    "10:10:04 startup token targeting: /tokenize exact",
+    "10:10:04 startup startup preparation done",
+    "10:10:04 hardware monitor interval=2s",
+    "10:10:04 decode warmup start",
+    "10:10:05 decode warmup start C=1 ctx=8k 3s",
+    "10:10:05 cell start C=1 ctx=8k",
+    "10:10:10 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "10:10:21 cell done C=1 ctx=8k 212.1 tok/s",
+    "10:10:21 decode warmup done C=1 ctx=8k",
+    "10:10:23 cell start C=1 ctx=8k",
+    "10:10:28 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "10:10:48 cell done C=1 ctx=8k 215.3 tok/s"
+  ],
+  "prefill": {},
+  "results": [
+    {
+      "concurrency": 1,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.989377,
+      "measurement_wall_seconds": 20.000509,
+      "client_output_tokens": 4303,
+      "server_output_tokens": 4306,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 215.26434223990609,
+      "per_request_avg_tps": 215.26434223990609,
+      "ttft_avg": 0.5413568730000407,
+      "ttft_p50": 0.5413568730000407,
+      "ttft_p90": 0.5413568730000407,
+      "ttft_p99": 0.5413568730000407,
+      "time_to_second_token_avg": 0.014796818839386106,
+      "time_to_second_token_p50": 0.014796818839386106,
+      "time_to_second_token_p90": 0.014796818839386106,
+      "time_to_second_token_p99": 0.014796818839386106,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.004611418287442928,
+      "inter_token_latency_p50": 0.004611418287442928,
+      "inter_token_latency_p90": 0.004611418287442928,
+      "inter_token_latency_p99": 0.004611418287442928,
+      "output_tps_per_user_avg": 216.8530238783671,
+      "output_tps_per_user_p50": 216.8530238783671,
+      "output_tps_per_user_p90": 216.8530238783671,
+      "output_tps_per_user_p99": 216.8530238783671,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.012825661236893406,
+      "chunk_inter_token_latency_p50": 0.012825661236893406,
+      "chunk_inter_token_latency_p90": 0.012825661236893406,
+      "chunk_inter_token_latency_p99": 0.012825661236893406,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 5202.0,
+      "output_seq_len_p50": 5202.0,
+      "output_seq_len_p90": 5202.0,
+      "output_seq_len_p99": 5202.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5413568730000407,
+          "time_to_second_token": 0.014796818839386106,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.004611418287442928,
+          "chunk_inter_token_latency_avg": 0.012825661236893406,
+          "input_tokens": 8192,
+          "output_tokens": 5202,
+          "output_tps_per_user": 216.8530238783671,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4303,
+      "wall_time": 25.557856339029968,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 215.2383033924078,
+      "server_utilization": 0.006176305446378483,
+      "server_spec_accept_rate": 0.3924050632911392,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.552,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.391,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.9,
+        "cpu_temp_max_c": 75.12,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 56.72,
+        "mem_util_max_pct": 60.0,
+        "temp_avg_c": 53.08,
+        "temp_max_c": 67.0,
+        "power_total_avg_w": 1148.08,
+        "power_total_max_w": 1154.84,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 386578.0,
+        "vram_used_max_mb": 386578.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.73,
+        "vram_used_max_pct": 98.73,
+        "pcie_rx_avg_mb_s": 9520.33,
+        "pcie_rx_max_mb_s": 9579.0,
+        "pcie_tx_avg_mb_s": 9139.0,
+        "pcie_tx_max_mb_s": 9237.0
+      }
+    }
+  ],
+  "summary_table": {
+    "8192": {
+      "1": 215.26434223990609
+    }
+  },
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": false,
+      "mode": "skipped",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": true,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-graph-profile-02/results-01/decode-c4-8k/result.json

@@ -0,0 +1,381 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T10:11:44.827451",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "skipped",
+    "standalone_prefill": false,
+    "prefill_only": false,
+    "skip_prefill": true,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 8192,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": 0.0,
+    "ignore_eos": true,
+    "max_total_tokens": 29188096,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      4
+    ],
+    "context_lengths": [
+      8192
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "4",
+      "contexts": "8k",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": false,
+      "prefill_only": false,
+      "skip_prefill": true,
+      "prefill_contexts": "8k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 21,
+    "duration_seconds": 48.105,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 9.28,
+    "cpu_temp_max_c": 76.0,
+    "gpu_util_avg_pct": 76.04,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 35.81,
+    "mem_util_max_pct": 59.0,
+    "temp_avg_c": 57.21,
+    "temp_max_c": 75.0,
+    "power_total_avg_w": 970.98,
+    "power_total_max_w": 1182.0,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 386578.0,
+    "vram_used_max_mb": 386578.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.73,
+    "vram_used_max_pct": 98.73,
+    "pcie_rx_avg_mb_s": 9912.24,
+    "pcie_rx_max_mb_s": 43706.0,
+    "pcie_tx_avg_mb_s": 10658.48,
+    "pcie_tx_max_mb_s": 52315.0
+  },
+  "event_log": [
+    "10:10:54 benchmark start engine=vllm",
+    "10:10:54 startup server=http://127.0.0.1:8001 model=glm53-flash-trellismx-p8-k45",
+    "10:10:54 startup decode concurrency=4 contexts=8k",
+    "10:10:54 startup NVIDIA P2P override: enabled: runtime NVIDIA P2P override matches expected RegistryDwords",
+    "10:10:54 startup engine vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f models=['glm53-flash-trellismx-p8-k45']",
+    "10:10:54 startup KV cache budget from vLLM metrics: 29,188,096 tokens (3563 blocks x 2048; local 7,297,024 \u00d7 CP 4; CP source: local process)",
+    "10:10:54 startup model context length: 1,000,000 tokens",
+    "10:10:54 startup prefill tests: skipped",
+    "10:10:54 startup calibrating padding text run=tmxrepeatabc up_to=8k",
+    "10:10:54 startup context 8k: 50,558 chars (8,192 prompt tokens via /tokenize)",
+    "10:10:54 startup token targeting: /tokenize exact",
+    "10:10:54 startup startup preparation done",
+    "10:10:54 hardware monitor interval=2s",
+    "10:10:54 decode warmup start",
+    "10:10:55 decode warmup start C=1 ctx=8k 3s",
+    "10:10:55 cell start C=1 ctx=8k",
+    "10:11:00 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "10:11:03 cell done C=1 ctx=8k 219.4 tok/s",
+    "10:11:03 decode warmup done C=1 ctx=8k",
+    "10:11:05 cell start C=4 ctx=8k",
+    "10:11:22 ready C=4 ctx=8k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "10:11:42 cell done C=4 ctx=8k 335.7 tok/s"
+  ],
+  "prefill": {},
+  "results": [
+    {
+      "concurrency": 4,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.991854,
+      "measurement_wall_seconds": 20.00195,
+      "client_output_tokens": 6712,
+      "server_output_tokens": 6712,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 335.7367410580083,
+      "per_request_avg_tps": 83.93418526450208,
+      "ttft_avg": 4.347738385258708,
+      "ttft_p50": 1.9870852205203846,
+      "ttft_p90": 9.619254515157083,
+      "ttft_p99": 12.563032215915152,
+      "time_to_second_token_avg": 0.031753852672409266,
+      "time_to_second_token_p50": 0.03671444847714156,
+      "time_to_second_token_p90": 0.036814691172912715,
+      "time_to_second_token_p99": 0.036851499965414404,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.01416472446590232,
+      "inter_token_latency_p50": 0.014673835583566086,
+      "inter_token_latency_p90": 0.015258161326212282,
+      "inter_token_latency_p99": 0.015378115281519295,
+      "output_tps_per_user_avg": 71.30240784736792,
+      "output_tps_per_user_p50": 68.1721620239172,
+      "output_tps_per_user_p90": 79.55850802561741,
+      "output_tps_per_user_p99": 83.46057979574924,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.03829464632355565,
+      "chunk_inter_token_latency_p50": 0.04061762081634814,
+      "chunk_inter_token_latency_p90": 0.04061850702094191,
+      "chunk_inter_token_latency_p99": 0.04061852844490954,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 2266.5,
+      "output_seq_len_p50": 2333.0,
+      "output_seq_len_p90": 2389.0,
+      "output_seq_len_p99": 2405.2,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5266644728835672,
+          "time_to_second_token": 0.01673092390410602,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.01539144349877563,
+          "chunk_inter_token_latency_avg": 0.04061679015537416,
+          "input_tokens": 8192,
+          "output_tokens": 2347,
+          "output_tps_per_user": 64.97116401587341,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.9872382539324462,
+          "time_to_second_token": 0.036709635984152555,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014947169590231138,
+          "chunk_inter_token_latency_avg": 0.040618451477322126,
+          "input_tokens": 8192,
+          "output_tokens": 2319,
+          "output_tps_per_user": 66.90229838922544,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.986932187108323,
+          "time_to_second_token": 0.03671926097013056,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014400501576901033,
+          "chunk_inter_token_latency_avg": 0.04061853082535039,
+          "input_tokens": 8192,
+          "output_tokens": 2407,
+          "output_tps_per_user": 69.44202565860894,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 12.890118627110496,
+          "time_to_second_token": 0.036855589831247926,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.011919783197701478,
+          "chunk_inter_token_latency_avg": 0.031324812836175914,
+          "input_tokens": 8192,
+          "output_tokens": 1993,
+          "output_tps_per_user": 83.89414332576389,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 6712,
+      "wall_time": 37.68367512291297,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 335.4828005774738,
+      "server_utilization": 0.02470522178551371,
+      "server_spec_accept_rate": 0.5364583333333334,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 17.659,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.884,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.94,
+        "cpu_temp_max_c": 75.75,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 45.19,
+        "mem_util_max_pct": 48.0,
+        "temp_avg_c": 60.38,
+        "temp_max_c": 75.0,
+        "power_total_avg_w": 1179.07,
+        "power_total_max_w": 1180.51,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 386578.0,
+        "vram_used_max_mb": 386578.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.73,
+        "vram_used_max_pct": 98.73,
+        "pcie_rx_avg_mb_s": 8712.12,
+        "pcie_rx_max_mb_s": 9059.0,
+        "pcie_tx_avg_mb_s": 8716.25,
+        "pcie_tx_max_mb_s": 9186.0
+      }
+    }
+  ],
+  "summary_table": {
+    "8192": {
+      "4": 335.7367410580083
+    }
+  },
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": false,
+      "mode": "skipped",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": true,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-graph-profile-02/results-01/prefill-32k/result.json

@@ -0,0 +1,257 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T10:11:59.320922",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "standalone_cold",
+    "standalone_prefill": true,
+    "prefill_only": true,
+    "skip_prefill": false,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 0,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": 0.0,
+    "ignore_eos": true,
+    "max_total_tokens": 29188096,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      4
+    ],
+    "context_lengths": [
+      8192
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,4",
+      "contexts": "8k",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": true,
+      "prefill_only": true,
+      "skip_prefill": false,
+      "prefill_contexts": "32k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 4,
+    "duration_seconds": 7.229,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 8.88,
+    "cpu_temp_max_c": 73.88,
+    "gpu_util_avg_pct": 72.19,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 19.81,
+    "mem_util_max_pct": 32.0,
+    "temp_avg_c": 59.81,
+    "temp_max_c": 74.0,
+    "power_total_avg_w": 936.62,
+    "power_total_max_w": 1134.55,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 386578.0,
+    "vram_used_max_mb": 386578.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.73,
+    "vram_used_max_pct": 98.73,
+    "pcie_rx_avg_mb_s": 43008.75,
+    "pcie_rx_max_mb_s": 58512.0,
+    "pcie_tx_avg_mb_s": 40414.0,
+    "pcie_tx_max_mb_s": 46717.0
+  },
+  "event_log": [],
+  "prefill": {
+    "32768": {
+      "ttft_seconds": 4.13,
+      "prefill_seconds": 4.13,
+      "tok_per_sec": 7935.0,
+      "client_ttft_seconds": 4.13,
+      "client_tok_per_sec": 7935.0,
+      "prompt_tokens": 32770,
+      "samples": 1,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 2,
+        "duration_seconds": 2.407,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 12.25,
+        "cpu_temp_max_c": 73.88,
+        "gpu_util_avg_pct": 94.5,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 26.25,
+        "mem_util_max_pct": 32.0,
+        "temp_avg_c": 61.75,
+        "temp_max_c": 74.0,
+        "power_total_avg_w": 1127.2,
+        "power_total_max_w": 1133.91,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 386578.0,
+        "vram_used_max_mb": 386578.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.73,
+        "vram_used_max_pct": 98.73,
+        "pcie_rx_avg_mb_s": 54350.5,
+        "pcie_rx_max_mb_s": 58512.0,
+        "pcie_tx_avg_mb_s": 45629.0,
+        "pcie_tx_max_mb_s": 46717.0
+      }
+    }
+  },
+  "results": [],
+  "summary_table": {},
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": true,
+      "mode": "standalone_cold",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": false,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-graph-profile-02/results-01/prefill-64k/result.json

@@ -0,0 +1,257 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T10:12:28.058934",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "standalone_cold",
+    "standalone_prefill": true,
+    "prefill_only": true,
+    "skip_prefill": false,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 0,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": 0.0,
+    "ignore_eos": true,
+    "max_total_tokens": 29188096,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      4
+    ],
+    "context_lengths": [
+      8192
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,4",
+      "contexts": "8k",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": true,
+      "prefill_only": true,
+      "skip_prefill": false,
+      "prefill_contexts": "64k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 6,
+    "duration_seconds": 12.079,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 9.23,
+    "cpu_temp_max_c": 74.12,
+    "gpu_util_avg_pct": 86.92,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 23.62,
+    "mem_util_max_pct": 37.0,
+    "temp_avg_c": 59.46,
+    "temp_max_c": 74.0,
+    "power_total_avg_w": 1002.06,
+    "power_total_max_w": 1132.26,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 386844.67,
+    "vram_used_max_mb": 386978.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.8,
+    "vram_used_max_pct": 98.83,
+    "pcie_rx_avg_mb_s": 43749.5,
+    "pcie_rx_max_mb_s": 56738.0,
+    "pcie_tx_avg_mb_s": 46172.67,
+    "pcie_tx_max_mb_s": 58712.0
+  },
+  "event_log": [],
+  "prefill": {
+    "65536": {
+      "ttft_seconds": 8.202,
+      "prefill_seconds": 8.202,
+      "tok_per_sec": 7991.0,
+      "client_ttft_seconds": 8.202,
+      "client_tok_per_sec": 7991.0,
+      "prompt_tokens": 65538,
+      "samples": 1,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 4,
+        "duration_seconds": 7.26,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.1,
+        "cpu_temp_max_c": 74.12,
+        "gpu_util_avg_pct": 99.75,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 26.44,
+        "mem_util_max_pct": 29.0,
+        "temp_avg_c": 61.12,
+        "temp_max_c": 74.0,
+        "power_total_avg_w": 1131.9,
+        "power_total_max_w": 1132.26,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 386928.0,
+        "vram_used_max_mb": 386978.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.82,
+        "vram_used_max_pct": 98.83,
+        "pcie_rx_avg_mb_s": 53121.5,
+        "pcie_rx_max_mb_s": 56738.0,
+        "pcie_tx_avg_mb_s": 52848.5,
+        "pcie_tx_max_mb_s": 57089.0
+      }
+    }
+  },
+  "results": [],
+  "summary_table": {},
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": true,
+      "mode": "standalone_cold",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": false,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-graph-profile-02/results-01/result.json

@@ -0,0 +1,7 @@
+{
+  "status": "four-profile-cells-completed",
+  "graph_join": "pending",
+  "target_p8_dispatch": "pending Nsight classification",
+  "throughput_claim": false,
+  "production_restarted": false
+}

results/speed-20260909/evidence/candidate-kernel-window-02/results-01/result.json

@@ -0,0 +1,23 @@
+{
+  "scope": "representative component equality",
+  "results": [
+    {
+      "rank": 0,
+      "status": "passed"
+    },
+    {
+      "rank": 1,
+      "status": "passed"
+    },
+    {
+      "rank": 2,
+      "status": "passed"
+    },
+    {
+      "rank": 3,
+      "status": "passed"
+    }
+  ],
+  "full_model": "NOT TESTED",
+  "throughput": "NOT MEASURED"
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap512-command.json

@@ -0,0 +1,27 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "512",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap512.json",
+  "--contexts",
+  "0,8k,32k",
+  "--concurrency",
+  "1,2,4",
+  "--skip-prefill",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap512-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "e084ee41fd496dbfcf36f6f66715a1e622256de26889bd9d64e4c6c0f7335b4d"
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap512.json

@@ -0,0 +1,2366 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T02:26:52.004095",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "skipped",
+    "standalone_prefill": false,
+    "prefill_only": false,
+    "skip_prefill": true,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 32768,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 512,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 29351936,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0,
+      8192,
+      32768
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0,8k,32k",
+      "max_tokens": 512,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": false,
+      "prefill_only": false,
+      "skip_prefill": true,
+      "prefill_contexts": "8k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 113,
+    "duration_seconds": 270.016,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 10.99,
+    "cpu_temp_max_c": 77.12,
+    "gpu_util_avg_pct": 91.62,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 33.39,
+    "mem_util_max_pct": 56.0,
+    "temp_avg_c": 67.84,
+    "temp_max_c": 84.0,
+    "power_total_avg_w": 1098.07,
+    "power_total_max_w": 1177.71,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 384778.0,
+    "vram_used_max_mb": 384778.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.27,
+    "vram_used_max_pct": 98.27,
+    "pcie_rx_avg_mb_s": 16074.05,
+    "pcie_rx_max_mb_s": 74971.0,
+    "pcie_tx_avg_mb_s": 16658.8,
+    "pcie_tx_max_mb_s": 78099.0
+  },
+  "event_log": [
+    "02:22:19 benchmark start engine=vllm",
+    "02:22:19 startup server=http://127.0.0.1:8001 model=glm53-flash-trellismx-p8-k45",
+    "02:22:19 startup decode concurrency=1,2,4 contexts=0,8k,32k",
+    "02:22:19 startup NVIDIA P2P override: enabled: runtime NVIDIA P2P override matches expected RegistryDwords",
+    "02:22:19 startup engine vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f models=['glm53-flash-trellismx-p8-k45']",
+    "02:22:19 startup KV cache budget from vLLM metrics: 29,351,936 tokens (3583 blocks x 2048; local 7,337,984 \u00d7 CP 4; CP source: local process)",
+    "02:22:19 startup model context length: 1,000,000 tokens",
+    "02:22:19 startup prefill tests: skipped",
+    "02:22:19 startup calibrating padding text run=ftbfeppyynkm up_to=32k",
+    "02:22:19 startup context 8k: 50,558 chars (8,192 prompt tokens via /tokenize)",
+    "02:22:19 startup context 32k: 205,152 chars (32,768 prompt tokens via /tokenize)",
+    "02:22:19 startup token targeting: /tokenize exact",
+    "02:22:19 startup startup preparation done",
+    "02:22:19 hardware monitor interval=2s",
+    "02:22:19 decode warmup start",
+    "02:22:19 decode warmup start C=1 ctx=32k 3s",
+    "02:22:19 cell start C=1 ctx=32k",
+    "02:22:27 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:22:30 cell done C=1 ctx=32k 172.8 tok/s",
+    "02:22:30 decode warmup done C=1 ctx=32k",
+    "02:22:32 cell start C=1 ctx=0",
+    "02:22:38 ready C=1 ctx=0 running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:22:58 cell done C=1 ctx=0 181.4 tok/s",
+    "02:23:00 cell start C=1 ctx=8k",
+    "02:23:06 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:23:26 cell done C=1 ctx=8k 161.5 tok/s",
+    "02:23:28 cell start C=1 ctx=32k",
+    "02:23:37 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:23:57 cell done C=1 ctx=32k 162.9 tok/s",
+    "02:23:59 cell start C=2 ctx=0",
+    "02:24:05 ready C=2 ctx=0 running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:24:25 cell done C=2 ctx=0 257.3 tok/s",
+    "02:24:27 cell start C=4 ctx=0",
+    "02:24:32 ready C=4 ctx=0 running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:24:52 cell done C=4 ctx=0 309.0 tok/s",
+    "02:24:54 cell start C=2 ctx=8k",
+    "02:25:00 ready C=2 ctx=8k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:25:20 cell done C=2 ctx=8k 179.5 tok/s",
+    "02:25:22 cell start C=4 ctx=8k",
+    "02:25:31 ready C=4 ctx=8k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:25:51 cell done C=4 ctx=8k 217.9 tok/s",
+    "02:25:53 cell start C=2 ctx=32k",
+    "02:25:58 ready C=2 ctx=32k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:26:18 cell done C=2 ctx=32k 187.5 tok/s",
+    "02:26:20 cell start C=4 ctx=32k",
+    "02:26:29 ready C=4 ctx=32k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:26:49 cell done C=4 ctx=32k 217.2 tok/s"
+  ],
+  "prefill": {},
+  "results": [
+    {
+      "concurrency": 1,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.997167,
+      "measurement_wall_seconds": 20.00031,
+      "client_output_tokens": 3627,
+      "server_output_tokens": 3627,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 181.37569616773123,
+      "per_request_avg_tps": 181.37569616773123,
+      "ttft_avg": 0.08000486893579364,
+      "ttft_p50": 0.08374336501583457,
+      "ttft_p90": 0.08476541170384735,
+      "ttft_p99": 0.08516110226279125,
+      "time_to_second_token_avg": 0.013620631862431764,
+      "time_to_second_token_p50": 0.013637091615237296,
+      "time_to_second_token_p90": 0.01392088970169425,
+      "time_to_second_token_p99": 0.014014782523736358,
+      "request_latency_avg": 2.806835822191917,
+      "request_latency_p50": 2.8223699629306793,
+      "request_latency_p90": 2.8927299182862045,
+      "request_latency_p99": 2.9908720326423643,
+      "inter_token_latency_avg": 0.0052722589844875455,
+      "inter_token_latency_p50": 0.00534923418599077,
+      "inter_token_latency_p90": 0.005470371673862338,
+      "inter_token_latency_p99": 0.005685154040414228,
+      "output_tps_per_user_avg": 190.17533606228403,
+      "output_tps_per_user_p50": 186.94466223296394,
+      "output_tps_per_user_p90": 203.41232058199725,
+      "output_tps_per_user_p99": 211.44483943260568,
+      "e2e_output_tps_per_user_avg": 182.67236866069015,
+      "e2e_output_tps_per_user_p50": 181.4078263036614,
+      "e2e_output_tps_per_user_p90": 189.76977705704618,
+      "e2e_output_tps_per_user_p99": 196.55013284253087,
+      "chunk_inter_token_latency_avg": 0.014190152055605577,
+      "chunk_inter_token_latency_p50": 0.01418408028440155,
+      "chunk_inter_token_latency_p90": 0.014293591843861054,
+      "chunk_inter_token_latency_p99": 0.01437531164027474,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 10,
+      "completed_request_count": 9,
+      "request_samples": [
+        {
+          "ttft": 0.07054085680283606,
+          "time_to_second_token": 0.013252795208245516,
+          "latency": 2.5949868359602988,
+          "inter_token_latency_avg": 0.004940207395611473,
+          "chunk_inter_token_latency_avg": 0.014024699884208127,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 202.42065158809498,
+          "e2e_output_tps_per_user": 197.30350570758472,
+          "completed": true
+        },
+        {
+          "ttft": 0.08365814504213631,
+          "time_to_second_token": 0.013909297995269299,
+          "latency": 2.8654682198539376,
+          "inter_token_latency_avg": 0.0054438553323127225,
+          "chunk_inter_token_latency_avg": 0.014120863323917774,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 183.69334579197354,
+          "e2e_output_tps_per_user": 178.67935035974622,
+          "completed": true
+        },
+        {
+          "ttft": 0.08446813188493252,
+          "time_to_second_token": 0.013657593168318272,
+          "latency": 3.001776712015271,
+          "inter_token_latency_avg": 0.005709018747808882,
+          "chunk_inter_token_latency_avg": 0.014230773561611409,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 175.16144965959333,
+          "e2e_output_tps_per_user": 170.56565131930282,
+          "completed": true
+        },
+        {
+          "ttft": 0.07495116395875812,
+          "time_to_second_token": 0.01324723707512021,
+          "latency": 2.8223699629306793,
+          "inter_token_latency_avg": 0.005376553422645638,
+          "chunk_inter_token_latency_avg": 0.014384391617654037,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 185.99275807212763,
+          "e2e_output_tps_per_user": 181.4078263036614,
+          "completed": true
+        },
+        {
+          "ttft": 0.08520506788045168,
+          "time_to_second_token": 0.013890709029510617,
+          "latency": 2.7519469358958304,
+          "inter_token_latency_avg": 0.0052186729315369445,
+          "chunk_inter_token_latency_avg": 0.014184797170294567,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 191.61959623047144,
+          "e2e_output_tps_per_user": 186.05009904863252,
+          "completed": true
+        },
+        {
+          "ttft": 0.08382858498953283,
+          "time_to_second_token": 0.014025215059518814,
+          "latency": 2.826261157169938,
+          "inter_token_latency_avg": 0.005366795640274766,
+          "chunk_inter_token_latency_avg": 0.014283502980106277,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 186.3309257568083,
+          "e2e_output_tps_per_user": 181.158064144606,
+          "completed": true
+        },
+        {
+          "ttft": 0.07493811403401196,
+          "time_to_second_token": 0.013330142945051193,
+          "latency": 2.725051681045443,
+          "inter_token_latency_avg": 0.005186132225071294,
+          "chunk_inter_token_latency_avg": 0.0140963487606991,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 192.8219252038552,
+          "e2e_output_tps_per_user": 187.88634489441154,
+          "completed": true
+        },
+        {
+          "ttft": 0.08471656101755798,
+          "time_to_second_token": 0.013534185010939837,
+          "latency": 2.8092013269197196,
+          "inter_token_latency_avg": 0.005331672731706774,
+          "chunk_inter_token_latency_avg": 0.014264318146084616,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 187.5583987091196,
+          "e2e_output_tps_per_user": 182.25820808699618,
+          "completed": true
+        },
+        {
+          "ttft": 0.08452034182846546,
+          "time_to_second_token": 0.01361659006215632,
+          "latency": 2.8644595679361373,
+          "inter_token_latency_avg": 0.005440194180249847,
+          "chunk_inter_token_latency_avg": 0.01418336339850853,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 183.81696808367857,
+          "e2e_output_tps_per_user": 178.74226808127003,
+          "completed": true
+        },
+        {
+          "ttft": 0.07322172191925347,
+          "time_to_second_token": 0.013742553070187569,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.00470948723765711,
+          "chunk_inter_token_latency_avg": 0.01412846171297133,
+          "input_tokens": 78,
+          "output_tokens": 43,
+          "output_tps_per_user": 212.33734152711773,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3627,
+      "wall_time": 25.549715572968125,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 181.29852055793228,
+      "server_utilization": 0.005862646566164198,
+      "server_spec_accept_rate": 0.5046296296296297,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.538,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.928,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.61,
+        "cpu_temp_max_c": 76.0,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 44.03,
+        "mem_util_max_pct": 56.0,
+        "temp_avg_c": 67.31,
+        "temp_max_c": 82.0,
+        "power_total_avg_w": 1151.08,
+        "power_total_max_w": 1152.57,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8368.88,
+        "pcie_rx_max_mb_s": 8659.0,
+        "pcie_tx_avg_mb_s": 8253.75,
+        "pcie_tx_max_mb_s": 8495.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.997908,
+      "measurement_wall_seconds": 20.001023,
+      "client_output_tokens": 3229,
+      "server_output_tokens": 3229,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 161.46688548018994,
+      "per_request_avg_tps": 161.46688548018994,
+      "ttft_avg": 0.6216339806560427,
+      "ttft_p50": 0.6253726300783455,
+      "ttft_p90": 0.6310847522690892,
+      "ttft_p99": 0.631135935941711,
+      "time_to_second_token_avg": 0.017570865049492568,
+      "time_to_second_token_p50": 0.01762134348973632,
+      "time_to_second_token_p90": 0.018190144654363395,
+      "time_to_second_token_p99": 0.0183273093868047,
+      "request_latency_avg": 3.1911156099023565,
+      "request_latency_p50": 3.198385320138186,
+      "request_latency_p90": 3.315028171055019,
+      "request_latency_p99": 3.317255131038837,
+      "inter_token_latency_avg": 0.00510038525208451,
+      "inter_token_latency_p50": 0.0050694173082997274,
+      "inter_token_latency_p90": 0.005415639657450936,
+      "inter_token_latency_p99": 0.005569194878906556,
+      "output_tps_per_user_avg": 196.5708670855281,
+      "output_tps_per_user_p50": 197.2764163282376,
+      "output_tps_per_user_p90": 207.7094766010595,
+      "output_tps_per_user_p99": 208.07124550938897,
+      "e2e_output_tps_per_user_avg": 160.58636690583256,
+      "e2e_output_tps_per_user_p50": 160.0807747510169,
+      "e2e_output_tps_per_user_p90": 165.8642163873898,
+      "e2e_output_tps_per_user_p99": 166.33389003015049,
+      "chunk_inter_token_latency_avg": 0.014235832994823009,
+      "chunk_inter_token_latency_p50": 0.014211497073928412,
+      "chunk_inter_token_latency_p90": 0.014303687557771127,
+      "chunk_inter_token_latency_p99": 0.014314021112869283,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 8,
+      "completed_request_count": 7,
+      "request_samples": [
+        {
+          "ttft": 0.5874758099671453,
+          "time_to_second_token": 0.01834254991263151,
+          "latency": 3.317502571037039,
+          "inter_token_latency_avg": 0.00534251812342445,
+          "chunk_inter_token_latency_avg": 0.014218889380572364,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 187.17765235375927,
+          "e2e_output_tps_per_user": 154.33296253330428,
+          "completed": true
+        },
+        {
+          "ttft": 0.6288027700502425,
+          "time_to_second_token": 0.017373620066791773,
+          "latency": 3.3133785710670054,
+          "inter_token_latency_avg": 0.005253572996118909,
+          "chunk_inter_token_latency_avg": 0.01420410476728446,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 190.34664612802612,
+          "e2e_output_tps_per_user": 154.52505321030097,
+          "completed": true
+        },
+        {
+          "ttft": 0.6219424901064485,
+          "time_to_second_token": 0.017552112927660346,
+          "latency": 3.1045689159072936,
+          "inter_token_latency_avg": 0.004858368739336292,
+          "chunk_inter_token_latency_avg": 0.014267967964372673,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 205.8304039179643,
+          "e2e_output_tps_per_user": 164.91822661001254,
+          "completed": true
+        },
+        {
+          "ttft": 0.6217653600033373,
+          "time_to_second_token": 0.01812482811510563,
+          "latency": 3.077180569060147,
+          "inter_token_latency_avg": 0.004805117825942876,
+          "chunk_inter_token_latency_avg": 0.014193151497438206,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 208.11144205475892,
+          "e2e_output_tps_per_user": 166.38607599045721,
+          "completed": true
+        },
+        {
+          "ttft": 0.6305665450636297,
+          "time_to_second_token": 0.016708664130419493,
+          "latency": 3.198385320138186,
+          "inter_token_latency_avg": 0.005025085665507938,
+          "chunk_inter_token_latency_avg": 0.014186844061185394,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 199.00158257280566,
+          "e2e_output_tps_per_user": 160.0807747510169,
+          "completed": true
+        },
+        {
+          "ttft": 0.6203168679494411,
+          "time_to_second_token": 0.01790391909889877,
+          "latency": 3.233442581957206,
+          "inter_token_latency_avg": 0.005113748951091517,
+          "chunk_inter_token_latency_avg": 0.01420177018482481,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 195.55125008366954,
+          "e2e_output_tps_per_user": 158.34516526039127,
+          "completed": true
+        },
+        {
+          "ttft": 0.6311416230164468,
+          "time_to_second_token": 0.01687065209262073,
+          "latency": 3.093350740149617,
+          "inter_token_latency_avg": 0.004818413145074698,
+          "chunk_inter_token_latency_avg": 0.014315169285657967,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 207.53720569233118,
+          "e2e_output_tps_per_user": 165.51630998534486,
+          "completed": true
+        },
+        {
+          "ttft": 0.6310603790916502,
+          "time_to_second_token": 0.01769057405181229,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005586256570179402,
+          "chunk_inter_token_latency_avg": 0.014298766817248195,
+          "input_tokens": 8192,
+          "output_tokens": 280,
+          "output_tps_per_user": 179.01075388090973,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3229,
+      "wall_time": 26.068293957971036,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 161.40326963780146,
+      "server_utilization": 0.006141820212172022,
+      "server_spec_accept_rate": 0.5918367346938775,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 6.056,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.899,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.6,
+        "cpu_temp_max_c": 75.88,
+        "gpu_util_avg_pct": 99.12,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.22,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 67.81,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1153.3,
+        "power_total_max_w": 1156.82,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 12913.25,
+        "pcie_rx_max_mb_s": 39230.0,
+        "pcie_tx_avg_mb_s": 11280.75,
+        "pcie_tx_max_mb_s": 30681.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.99527,
+      "measurement_wall_seconds": 20.000365,
+      "client_output_tokens": 3258,
+      "server_output_tokens": 3258,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 162.9385363574482,
+      "per_request_avg_tps": 162.9385363574482,
+      "ttft_avg": 0.6264620197180193,
+      "ttft_p50": 0.6282767069060355,
+      "ttft_p90": 0.6339533890830353,
+      "ttft_p99": 0.6355810678633861,
+      "time_to_second_token_avg": 0.011372056760592386,
+      "time_to_second_token_p50": 0.011673877947032452,
+      "time_to_second_token_p90": 0.013470914796926081,
+      "time_to_second_token_p99": 0.013671491749119014,
+      "request_latency_avg": 3.179016282981528,
+      "request_latency_p50": 3.1919217659160495,
+      "request_latency_p90": 3.2463170818053184,
+      "request_latency_p99": 3.250220402767882,
+      "inter_token_latency_avg": 0.004995384074006483,
+      "inter_token_latency_p50": 0.005010807180026302,
+      "inter_token_latency_p90": 0.005126464669228168,
+      "inter_token_latency_p99": 0.00513858911541697,
+      "output_tps_per_user_avg": 200.31161173007573,
+      "output_tps_per_user_p50": 199.57010444464703,
+      "output_tps_per_user_p90": 206.65625468620667,
+      "output_tps_per_user_p99": 210.60956431602483,
+      "e2e_output_tps_per_user_avg": 161.12331942575418,
+      "e2e_output_tps_per_user_p50": 160.40493393893104,
+      "e2e_output_tps_per_user_p90": 165.31501161627682,
+      "e2e_output_tps_per_user_p99": 167.56189180026823,
+      "chunk_inter_token_latency_avg": 0.014215011037926982,
+      "chunk_inter_token_latency_p50": 0.014209193651253612,
+      "chunk_inter_token_latency_p90": 0.014249588821529074,
+      "chunk_inter_token_latency_p99": 0.014264281110023109,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 8,
+      "completed_request_count": 7,
+      "request_samples": [
+        {
+          "ttft": 0.6073537350166589,
+          "time_to_second_token": 0.01270562899298966,
+          "latency": 3.2036298899911344,
+          "inter_token_latency_avg": 0.005080775254353181,
+          "chunk_inter_token_latency_avg": 0.014187301393303145,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 196.82035711837585,
+          "e2e_output_tps_per_user": 159.81871114375727,
+          "completed": true
+        },
+        {
+          "ttft": 0.624475359916687,
+          "time_to_second_token": 0.013375401962548494,
+          "latency": 3.1919217659160495,
+          "inter_token_latency_avg": 0.005024356958902862,
+          "chunk_inter_token_latency_avg": 0.014184786773477141,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 199.03044472747092,
+          "e2e_output_tps_per_user": 160.40493393893104,
+          "completed": true
+        },
+        {
+          "ttft": 0.6302267559804022,
+          "time_to_second_token": 0.01065197098068893,
+          "latency": 3.1838252879679203,
+          "inter_token_latency_avg": 0.004997257401149742,
+          "chunk_inter_token_latency_avg": 0.014265913586522447,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 200.10976416182314,
+          "e2e_output_tps_per_user": 160.81284420188285,
+          "completed": true
+        },
+        {
+          "ttft": 0.6267525688745081,
+          "time_to_second_token": 0.010681336978450418,
+          "latency": 3.2434257329441607,
+          "inter_token_latency_avg": 0.005120691123423978,
+          "chunk_inter_token_latency_avg": 0.014221049804726372,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 195.28613929194475,
+          "e2e_output_tps_per_user": 157.8577843788769,
+          "completed": true
+        },
+        {
+          "ttft": 0.6241466680075973,
+          "time_to_second_token": 0.012666418915614486,
+          "latency": 3.2506541050970554,
+          "inter_token_latency_avg": 0.005139936276104614,
+          "chunk_inter_token_latency_avg": 0.014197337497780854,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 194.5549412059767,
+          "e2e_output_tps_per_user": 157.50676123835487,
+          "completed": true
+        },
+        {
+          "ttft": 0.6298008449375629,
+          "time_to_second_token": 0.01369377807714045,
+          "latency": 3.0510415688622743,
+          "inter_token_latency_avg": 0.004738240164236226,
+          "chunk_inter_token_latency_avg": 0.014242592493674773,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 211.0488209415602,
+          "e2e_output_tps_per_user": 167.81154515404506,
+          "completed": true
+        },
+        {
+          "ttft": 0.6331783039495349,
+          "time_to_second_token": 0.00986177520826459,
+          "latency": 3.1286156300920993,
+          "inter_token_latency_avg": 0.00488343899440815,
+          "chunk_inter_token_latency_avg": 0.01417862117126457,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 204.77372629105514,
+          "e2e_output_tps_per_user": 163.6506559244313,
+          "completed": true
+        },
+        {
+          "ttft": 0.6357619210612029,
+          "time_to_second_token": 0.007340142969042063,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.004978376419473111,
+          "chunk_inter_token_latency_avg": 0.014242485582666553,
+          "input_tokens": 32768,
+          "output_tokens": 330,
+          "output_tps_per_user": 200.86870010239915,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3258,
+      "wall_time": 29.116754403105006,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 162.8569571510368,
+      "server_utilization": 0.006979341150195384,
+      "server_spec_accept_rate": 0.6607142857142857,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 0.9,
+      "max_running_reqs": 1,
+      "effective_concurrency": 0.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 9.107,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.912,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.53,
+        "cpu_temp_max_c": 76.5,
+        "gpu_util_avg_pct": 97.97,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 38.0,
+        "mem_util_max_pct": 56.0,
+        "temp_avg_c": 68.16,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1149.9,
+        "power_total_max_w": 1155.94,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8327.0,
+        "pcie_rx_max_mb_s": 8635.0,
+        "pcie_tx_avg_mb_s": 8132.62,
+        "pcie_tx_max_mb_s": 8389.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.997106,
+      "measurement_wall_seconds": 20.0002,
+      "client_output_tokens": 5145,
+      "server_output_tokens": 5145,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 257.2872334123046,
+      "per_request_avg_tps": 128.6436167061523,
+      "ttft_avg": 0.12196867128035851,
+      "ttft_p50": 0.1225746686104685,
+      "ttft_p90": 0.14186890744604172,
+      "ttft_p99": 0.16953610270284114,
+      "time_to_second_token_avg": 0.01940517939094986,
+      "time_to_second_token_p50": 0.02044149092398584,
+      "time_to_second_token_p90": 0.020944337057881058,
+      "time_to_second_token_p99": 0.021031919752713294,
+      "request_latency_avg": 3.9966156358908242,
+      "request_latency_p50": 3.9488427881151438,
+      "request_latency_p90": 4.166194258979521,
+      "request_latency_p99": 4.237951860819012,
+      "inter_token_latency_avg": 0.007504969642701137,
+      "inter_token_latency_p50": 0.00753824443737712,
+      "inter_token_latency_p90": 0.007882004104182853,
+      "inter_token_latency_p99": 0.008046831693049199,
+      "output_tps_per_user_avg": 133.69662528358077,
+      "output_tps_per_user_p50": 132.65955785556645,
+      "output_tps_per_user_p90": 139.45345815466882,
+      "output_tps_per_user_p99": 155.8549841566213,
+      "e2e_output_tps_per_user_avg": 128.2813132485478,
+      "e2e_output_tps_per_user_p50": 129.6588243765429,
+      "e2e_output_tps_per_user_p90": 134.02921814320612,
+      "e2e_output_tps_per_user_p99": 135.85115936191062,
+      "chunk_inter_token_latency_avg": 0.02096237495683361,
+      "chunk_inter_token_latency_p50": 0.02092384556948202,
+      "chunk_inter_token_latency_p90": 0.021176504729596514,
+      "chunk_inter_token_latency_p99": 0.021264344922454187,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 14,
+      "completed_request_count": 12,
+      "request_samples": [
+        {
+          "ttft": 0.07110429811291397,
+          "time_to_second_token": 0.013481322908774018,
+          "latency": 3.940448695095256,
+          "inter_token_latency_avg": 0.007572102538125914,
+          "chunk_inter_token_latency_avg": 0.020915375118823472,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 132.06371611648814,
+          "e2e_output_tps_per_user": 129.93444138412337,
+          "completed": true
+        },
+        {
+          "ttft": 0.17250841204077005,
+          "time_to_second_token": 0.020715914899483323,
+          "latency": 4.167202410986647,
+          "inter_token_latency_avg": 0.007817405085999759,
+          "chunk_inter_token_latency_avg": 0.021135947084369718,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 127.91968549652191,
+          "e2e_output_tps_per_user": 122.86420228835883,
+          "completed": true
+        },
+        {
+          "ttft": 0.12372587202116847,
+          "time_to_second_token": 0.020451861899346113,
+          "latency": 4.157120890915394,
+          "inter_token_latency_avg": 0.007893140937170695,
+          "chunk_inter_token_latency_avg": 0.02089841978701671,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 126.69227725185547,
+          "e2e_output_tps_per_user": 123.16216281294098,
+          "completed": true
+        },
+        {
+          "ttft": 0.11786476895213127,
+          "time_to_second_token": 0.02033530385233462,
+          "latency": 3.869324626866728,
+          "inter_token_latency_avg": 0.007341408723903321,
+          "chunk_inter_token_latency_avg": 0.020841443655081095,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 136.21363931748436,
+          "e2e_output_tps_per_user": 132.32283392427672,
+          "completed": true
+        },
+        {
+          "ttft": 0.12338922801427543,
+          "time_to_second_token": 0.020431119948625565,
+          "latency": 4.137814508052543,
+          "inter_token_latency_avg": 0.007856018160544554,
+          "chunk_inter_token_latency_avg": 0.02090846500019931,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 127.2909481068057,
+          "e2e_output_tps_per_user": 123.73681783066978,
+          "completed": true
+        },
+        {
+          "ttft": 0.12249546311795712,
+          "time_to_second_token": 0.02092183893546462,
+          "latency": 3.9572368811350316,
+          "inter_token_latency_avg": 0.007504386336628326,
+          "chunk_inter_token_latency_avg": 0.02107000779130261,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 133.25539959464479,
+          "e2e_output_tps_per_user": 129.38320736896245,
+          "completed": true
+        },
+        {
+          "ttft": 0.1226538741029799,
+          "time_to_second_token": 0.020244625862687826,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.0063211789832794095,
+          "chunk_inter_token_latency_avg": 0.020737902980232446,
+          "input_tokens": 78,
+          "output_tokens": 188,
+          "output_tps_per_user": 158.198336520002,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.14964449405670166,
+          "time_to_second_token": 0.017545362003147602,
+          "latency": 3.9196609100326896,
+          "inter_token_latency_avg": 0.007377722927545964,
+          "chunk_inter_token_latency_avg": 0.020714375911955976,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 135.54317637306931,
+          "e2e_output_tps_per_user": 130.62354416666363,
+          "completed": true
+        },
+        {
+          "ttft": 0.10573618090711534,
+          "time_to_second_token": 0.013638033997267485,
+          "latency": 3.814666331978515,
+          "inter_token_latency_avg": 0.0072581803347776894,
+          "chunk_inter_token_latency_avg": 0.021193886577550853,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 137.77557926033936,
+          "e2e_output_tps_per_user": 134.21881638975384,
+          "completed": true
+        },
+        {
+          "ttft": 0.12302991887554526,
+          "time_to_second_token": 0.020953979110345244,
+          "latency": 4.246696174843237,
+          "inter_token_latency_avg": 0.008069796978410355,
+          "chunk_inter_token_latency_avg": 0.020932316020140566,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 123.91885479589686,
+          "e2e_output_tps_per_user": 120.56431138940616,
+          "completed": true
+        },
+        {
+          "ttft": 0.11646403884515166,
+          "time_to_second_token": 0.021043566055595875,
+          "latency": 3.930458223912865,
+          "inter_token_latency_avg": 0.00746378509797987,
+          "chunk_inter_token_latency_avg": 0.020841498279058544,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 133.98027768386012,
+          "e2e_output_tps_per_user": 130.26470982059993,
+          "completed": true
+        },
+        {
+          "ttft": 0.11775453994050622,
+          "time_to_second_token": 0.020746542140841484,
+          "latency": 4.055516151012853,
+          "inter_token_latency_avg": 0.007705991411100482,
+          "chunk_inter_token_latency_avg": 0.021057548722312015,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 129.7691557973319,
+          "e2e_output_tps_per_user": 126.2478019899217,
+          "completed": true
+        },
+        {
+          "ttft": 0.11773488996550441,
+          "time_to_second_token": 0.020856699906289577,
+          "latency": 3.763241825858131,
+          "inter_token_latency_avg": 0.007134064453801618,
+          "chunk_inter_token_latency_avg": 0.020951189286739235,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 140.17254910938146,
+          "e2e_output_tps_per_user": 136.05290961689627,
+          "completed": true
+        },
+        {
+          "ttft": 0.1234554189722985,
+          "time_to_second_token": 0.02030633995309472,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.00775439302854797,
+          "chunk_inter_token_latency_avg": 0.02127487318088802,
+          "input_tokens": 78,
+          "output_tokens": 215,
+          "output_tps_per_user": 128.95915854644946,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5145,
+      "wall_time": 25.557628852082416,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 257.1817350436559,
+      "server_utilization": 0.011725293132328285,
+      "server_spec_accept_rate": 0.7541666666666667,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.539,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.896,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.54,
+        "cpu_temp_max_c": 76.38,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.28,
+        "mem_util_max_pct": 50.0,
+        "temp_avg_c": 68.16,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1172.61,
+        "power_total_max_w": 1175.0,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11187.12,
+        "pcie_rx_max_mb_s": 11336.0,
+        "pcie_tx_avg_mb_s": 11117.38,
+        "pcie_tx_max_mb_s": 11239.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.982047,
+      "measurement_wall_seconds": 20.00121,
+      "client_output_tokens": 6174,
+      "server_output_tokens": 6174,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 308.9773476463454,
+      "per_request_avg_tps": 77.24433691158634,
+      "ttft_avg": 0.17778973274535553,
+      "ttft_p50": 0.1807985259220004,
+      "ttft_p90": 0.1864776147995144,
+      "ttft_p99": 0.21677800566889344,
+      "time_to_second_token_avg": 0.03174264398694504,
+      "time_to_second_token_p50": 0.03366913797799498,
+      "time_to_second_token_p90": 0.03429299045819789,
+      "time_to_second_token_p99": 0.03469313668319955,
+      "request_latency_avg": 6.506094520983215,
+      "request_latency_p50": 6.382211998803541,
+      "request_latency_p90": 6.8716935135424135,
+      "request_latency_p99": 7.440296533070504,
+      "inter_token_latency_avg": 0.01244116428728022,
+      "inter_token_latency_p50": 0.012306649429466642,
+      "inter_token_latency_p90": 0.01313604974973618,
+      "inter_token_latency_p99": 0.014173560375473144,
+      "output_tps_per_user_avg": 80.63692147986394,
+      "output_tps_per_user_p50": 81.27236798491623,
+      "output_tps_per_user_p90": 85.68934634961276,
+      "output_tps_per_user_p99": 87.43239080011531,
+      "e2e_output_tps_per_user_avg": 78.95186391949123,
+      "e2e_output_tps_per_user_p50": 80.2229697314949,
+      "e2e_output_tps_per_user_p90": 83.75147031670433,
+      "e2e_output_tps_per_user_p99": 84.93555512907336,
+      "chunk_inter_token_latency_avg": 0.03454775971555825,
+      "chunk_inter_token_latency_p50": 0.034643082201032946,
+      "chunk_inter_token_latency_p90": 0.035144349182500076,
+      "chunk_inter_token_latency_p99": 0.035562162856296174,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 17,
+      "completed_request_count": 13,
+      "request_samples": [
+        {
+          "ttft": 0.0723429499194026,
+          "time_to_second_token": 0.013155136024579406,
+          "latency": 6.662627859041095,
+          "inter_token_latency_avg": 0.012896839352488634,
+          "chunk_inter_token_latency_avg": 0.03450410947184132,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 77.53837763413215,
+          "e2e_output_tps_per_user": 76.84655526801231,
+          "completed": true
+        },
+        {
+          "ttft": 0.1807985259220004,
+          "time_to_second_token": 0.03360512410290539,
+          "latency": 6.19808776397258,
+          "inter_token_latency_avg": 0.011775517099903288,
+          "chunk_inter_token_latency_avg": 0.03399598439576599,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 84.92196066771564,
+          "e2e_output_tps_per_user": 82.60612296845576,
+          "completed": true
+        },
+        {
+          "ttft": 0.18202503910288215,
+          "time_to_second_token": 0.033671696903184056,
+          "latency": 6.092495953198522,
+          "inter_token_latency_avg": 0.011566479283944501,
+          "chunk_inter_token_latency_avg": 0.03456415739237217,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 86.45673203150989,
+          "e2e_output_tps_per_user": 84.03780715376647,
+          "completed": true
+        },
+        {
+          "ttft": 0.18031293293461204,
+          "time_to_second_token": 0.03369922889396548,
+          "latency": 6.382211998803541,
+          "inter_token_latency_avg": 0.012136788778608472,
+          "chunk_inter_token_latency_avg": 0.03503897777327079,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 82.39411744254264,
+          "e2e_output_tps_per_user": 80.2229697314949,
+          "completed": true
+        },
+        {
+          "ttft": 0.1809463920071721,
+          "time_to_second_token": 0.0,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.0,
+          "chunk_inter_token_latency_avg": 0.0,
+          "input_tokens": 78,
+          "output_tokens": 1,
+          "output_tps_per_user": 0.0,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18660231097601354,
+          "time_to_second_token": 0.02942012087441981,
+          "latency": 6.2684185188263655,
+          "inter_token_latency_avg": 0.01190179297035294,
+          "chunk_inter_token_latency_avg": 0.03378786782139084,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 84.0209540269247,
+          "e2e_output_tps_per_user": 81.67929414130147,
+          "completed": true
+        },
+        {
+          "ttft": 0.17968551511876285,
+          "time_to_second_token": 0.03475157590582967,
+          "latency": 6.310488264076412,
+          "inter_token_latency_avg": 0.01199765704296996,
+          "chunk_inter_token_latency_avg": 0.034637303666427394,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 83.34960704564823,
+          "e2e_output_tps_per_user": 81.13476779834168,
+          "completed": true
+        },
+        {
+          "ttft": 0.17955420492216945,
+          "time_to_second_token": 0.033856919035315514,
+          "latency": 6.5550508559681475,
+          "inter_token_latency_avg": 0.01247651008032481,
+          "chunk_inter_token_latency_avg": 0.03561729972651385,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 80.15061852728982,
+          "e2e_output_tps_per_user": 78.10770827717403,
+          "completed": true
+        },
+        {
+          "ttft": 0.17957319412380457,
+          "time_to_second_token": 0.034223999828100204,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012843022881727473,
+          "chunk_inter_token_latency_avg": 0.03484932613412567,
+          "input_tokens": 78,
+          "output_tokens": 484,
+          "output_tps_per_user": 77.86328882297322,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18639448401518166,
+          "time_to_second_token": 0.029375833924859762,
+          "latency": 6.019423788879067,
+          "inter_token_latency_avg": 0.011414930146504668,
+          "chunk_inter_token_latency_avg": 0.03352315692450509,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 87.60456587692804,
+          "e2e_output_tps_per_user": 85.0579753075243,
+          "completed": true
+        },
+        {
+          "ttft": 0.1805800509173423,
+          "time_to_second_token": 0.0336665790528059,
+          "latency": 7.51252193399705,
+          "inter_token_latency_avg": 0.014348222863169682,
+          "chunk_inter_token_latency_avg": 0.035249720591729365,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 69.69504234331978,
+          "e2e_output_tps_per_user": 68.15287921929428,
+          "completed": true
+        },
+        {
+          "ttft": 0.18062497000209987,
+          "time_to_second_token": 0.03364452510140836,
+          "latency": 6.614234033040702,
+          "inter_token_latency_avg": 0.012590233000075543,
+          "chunk_inter_token_latency_avg": 0.03477626520561407,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 79.42664762391608,
+          "e2e_output_tps_per_user": 77.40881218329416,
+          "completed": true
+        },
+        {
+          "ttft": 0.18063039891421795,
+          "time_to_second_token": 0.033818941097706556,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013183806278526097,
+          "chunk_inter_token_latency_avg": 0.03436578836602469,
+          "input_tokens": 78,
+          "output_tokens": 392,
+          "output_tps_per_user": 75.8506290879599,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18626655801199377,
+          "time_to_second_token": 0.029395153978839517,
+          "latency": 6.337131014093757,
+          "inter_token_latency_avg": 0.012036916743799928,
+          "chunk_inter_token_latency_avg": 0.03379595854989979,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 83.07775332209455,
+          "e2e_output_tps_per_user": 80.79365865425756,
+          "completed": true
+        },
+        {
+          "ttft": 0.22252575703896582,
+          "time_to_second_token": 0.03383295307867229,
+          "latency": 6.910643592942506,
+          "inter_token_latency_avg": 0.013088293220946262,
+          "chunk_inter_token_latency_avg": 0.03465346028965565,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 76.40415622715561,
+          "e2e_output_tps_per_user": 74.0886131825522,
+          "completed": true
+        },
+        {
+          "ttft": 0.18178053596056998,
+          "time_to_second_token": 0.03340253490023315,
+          "latency": 6.715893195942044,
+          "inter_token_latency_avg": 0.012786913228926564,
+          "chunk_inter_token_latency_avg": 0.03475591840415678,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 78.20495706014485,
+          "e2e_output_tps_per_user": 76.23706706791684,
+          "completed": true
+        },
+        {
+          "ttft": 0.18178163678385317,
+          "time_to_second_token": 0.03436198108829558,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012014705624214687,
+          "chunk_inter_token_latency_avg": 0.03464886073563849,
+          "input_tokens": 78,
+          "output_tokens": 448,
+          "output_tps_per_user": 83.23133593756798,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 6174,
+      "wall_time": 25.552133698016405,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 308.60564141811994,
+      "server_utilization": 0.02345058626465657,
+      "server_spec_accept_rate": 0.603448275862069,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 3.9,
+      "max_running_reqs": 4,
+      "effective_concurrency": 3.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.536,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.827,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.47,
+        "cpu_temp_max_c": 77.12,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 35.47,
+        "mem_util_max_pct": 45.0,
+        "temp_avg_c": 68.72,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1170.25,
+        "power_total_max_w": 1172.14,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 7813.12,
+        "pcie_rx_max_mb_s": 8161.0,
+        "pcie_tx_avg_mb_s": 8053.88,
+        "pcie_tx_max_mb_s": 8437.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.985498,
+      "measurement_wall_seconds": 20.000656,
+      "client_output_tokens": 3587,
+      "server_output_tokens": 3587,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 179.48013878635928,
+      "per_request_avg_tps": 89.74006939317964,
+      "ttft_avg": 1.1894093764014542,
+      "ttft_p50": 1.2212169540580362,
+      "ttft_p90": 1.3684029981726777,
+      "ttft_p99": 1.9125716026616284,
+      "time_to_second_token_avg": 0.025312776444479823,
+      "time_to_second_token_p50": 0.027226490550674498,
+      "time_to_second_token_p90": 0.02769280921202153,
+      "time_to_second_token_p99": 0.0277038907376118,
+      "request_latency_avg": 5.413165779871633,
+      "request_latency_p50": 5.438335195998661,
+      "request_latency_p90": 5.877898134151473,
+      "request_latency_p99": 6.255758887748234,
+      "inter_token_latency_avg": 0.00830825860560843,
+      "inter_token_latency_p50": 0.008347879003080923,
+      "inter_token_latency_p90": 0.008825519827072598,
+      "inter_token_latency_p99": 0.009230215194777226,
+      "output_tps_per_user_avg": 120.7893245972912,
+      "output_tps_per_user_p50": 119.79434916732279,
+      "output_tps_per_user_p90": 129.7361418336343,
+      "output_tps_per_user_p99": 131.83454344392672,
+      "e2e_output_tps_per_user_avg": 95.20339521438798,
+      "e2e_output_tps_per_user_p50": 94.14836874853962,
+      "e2e_output_tps_per_user_p90": 103.85715540642657,
+      "e2e_output_tps_per_user_p99": 108.00400096911441,
+      "chunk_inter_token_latency_avg": 0.023168869852685122,
+      "chunk_inter_token_latency_p50": 0.023156017746843782,
+      "chunk_inter_token_latency_p90": 0.024457862475522812,
+      "chunk_inter_token_latency_p99": 0.025339872482635258,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 10,
+      "completed_request_count": 8,
+      "request_samples": [
+        {
+          "ttft": 0.5938855609856546,
+          "time_to_second_token": 0.01751627796329558,
+          "latency": 5.02539852890186,
+          "inter_token_latency_avg": 0.008672236727820363,
+          "chunk_inter_token_latency_avg": 0.024348972351187943,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 115.31050539614768,
+          "e2e_output_tps_per_user": 101.88246704324189,
+          "completed": true
+        },
+        {
+          "ttft": 0.7053975961171091,
+          "time_to_second_token": 0.01768357283435762,
+          "latency": 4.720427100080997,
+          "inter_token_latency_avg": 0.007857200594841268,
+          "chunk_inter_token_latency_avg": 0.02206060167013125,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 127.27179202431984,
+          "e2e_output_tps_per_user": 108.46476158719085,
+          "completed": true
+        },
+        {
+          "ttft": 1.2253350547980517,
+          "time_to_second_token": 0.02671587117947638,
+          "latency": 5.5139663990121335,
+          "inter_token_latency_avg": 0.008392624939753585,
+          "chunk_inter_token_latency_avg": 0.02330777904464175,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 119.1522327255769,
+          "e2e_output_tps_per_user": 92.85511788605177,
+          "completed": true
+        },
+        {
+          "ttft": 1.2199293570593,
+          "time_to_second_token": 0.027305281022563577,
+          "latency": 5.4628303539939225,
+          "inter_token_latency_avg": 0.008303133066408263,
+          "chunk_inter_token_latency_avg": 0.023185251349369523,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 120.4364656090687,
+          "e2e_output_tps_per_user": 93.72430897944183,
+          "completed": true
+        },
+        {
+          "ttft": 1.2225045510567725,
+          "time_to_second_token": 0.02714770007878542,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.009275181346744406,
+          "chunk_inter_token_latency_avg": 0.02543787359453664,
+          "input_tokens": 8192,
+          "output_tokens": 278,
+          "output_tps_per_user": 107.81460357656518,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.3012216889765114,
+          "time_to_second_token": 0.02770512201823294,
+          "latency": 5.4138400380034,
+          "inter_token_latency_avg": 0.008048176808271797,
+          "chunk_inter_token_latency_avg": 0.022596804115532356,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 124.25174344731278,
+          "e2e_output_tps_per_user": 94.57242851763742,
+          "completed": true
+        },
+        {
+          "ttft": 1.9730347809381783,
+          "time_to_second_token": 0.02644847217015922,
+          "latency": 6.297743415925652,
+          "inter_token_latency_avg": 0.008463226291560613,
+          "chunk_inter_token_latency_avg": 0.02312678414431804,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 118.15824905889414,
+          "e2e_output_tps_per_user": 81.2989615780886,
+          "completed": true
+        },
+        {
+          "ttft": 1.213654592167586,
+          "time_to_second_token": 0.027514670975506306,
+          "latency": 5.69796444196254,
+          "inter_token_latency_avg": 0.008775557435997953,
+          "chunk_inter_token_latency_avg": 0.023114999225747185,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 113.9528750501854,
+          "e2e_output_tps_per_user": 89.85665060128959,
+          "completed": true
+        },
+        {
+          "ttft": 1.2265115019399673,
+          "time_to_second_token": 0.027691441122442484,
+          "latency": 5.1731559610925615,
+          "inter_token_latency_avg": 0.00772337467544539,
+          "chunk_inter_token_latency_avg": 0.023352925793802333,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 129.4770799064377,
+          "e2e_output_tps_per_user": 98.97246552216193,
+          "completed": true
+        },
+        {
+          "ttft": 1.2126190799754113,
+          "time_to_second_token": 0.027399355079978704,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007571874169240656,
+          "chunk_inter_token_latency_avg": 0.02115670723758419,
+          "input_tokens": 8192,
+          "output_tokens": 96,
+          "output_tps_per_user": 132.06769917840364,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3587,
+      "wall_time": 25.556549122091383,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 179.29840638216805,
+      "server_utilization": 0.012283640424343933,
+      "server_spec_accept_rate": 0.6568627450980392,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1.9,
+      "max_running_reqs": 2,
+      "effective_concurrency": 1.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.55,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.921,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.51,
+        "cpu_temp_max_c": 76.38,
+        "gpu_util_avg_pct": 99.88,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 36.44,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 68.5,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1162.41,
+        "power_total_max_w": 1177.27,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 29258.88,
+        "pcie_rx_max_mb_s": 70866.0,
+        "pcie_tx_avg_mb_s": 28010.62,
+        "pcie_tx_max_mb_s": 69989.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.930302,
+      "measurement_wall_seconds": 20.000584,
+      "client_output_tokens": 4342,
+      "server_output_tokens": 4342,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 217.85922059414168,
+      "per_request_avg_tps": 54.46480514853542,
+      "ttft_avg": 1.6845163684144306,
+      "ttft_p50": 1.2621239490108564,
+      "ttft_p90": 2.2127018794883044,
+      "ttft_p99": 4.023498326067348,
+      "time_to_second_token_avg": 0.03870494751026854,
+      "time_to_second_token_p50": 0.04049360693898052,
+      "time_to_second_token_p90": 0.041569333686493334,
+      "time_to_second_token_p99": 0.044618300595320765,
+      "request_latency_avg": 9.701175137112537,
+      "request_latency_p50": 9.277128694113344,
+      "request_latency_p90": 11.318623020872474,
+      "request_latency_p99": 12.65666121724993,
+      "inter_token_latency_avg": 0.015277014782169696,
+      "inter_token_latency_p50": 0.015349018777003027,
+      "inter_token_latency_p90": 0.016694702791212282,
+      "inter_token_latency_p99": 0.017054498783689094,
+      "output_tps_per_user_avg": 65.75017254999644,
+      "output_tps_per_user_p50": 65.15961146932285,
+      "output_tps_per_user_p90": 71.0132044249438,
+      "output_tps_per_user_p99": 72.7397198355044,
+      "e2e_output_tps_per_user_avg": 53.62687357686466,
+      "e2e_output_tps_per_user_p50": 55.18948986068087,
+      "e2e_output_tps_per_user_p90": 60.13891332324956,
+      "e2e_output_tps_per_user_p99": 62.607782503263365,
+      "chunk_inter_token_latency_avg": 0.04311462391925306,
+      "chunk_inter_token_latency_p50": 0.04431330813678551,
+      "chunk_inter_token_latency_p90": 0.04551846648306814,
+      "chunk_inter_token_latency_p99": 0.0457929443786719,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 12,
+      "completed_request_count": 9,
+      "request_samples": [
+        {
+          "ttft": 0.5955398578662425,
+          "time_to_second_token": 0.016524713020771742,
+          "latency": 8.142221544869244,
+          "inter_token_latency_avg": 0.01476845731311742,
+          "chunk_inter_token_latency_avg": 0.04035658656151338,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 67.71187936547678,
+          "e2e_output_tps_per_user": 62.882101301042674,
+          "completed": true
+        },
+        {
+          "ttft": 1.2590207229368389,
+          "time_to_second_token": 0.0395864921156317,
+          "latency": 9.238898925017565,
+          "inter_token_latency_avg": 0.015616200004071872,
+          "chunk_inter_token_latency_avg": 0.045599304011889864,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 64.03606509517381,
+          "e2e_output_tps_per_user": 55.41785922276735,
+          "completed": true
+        },
+        {
+          "ttft": 1.2618258679285645,
+          "time_to_second_token": 0.03996979701332748,
+          "latency": 9.196670216973871,
+          "inter_token_latency_avg": 0.01552807113316107,
+          "chunk_inter_token_latency_avg": 0.044577777241827564,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 64.39949890907208,
+          "e2e_output_tps_per_user": 55.67232356065407,
+          "completed": true
+        },
+        {
+          "ttft": 2.2127146429847926,
+          "time_to_second_token": 0.040582613088190556,
+          "latency": 10.946945744100958,
+          "inter_token_latency_avg": 0.017092428769307565,
+          "chunk_inter_token_latency_avg": 0.04479092872367264,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 58.50543614934785,
+          "e2e_output_tps_per_user": 46.771036594924595,
+          "completed": true
+        },
+        {
+          "ttft": 1.2621886630076915,
+          "time_to_second_token": 0.04047246486879885,
+          "latency": 8.611827800050378,
+          "inter_token_latency_avg": 0.014382855454095277,
+          "chunk_inter_token_latency_avg": 0.0445432674972284,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 69.52722310195134,
+          "e2e_output_tps_per_user": 59.453116328801286,
+          "completed": true
+        },
+        {
+          "ttft": 1.923778808210045,
+          "time_to_second_token": 0.04072440997697413,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014053212739941631,
+          "chunk_inter_token_latency_avg": 0.04110083452024025,
+          "input_tokens": 8192,
+          "output_tokens": 428,
+          "output_tps_per_user": 71.15810587267559,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2125870080199093,
+          "time_to_second_token": 0.04051474900916219,
+          "latency": 9.543051223969087,
+          "inter_token_latency_avg": 0.01434533114667158,
+          "chunk_inter_token_latency_avg": 0.04188836694828101,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 69.70909139535765,
+          "e2e_output_tps_per_user": 53.65160345299416,
+          "completed": true
+        },
+        {
+          "ttft": 1.2591751390136778,
+          "time_to_second_token": 0.040605568094179034,
+          "latency": 9.277128694113344,
+          "inter_token_latency_avg": 0.015690711458120676,
+          "chunk_inter_token_latency_avg": 0.04581687745771238,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 63.7319730637487,
+          "e2e_output_tps_per_user": 55.18948986068087,
+          "completed": true
+        },
+        {
+          "ttft": 1.4571730380412191,
+          "time_to_second_token": 0.04498353600502014,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.015169966420844982,
+          "chunk_inter_token_latency_avg": 0.042138795613458284,
+          "input_tokens": 8192,
+          "output_tokens": 476,
+          "output_tps_per_user": 65.91972402957363,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 4.247303050942719,
+          "time_to_second_token": 0.039115041960030794,
+          "latency": 12.805332127958536,
+          "inter_token_latency_avg": 0.01674761071823056,
+          "chunk_inter_token_latency_avg": 0.04457306810945738,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 59.71000979330461,
+          "e2e_output_tps_per_user": 39.98334403854502,
+          "completed": true
+        },
+        {
+          "ttft": 1.260830387007445,
+          "time_to_second_token": 0.03971677087247372,
+          "latency": 9.548499956959859,
+          "inter_token_latency_avg": 0.016218531448047777,
+          "chunk_inter_token_latency_avg": 0.044083348776342623,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 61.65786361134256,
+          "e2e_output_tps_per_user": 53.620987831371934,
+          "completed": true
+        },
+        {
+          "ttft": 1.2620592350140214,
+          "time_to_second_token": 0.04166321409866214,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013710800780425945,
+          "chunk_inter_token_latency_avg": 0.03790633156941291,
+          "input_tokens": 8192,
+          "output_tokens": 283,
+          "output_tps_per_user": 72.93520021293268,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4342,
+      "wall_time": 28.853528001811355,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 217.03898889784327,
+      "server_utilization": 0.027359017308765998,
+      "server_spec_accept_rate": 0.6283185840707964,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 3.9,
+      "max_running_reqs": 4,
+      "effective_concurrency": 3.9,
+      "avg_queue_reqs": 0.1,
+      "max_queue_reqs": 1,
+      "queue_fraction": 0.05,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 8.573,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": true,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.883,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.5,
+        "cpu_temp_max_c": 76.25,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 34.25,
+        "mem_util_max_pct": 46.0,
+        "temp_avg_c": 68.66,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1159.22,
+        "power_total_max_w": 1172.44,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 27724.5,
+        "pcie_rx_max_mb_s": 67581.0,
+        "pcie_tx_avg_mb_s": 27474.12,
+        "pcie_tx_max_mb_s": 67821.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.997661,
+      "measurement_wall_seconds": 20.00078,
+      "client_output_tokens": 3749,
+      "server_output_tokens": 3749,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 187.47192708215627,
+      "per_request_avg_tps": 93.73596354107814,
+      "ttft_avg": 1.1601904219947756,
+      "ttft_p50": 1.2142878150334582,
+      "ttft_p90": 1.3290542589034884,
+      "ttft_p99": 1.4677723066555337,
+      "time_to_second_token_avg": 0.017825737223029138,
+      "time_to_second_token_p50": 0.019167354563251138,
+      "time_to_second_token_p90": 0.020459615555591882,
+      "time_to_second_token_p99": 0.021644204219337552,
+      "request_latency_avg": 5.313494802772766,
+      "request_latency_p50": 5.375003230990842,
+      "request_latency_p90": 5.586414952063933,
+      "request_latency_p99": 5.701329016662203,
+      "inter_token_latency_avg": 0.008015108251874231,
+      "inter_token_latency_p50": 0.008146798233829671,
+      "inter_token_latency_p90": 0.008546021327498477,
+      "inter_token_latency_p99": 0.009447031634392257,
+      "output_tps_per_user_avg": 125.91902973995667,
+      "output_tps_per_user_p50": 122.74874938988106,
+      "output_tps_per_user_p90": 143.33476272223038,
+      "output_tps_per_user_p99": 150.46171413409994,
+      "e2e_output_tps_per_user_avg": 96.63340093514198,
+      "e2e_output_tps_per_user_p50": 95.25646775312458,
+      "e2e_output_tps_per_user_p90": 104.75414728581389,
+      "e2e_output_tps_per_user_p99": 105.26579246796483,
+      "chunk_inter_token_latency_avg": 0.023044757916286292,
+      "chunk_inter_token_latency_p50": 0.02316345668564709,
+      "chunk_inter_token_latency_p90": 0.02447127468927797,
+      "chunk_inter_token_latency_p99": 0.025079763939748784,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 10,
+      "completed_request_count": 8,
+      "request_samples": [
+        {
+          "ttft": 0.6090631559491158,
+          "time_to_second_token": 0.01044343295507133,
+          "latency": 5.4876536841038615,
+          "inter_token_latency_avg": 0.009547143890713788,
+          "chunk_inter_token_latency_avg": 0.025147373856467762,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 104.74336738264405,
+          "e2e_output_tps_per_user": 93.30034828602892,
+          "completed": true
+        },
+        {
+          "ttft": 1.4831854230724275,
+          "time_to_second_token": 0.016494586830958724,
+          "latency": 5.7140972460620105,
+          "inter_token_latency_avg": 0.00827967088647668,
+          "chunk_inter_token_latency_avg": 0.023119736737647993,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 120.77774753502777,
+          "e2e_output_tps_per_user": 89.60295527921852,
+          "completed": true
+        },
+        {
+          "ttft": 1.2215185849927366,
+          "time_to_second_token": 0.01897827093489468,
+          "latency": 5.5316939689219,
+          "inter_token_latency_avg": 0.00843478548714122,
+          "chunk_inter_token_latency_avg": 0.023049066224220125,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 118.55666057239915,
+          "e2e_output_tps_per_user": 92.55754256770396,
+          "completed": true
+        },
+        {
+          "ttft": 1.213982854038477,
+          "time_to_second_token": 0.017720089061185718,
+          "latency": 5.389688584022224,
+          "inter_token_latency_avg": 0.008171635479420248,
+          "chunk_inter_token_latency_avg": 0.02332796497197624,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 122.37452374355627,
+          "e2e_output_tps_per_user": 94.99621212213043,
+          "completed": true
+        },
+        {
+          "ttft": 1.2081716759130359,
+          "time_to_second_token": 0.02177582518197596,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.0066114129892226245,
+          "chunk_inter_token_latency_avg": 0.020994135983320967,
+          "input_tokens": 32768,
+          "output_tokens": 182,
+          "output_tps_per_user": 151.25359762430767,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.3119285739958286,
+          "time_to_second_token": 0.020313370041549206,
+          "latency": 4.8990289689973,
+          "inter_token_latency_avg": 0.0070197659393375165,
+          "chunk_inter_token_latency_avg": 0.020734684364170353,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 142.45489217755514,
+          "e2e_output_tps_per_user": 104.51050672288487,
+          "completed": true
+        },
+        {
+          "ttft": 0.9148407790344208,
+          "time_to_second_token": 0.013460942078381777,
+          "latency": 4.861252913950011,
+          "inter_token_latency_avg": 0.007722920029189022,
+          "chunk_inter_token_latency_avg": 0.02335155109417509,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 129.48470218783416,
+          "e2e_output_tps_per_user": 105.32264193264827,
+          "completed": true
+        },
+        {
+          "ttft": 1.2099958129692823,
+          "time_to_second_token": 0.02024669898673892,
+          "latency": 5.36031787795946,
+          "inter_token_latency_avg": 0.008121960988239096,
+          "chunk_inter_token_latency_avg": 0.023186156787654625,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 123.12297503620586,
+          "e2e_output_tps_per_user": 95.51672338411872,
+          "completed": true
+        },
+        {
+          "ttft": 1.2145927760284394,
+          "time_to_second_token": 0.019467717967927456,
+          "latency": 5.264225178165361,
+          "inter_token_latency_avg": 0.007924916638232724,
+          "chunk_inter_token_latency_avg": 0.023140756583639555,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 126.18429261143653,
+          "e2e_output_tps_per_user": 97.26027718640209,
+          "completed": true
+        },
+        {
+          "ttft": 1.2146245839539915,
+          "time_to_second_token": 0.019356438191607594,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008316870190769392,
+          "chunk_inter_token_latency_avg": 0.024396152559590215,
+          "input_tokens": 32768,
+          "output_tokens": 353,
+          "output_tps_per_user": 120.23753852860004,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3749,
+      "wall_time": 25.56879776297137,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 187.39629296317187,
+      "server_utilization": 0.013121161362367406,
+      "server_spec_accept_rate": 0.6424242424242425,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1.9,
+      "max_running_reqs": 2,
+      "effective_concurrency": 1.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.55,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.323,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.59,
+        "cpu_temp_max_c": 75.75,
+        "gpu_util_avg_pct": 99.92,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 36.47,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 68.5,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1160.0,
+        "power_total_max_w": 1177.71,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 22504.67,
+        "pcie_rx_max_mb_s": 74746.0,
+        "pcie_tx_avg_mb_s": 23453.89,
+        "pcie_tx_max_mb_s": 70866.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.777601,
+      "measurement_wall_seconds": 20.001801,
+      "client_output_tokens": 4295,
+      "server_output_tokens": 4295,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 217.16486009639328,
+      "per_request_avg_tps": 54.29121502409832,
+      "ttft_avg": 1.7167683002503158,
+      "ttft_p50": 1.2517880250234157,
+      "ttft_p90": 2.2701463484205306,
+      "ttft_p99": 3.9549784664250893,
+      "time_to_second_token_avg": 0.03188446167713174,
+      "time_to_second_token_p50": 0.03449397184886038,
+      "time_to_second_token_p90": 0.03738179220817983,
+      "time_to_second_token_p99": 0.039646569741889834,
+      "request_latency_avg": 9.444077699189075,
+      "request_latency_p50": 9.303670058376156,
+      "request_latency_p90": 10.359732999047264,
+      "request_latency_p99": 11.179385469323025,
+      "inter_token_latency_avg": 0.014419010797828693,
+      "inter_token_latency_p50": 0.014763854274074246,
+      "inter_token_latency_p90": 0.015734248097034612,
+      "inter_token_latency_p99": 0.01580286538425228,
+      "output_tps_per_user_avg": 70.02365221779164,
+      "output_tps_per_user_p50": 67.73299041267488,
+      "output_tps_per_user_p90": 73.71836215116325,
+      "output_tps_per_user_p99": 90.54101765838234,
+      "e2e_output_tps_per_user_avg": 54.73059414564044,
+      "e2e_output_tps_per_user_p50": 55.03247278818422,
+      "e2e_output_tps_per_user_p90": 60.046589541051475,
+      "e2e_output_tps_per_user_p99": 65.10818536189939,
+      "chunk_inter_token_latency_avg": 0.042334877835426166,
+      "chunk_inter_token_latency_p50": 0.042865508716204204,
+      "chunk_inter_token_latency_p90": 0.04529354933856685,
+      "chunk_inter_token_latency_p99": 0.0453712748584173,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 13,
+      "completed_request_count": 10,
+      "request_samples": [
+        {
+          "ttft": 0.6077568109612912,
+          "time_to_second_token": 0.010776221984997392,
+          "latency": 7.796489109983668,
+          "inter_token_latency_avg": 0.014067969274016393,
+          "chunk_inter_token_latency_avg": 0.04016051563699652,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 71.08346489261992,
+          "e2e_output_tps_per_user": 65.67058489754916,
+          "completed": true
+        },
+        {
+          "ttft": 1.2430980298668146,
+          "time_to_second_token": 0.032302224077284336,
+          "latency": 8.787427563918754,
+          "inter_token_latency_avg": 0.014763854274074246,
+          "chunk_inter_token_latency_avg": 0.042865508716204204,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 67.73299041267488,
+          "e2e_output_tps_per_user": 58.2650606535041,
+          "completed": true
+        },
+        {
+          "ttft": 1.24380440893583,
+          "time_to_second_token": 0.03317554807290435,
+          "latency": 9.216824874980375,
+          "inter_token_latency_avg": 0.015602779776995196,
+          "chunk_inter_token_latency_avg": 0.04530125264798037,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 64.09114364828787,
+          "e2e_output_tps_per_user": 55.55058351926104,
+          "completed": true
+        },
+        {
+          "ttft": 1.2160316940862685,
+          "time_to_second_token": 0.03449397184886038,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.010777328099156248,
+          "chunk_inter_token_latency_avg": 0.03472694609728125,
+          "input_tokens": 32768,
+          "output_tokens": 30,
+          "output_tps_per_user": 92.78737649995917,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2123095341958106,
+          "time_to_second_token": 0.026011183857917786,
+          "latency": 9.824777495115995,
+          "inter_token_latency_avg": 0.014897197575186271,
+          "chunk_inter_token_latency_avg": 0.04349981691954392,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 67.1267193009284,
+          "e2e_output_tps_per_user": 52.1131394837716,
+          "completed": true
+        },
+        {
+          "ttft": 2.2846055519767106,
+          "time_to_second_token": 0.03989882906898856,
+          "latency": 10.258541336050257,
+          "inter_token_latency_avg": 0.015604571006014768,
+          "chunk_inter_token_latency_avg": 0.04333660752213884,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 64.08378670676373,
+          "e2e_output_tps_per_user": 49.909629763906594,
+          "completed": true
+        },
+        {
+          "ttft": 1.214527043979615,
+          "time_to_second_token": 0.03522572107613087,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014891711289040101,
+          "chunk_inter_token_latency_avg": 0.0437039353047916,
+          "input_tokens": 32768,
+          "output_tokens": 406,
+          "output_tps_per_user": 67.15144959437758,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.212038089055568,
+          "time_to_second_token": 0.025989510817453265,
+          "latency": 9.277765536913648,
+          "inter_token_latency_avg": 0.013827255279565714,
+          "chunk_inter_token_latency_avg": 0.04180903815300639,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 72.32093280853985,
+          "e2e_output_tps_per_user": 55.185701553126606,
+          "completed": true
+        },
+        {
+          "ttft": 1.4273314119782299,
+          "time_to_second_token": 0.0377966680098325,
+          "latency": 8.616380715044215,
+          "inter_token_latency_avg": 0.01406858963418001,
+          "chunk_inter_token_latency_avg": 0.0425387532725798,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 71.08033043841677,
+          "e2e_output_tps_per_user": 59.42170116810729,
+          "completed": true
+        },
+        {
+          "ttft": 1.2517880250234157,
+          "time_to_second_token": 0.03299012500792742,
+          "latency": 9.329574579838663,
+          "inter_token_latency_avg": 0.015807801477133558,
+          "chunk_inter_token_latency_avg": 0.045380823341658695,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 63.25990375363259,
+          "e2e_output_tps_per_user": 54.879244023241846,
+          "completed": true
+        },
+        {
+          "ttft": 4.1827565911225975,
+          "time_to_second_token": 0.03485451405867934,
+          "latency": 11.27045796602033,
+          "inter_token_latency_avg": 0.013870257093733334,
+          "chunk_inter_token_latency_avg": 0.041939061389927416,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 72.09671697086321,
+          "e2e_output_tps_per_user": 45.42850002578825,
+          "completed": true
+        },
+        {
+          "ttft": 2.005770788062364,
+          "time_to_second_token": 0.03572228900156915,
+          "latency": 10.062537814024836,
+          "inter_token_latency_avg": 0.015766667369789572,
+          "chunk_inter_token_latency_avg": 0.04526273610091276,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 63.424944317408176,
+          "e2e_output_tps_per_user": 50.88179636814792,
+          "completed": true
+        },
+        {
+          "ttft": 1.2161699240095913,
+          "time_to_second_token": 0.03526119492016733,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013501158222887602,
+          "chunk_inter_token_latency_avg": 0.03982841675751843,
+          "input_tokens": 32768,
+          "output_tokens": 355,
+          "output_tps_per_user": 74.0677194868191,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4295,
+      "wall_time": 29.076672724913806,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 214.67688879215112,
+      "server_utilization": 0.009771077610273626,
+      "server_spec_accept_rate": 0.625,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4.0,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4.0,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 8.573,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.242,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.54,
+        "cpu_temp_max_c": 76.75,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 31.81,
+        "mem_util_max_pct": 46.0,
+        "temp_avg_c": 68.78,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1160.18,
+        "power_total_max_w": 1172.54,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 19024.67,
+        "pcie_rx_max_mb_s": 74971.0,
+        "pcie_tx_avg_mb_s": 20175.22,
+        "pcie_tx_max_mb_s": 67212.0
+      }
+    }
+  ],
+  "summary_table": {
+    "0": {
+      "1": 181.37569616773123,
+      "2": 257.2872334123046,
+      "4": 308.9773476463454
+    },
+    "8192": {
+      "1": 161.46688548018994,
+      "2": 179.48013878635928,
+      "4": 217.85922059414168
+    },
+    "32768": {
+      "1": 162.9385363574482,
+      "2": 187.47192708215627,
+      "4": 217.16486009639328
+    }
+  },
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": false,
+      "mode": "skipped",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": true,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap512.log

@@ -0,0 +1,126 @@
+
+New version available: v0.6.2 (current: v0.4.29)
+Upgrade and restart? [Y/n]: Skipping update.
+
+╭──────────────────────────── NVIDIA P2P Override ─────────────────────────────╮
+│ Effective: yes                                                               │
+│ Configured file: yes (/etc/modprobe.d/nvidia-p2p-override.conf)              │
+│ Runtime: ForceP2P=0x11; RMForceP2PType=1; RMPcieP2PType=2;                   │
+│ GrdmaPciTopoCheckOverride=1; EnableResizableBar=1; DmaRemapPeerMmio=1        │
+╰──────────────────────────────────────────────────────────────────────────────╯
+╭─────────────────────────────── Configuration ────────────────────────────────╮
+│ LLM Inference Benchmark                                                      │
+│ Model: glm53-flash-trellismx-p8-k45 @ 127.0.0.1:8001                         │
+│ Decode concurrency: [1, 2, 4]                                                │
+│ Decode contexts: ['0', '8k', '32k']                                          │
+│ Duration: 20.0s per decode test | Max tokens: 512                            │
+│ Pre-decode warmup: C=1 max-runnable context for 3s                           │
+│ Prefill: skipped | Sustained decode: 9 cells                                 │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Engine: vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f  
+Models: ['glm53-flash-trellismx-p8-k45']
+KV cache budget (vLLM metrics): 29,351,936 tokens (3583 blocks × 2048; local 
+7,337,984 × CP 4; CP source: local process)
+Model context length: 1,000,000 tokens
+Prefill tests: skipped
+Calibrating padding text (run=ftbfeppyynkm, up to 32k)...
+  8k: 50,558 chars (8,192 prompt tokens via /tokenize)
+  32k: 205,152 chars (32,768 prompt tokens via /tokenize)
+  Token targeting: /tokenize exact
+Done.
+
+
+
+llm-decode-bench v0.4.29
+╭────────────────────────────────── Phase 2 ───────────────────────────────────╮
+│ Sustained Decode                                                             │
+│ Steady-state decode throughput after the engine has admitted the requested   │
+│ concurrency and passed warmup. Use this as the main tuning/regression signal │
+│ for kernels, NCCL, DCP, MTP, and scheduler changes.                          │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Aggregate tok/s  + TTFT/ITL                                         
+╭────────────┬─────────────┬──────────────────┬────────────────────╮
+│ ctx \ conc │           1 │                2 │                  4 │
+├────────────┼─────────────┼──────────────────┼────────────────────┤
+│ 0          │  181.4 84/5 │      257.3 123/8 │ 309.0 (4/4) 181/12 │
+│ 8k         │ 161.5 625/5 │ 179.5 (2/2) 1k/8 │     ∅ (4/4)* 1k/15 │
+│ 32k        │ 162.9 628/5 │ 187.5 (2/2) 1k/8 │        217.2 1k/15 │
+╰────────────┴─────────────┴──────────────────┴────────────────────╯
+Sustained Decode: aggregate tok/s uses OpenAI stream usage by default 
+(continuous completion_tokens when the server supports it). Prometheus is kept 
+as validation/scheduler data.
+Aggregate source(s): openai_continuous_usage
+∅ = skipped/hidden because the cell does not fit in KV cache; exact deficit is 
+kept in JSON timeout_reason
+(X/Y) = avg running / requested concurrency from Prometheus; * = 
+capacity-limited or warmup timed out
+Per-Request tok/s                               
+╭────────────┬───────┬────────────┬────────────╮
+│ ctx \ conc │     1 │          2 │          4 │
+├────────────┼───────┼──���─────────┼────────────┤
+│ 0          │ 181.4 │      128.6 │ 77.2 (4/4) │
+│ 8k         │ 161.5 │ 89.7 (2/2) │   ∅ (4/4)* │
+│ 32k        │ 162.9 │ 93.7 (2/2) │       54.3 │
+╰────────────┴───────┴────────────┴────────────╯
+Client request latency: p50 / p90 ms               
+╭────────────┬───────────┬───────────┬────────────╮
+│ ctx \ conc │         1 │         2 │          4 │
+├────────────┼───────────┼───────────┼────────────┤
+│ 0          │ 2.8k/2.9k │ 3.9k/4.2k │  6.4k/6.9k │
+│ 8k         │ 3.2k/3.3k │ 5.4k/5.9k │ 9.3k/11.3k │
+│ 32k        │ 3.2k/3.2k │ 5.4k/5.6k │ 9.3k/10.4k │
+╰────────────┴───────────┴───────────┴────────────╯
+Aggregate cells show dim detail as TTFT ms / ITL ms for the same ctx/conc 
+coordinate. ITL is computed from observed generated tokens, including streams 
+stopped at the measurement boundary; a missing ITL means no stream produced at 
+least two measured output tokens. Per-request tok/s and request latency are 
+shown in separate per-cell matrices. Completion/sample counts and full 
+request-level distributions remain in JSON under request_samples.
+Sustained mode: client latency metrics explain request UX variance; aggregate 
+tok/s remains the primary throughput signal. 
+ITL=(last_token_time-first_token_time)/(output_tokens-1), user tok/s=1/ITL.
+Hardware Summary                                                                
+╭───┬─┬───────┬───────────┬───────┬─────────┬─────┬──────┬─────┬───────────────╮
+│ … │ │ mode  │ GPU avg/… │ Mem … │ W avg/… │ T … │ CPU… │ VR… │ PCIe rx/tx a… │
+├───┼─┼───────┼───────────┼───────┼─────────┼─────┼──────┼─────┼───────────────┤
+│ 0 │ │ sust… │    99/99% │   44% │ 1151/1… │ 82C │  76C │ 98… │     8369/8254 │
+│ … │ │ sust… │   99/100% │   40% │ 1153/1… │ 83C │  76C │ 98… │   12913/11281 │
+│ … │ │ sust… │   98/100% │   38% │ 1150/1… │ 84C │  76C │ 98… │     8327/8133 │
+│ 0 │ │ sust… │  100/100% │   40% │ 1173/1… │ 84C │  76C │ 98… │   11187/11117 │
+│ 0 │ │ sust… │  100/100% │   35% │ 1170/1… │ 84C │  77C │ 98… │     7813/8054 │
+│ … │ │ sust… │  100/100% │   36% │ 1162/1… │ 84C │  76C │ 98… │   29259/28011 │
+│ … │ │ sust… │  100/100% │   34% │ 1159/1… │ 84C │  76C │ 98… │   27724/27474 │
+│ … │ │ sust… │  100/100% │   36% │ 1160/1… │ 84C │  76C │ 98… │   22505/23454 │
+│ … │ │ sust… │  100/100% │   32% │ 1160/1… │ 84C │  77C │ 98… │   19025/20175 │
+╰───┴─┴───────┴───────────┴───────┴─────────┴─────┴──────┴─────┴───────────────╯
+╭───────────────────────── Whole-run GPU Power ─────────────────────────╮
+│ avg 1,098 W | max 1,178 W | limit 1,200 W | over 4m 30s | 113 samples │
+╰───────────────────────────────────────────────────────────────────────╯
+Hardware summary is sampled from nvidia-smi during the measured part of each 
+cell. Whole-run GPU power is the sampled sum of GPU power draw across the 
+complete benchmark run, not wall-outlet system power. PCIe rx/tx is MB/s and is 
+a coarse live diagnostic, not a per-kernel NCCL profiler.
+
+╭────────────────────────────────── Phase 3 ───────────────────────────────────╮
+│ Burst / E2E Decode                                                           │
+│ Not run. Re-run with --run-burst to append a finite client-facing request    │
+│ burst after Sustained Decode. This is intentionally disabled by default      │
+│ because it adds another full decode matrix.                                  │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+╭────────────────────────────── Primary Summary ───────────────────────────────╮
+│ Primary matrices repeated last so the important numbers are visible without  │
+│ scrolling back through diagnostics.                                          │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Aggregate decode tok/s                            
+╭────────────┬───────┬─────────────┬─────────────╮
+│ ctx \ conc │     1 │           2 │           4 │
+├────────────┼───────┼─────────────┼─────────────┤
+│ 0          │ 181.4 │       257.3 │ 309.0 (4/4) │
+│ 8k         │ 161.5 │ 179.5 (2/2) │    ∅ (4/4)* │
+│ 32k        │ 162.9 │ 187.5 (2/2) │       217.2 │
+╰────────────┴───────┴─────────────┴─────────────╯
+
+Results saved to 
+<campaign>/candidate-speed-wi
+ndow-01/results-01/decode-warp-quant/rep-1/decode-cap512.json

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap8192-command.json

@@ -0,0 +1,27 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "8192",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap8192.json",
+  "--contexts",
+  "0,8k,32k",
+  "--concurrency",
+  "1,2,4",
+  "--skip-prefill",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap8192-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "b236cf123d887ede3ee9c5ba2a6ef16597fbf3e99450397dcb46488a24dd99b3"
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap8192.json

@@ -0,0 +1,1394 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T02:22:18.084779",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "skipped",
+    "standalone_prefill": false,
+    "prefill_only": false,
+    "skip_prefill": true,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 32768,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 29351936,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0,
+      8192,
+      32768
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0,8k,32k",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": false,
+      "prefill_only": false,
+      "skip_prefill": true,
+      "prefill_contexts": "8k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 114,
+    "duration_seconds": 272.443,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 10.97,
+    "cpu_temp_max_c": 76.25,
+    "gpu_util_avg_pct": 90.62,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 35.29,
+    "mem_util_max_pct": 57.0,
+    "temp_avg_c": 66.66,
+    "temp_max_c": 84.0,
+    "power_total_avg_w": 1094.77,
+    "power_total_max_w": 1178.23,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 384774.21,
+    "vram_used_max_mb": 384778.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.27,
+    "vram_used_max_pct": 98.27,
+    "pcie_rx_avg_mb_s": 12350.85,
+    "pcie_rx_max_mb_s": 60894.0,
+    "pcie_tx_avg_mb_s": 12097.24,
+    "pcie_tx_max_mb_s": 58766.0
+  },
+  "event_log": [
+    "02:17:43 benchmark start engine=vllm",
+    "02:17:43 startup server=http://127.0.0.1:8001 model=glm53-flash-trellismx-p8-k45",
+    "02:17:43 startup decode concurrency=1,2,4 contexts=0,8k,32k",
+    "02:17:43 startup NVIDIA P2P override: enabled: runtime NVIDIA P2P override matches expected RegistryDwords",
+    "02:17:43 startup engine vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f models=['glm53-flash-trellismx-p8-k45']",
+    "02:17:43 startup KV cache budget from vLLM metrics: 29,351,936 tokens (3583 blocks x 2048; local 7,337,984 \u00d7 CP 4; CP source: local process)",
+    "02:17:43 startup model context length: 1,000,000 tokens",
+    "02:17:43 startup prefill tests: skipped",
+    "02:17:43 startup calibrating padding text run=rumxqbvzafby up_to=32k",
+    "02:17:43 startup context 8k: 50,540 chars (8,192 prompt tokens via /tokenize)",
+    "02:17:43 startup context 32k: 205,139 chars (32,768 prompt tokens via /tokenize)",
+    "02:17:43 startup token targeting: /tokenize exact",
+    "02:17:43 startup startup preparation done",
+    "02:17:43 hardware monitor interval=2s",
+    "02:17:43 decode warmup start",
+    "02:17:43 decode warmup start C=1 ctx=32k 3s",
+    "02:17:43 cell start C=1 ctx=32k",
+    "02:17:51 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:17:54 cell done C=1 ctx=32k 187.9 tok/s",
+    "02:17:54 decode warmup done C=1 ctx=32k",
+    "02:17:56 cell start C=1 ctx=0",
+    "02:18:02 ready C=1 ctx=0 running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:18:22 cell done C=1 ctx=0 185.7 tok/s",
+    "02:18:24 cell start C=1 ctx=8k",
+    "02:18:30 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:18:50 cell done C=1 ctx=8k 166.9 tok/s",
+    "02:18:52 cell start C=1 ctx=32k",
+    "02:19:01 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:19:21 cell done C=1 ctx=32k 173.8 tok/s",
+    "02:19:23 cell start C=2 ctx=0",
+    "02:19:28 ready C=2 ctx=0 running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:19:49 cell done C=2 ctx=0 246.2 tok/s",
+    "02:19:51 cell start C=4 ctx=0",
+    "02:19:56 ready C=4 ctx=0 running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:20:16 cell done C=4 ctx=0 303.4 tok/s",
+    "02:20:18 cell start C=2 ctx=8k",
+    "02:20:24 ready C=2 ctx=8k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:20:44 cell done C=2 ctx=8k 236.9 tok/s",
+    "02:20:46 cell start C=4 ctx=8k",
+    "02:20:57 ready C=4 ctx=8k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:21:17 cell done C=4 ctx=8k 291.1 tok/s",
+    "02:21:19 cell start C=2 ctx=32k",
+    "02:21:25 ready C=2 ctx=32k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:21:45 cell done C=2 ctx=32k 231.8 tok/s",
+    "02:21:47 cell start C=4 ctx=32k",
+    "02:21:56 ready C=4 ctx=32k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:22:16 cell done C=4 ctx=32k 303.9 tok/s"
+  ],
+  "prefill": {},
+  "results": [
+    {
+      "concurrency": 1,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.990511,
+      "measurement_wall_seconds": 20.000618,
+      "client_output_tokens": 3712,
+      "server_output_tokens": 3712,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 185.68810232786203,
+      "per_request_avg_tps": 185.68810232786203,
+      "ttft_avg": 0.07161088101565838,
+      "ttft_p50": 0.07161088101565838,
+      "ttft_p90": 0.07161088101565838,
+      "ttft_p99": 0.07161088101565838,
+      "time_to_second_token_avg": 0.01281460584141314,
+      "time_to_second_token_p50": 0.01281460584141314,
+      "time_to_second_token_p90": 0.01281460584141314,
+      "time_to_second_token_p99": 0.01281460584141314,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005329198802730078,
+      "inter_token_latency_p50": 0.005329198802730078,
+      "inter_token_latency_p90": 0.005329198802730078,
+      "inter_token_latency_p99": 0.005329198802730078,
+      "output_tps_per_user_avg": 187.64546736138146,
+      "output_tps_per_user_p50": 187.64546736138146,
+      "output_tps_per_user_p90": 187.64546736138146,
+      "output_tps_per_user_p99": 187.64546736138146,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.014062018498253509,
+      "chunk_inter_token_latency_p50": 0.014062018498253509,
+      "chunk_inter_token_latency_p90": 0.014062018498253509,
+      "chunk_inter_token_latency_p99": 0.014062018498253509,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 4777.0,
+      "output_seq_len_p50": 4777.0,
+      "output_seq_len_p90": 4777.0,
+      "output_seq_len_p99": 4777.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.07161088101565838,
+          "time_to_second_token": 0.01281460584141314,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005329198802730078,
+          "chunk_inter_token_latency_avg": 0.014062018498253509,
+          "input_tokens": 78,
+          "output_tokens": 4777,
+          "output_tps_per_user": 187.64546736138146,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3712,
+      "wall_time": 25.54038013308309,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 185.54062187903287,
+      "server_utilization": 0.005862646566164198,
+      "server_spec_accept_rate": 0.5648148148148148,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.533,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.316,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.53,
+        "cpu_temp_max_c": 75.25,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 43.92,
+        "mem_util_max_pct": 56.0,
+        "temp_avg_c": 65.25,
+        "temp_max_c": 80.0,
+        "power_total_avg_w": 1149.49,
+        "power_total_max_w": 1153.43,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8501.89,
+        "pcie_rx_max_mb_s": 8783.0,
+        "pcie_tx_avg_mb_s": 8302.22,
+        "pcie_tx_max_mb_s": 8578.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 20.000286,
+      "measurement_wall_seconds": 20.000336,
+      "client_output_tokens": 3338,
+      "server_output_tokens": 3338,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 166.89761277085296,
+      "per_request_avg_tps": 166.89761277085296,
+      "ttft_avg": 0.5802665068767965,
+      "ttft_p50": 0.5802665068767965,
+      "ttft_p90": 0.5802665068767965,
+      "ttft_p99": 0.5802665068767965,
+      "time_to_second_token_avg": 0.016474336152896285,
+      "time_to_second_token_p50": 0.016474336152896285,
+      "time_to_second_token_p90": 0.016474336152896285,
+      "time_to_second_token_p99": 0.016474336152896285,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.0059050409250968015,
+      "inter_token_latency_p50": 0.0059050409250968015,
+      "inter_token_latency_p90": 0.0059050409250968015,
+      "inter_token_latency_p99": 0.0059050409250968015,
+      "output_tps_per_user_avg": 169.34683648845447,
+      "output_tps_per_user_p50": 169.34683648845447,
+      "output_tps_per_user_p90": 169.34683648845447,
+      "output_tps_per_user_p99": 169.34683648845447,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.014231043370286765,
+      "chunk_inter_token_latency_p50": 0.014231043370286765,
+      "chunk_inter_token_latency_p90": 0.014231043370286765,
+      "chunk_inter_token_latency_p99": 0.014231043370286765,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 4057.0,
+      "output_seq_len_p50": 4057.0,
+      "output_seq_len_p90": 4057.0,
+      "output_seq_len_p99": 4057.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5802665068767965,
+          "time_to_second_token": 0.016474336152896285,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.0059050409250968015,
+          "chunk_inter_token_latency_avg": 0.014231043370286765,
+          "input_tokens": 8192,
+          "output_tokens": 4057,
+          "output_tps_per_user": 169.34683648845447,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3338,
+      "wall_time": 26.075741773936898,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 166.8524947323479,
+      "server_utilization": 0.006141820212172022,
+      "server_spec_accept_rate": 0.3286384976525822,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 6.06,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.922,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.59,
+        "cpu_temp_max_c": 75.0,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 44.31,
+        "mem_util_max_pct": 56.0,
+        "temp_avg_c": 65.97,
+        "temp_max_c": 81.0,
+        "power_total_avg_w": 1152.41,
+        "power_total_max_w": 1153.89,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8303.25,
+        "pcie_rx_max_mb_s": 8596.0,
+        "pcie_tx_avg_mb_s": 8386.88,
+        "pcie_tx_max_mb_s": 8513.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.989714,
+      "measurement_wall_seconds": 20.000821,
+      "client_output_tokens": 3475,
+      "server_output_tokens": 3475,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 173.8394059270179,
+      "per_request_avg_tps": 173.8394059270179,
+      "ttft_avg": 0.5945898578502238,
+      "ttft_p50": 0.5945898578502238,
+      "ttft_p90": 0.5945898578502238,
+      "ttft_p99": 0.5945898578502238,
+      "time_to_second_token_avg": 0.011255914112553,
+      "time_to_second_token_p50": 0.011255914112553,
+      "time_to_second_token_p90": 0.011255914112553,
+      "time_to_second_token_p99": 0.011255914112553,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005599262254130903,
+      "inter_token_latency_p50": 0.005599262254130903,
+      "inter_token_latency_p90": 0.005599262254130903,
+      "inter_token_latency_p99": 0.005599262254130903,
+      "output_tps_per_user_avg": 178.59495673063742,
+      "output_tps_per_user_p50": 178.59495673063742,
+      "output_tps_per_user_p90": 178.59495673063742,
+      "output_tps_per_user_p99": 178.59495673063742,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.014177278953883576,
+      "chunk_inter_token_latency_p50": 0.014177278953883576,
+      "chunk_inter_token_latency_p90": 0.014177278953883576,
+      "chunk_inter_token_latency_p99": 0.014177278953883576,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 4275.0,
+      "output_seq_len_p50": 4275.0,
+      "output_seq_len_p90": 4275.0,
+      "output_seq_len_p99": 4275.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5945898578502238,
+          "time_to_second_token": 0.011255914112553,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005599262254130903,
+          "chunk_inter_token_latency_avg": 0.014177278953883576,
+          "input_tokens": 32768,
+          "output_tokens": 4275,
+          "output_tps_per_user": 178.59495673063742,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3475,
+      "wall_time": 29.124992428114638,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 173.70034429070287,
+      "server_utilization": 0.006979341150195384,
+      "server_spec_accept_rate": 0.38967136150234744,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 9.119,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.869,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.56,
+        "cpu_temp_max_c": 74.88,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 43.78,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 66.56,
+        "temp_max_c": 82.0,
+        "power_total_avg_w": 1153.2,
+        "power_total_max_w": 1153.93,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8358.0,
+        "pcie_rx_max_mb_s": 8664.0,
+        "pcie_tx_avg_mb_s": 8305.5,
+        "pcie_tx_max_mb_s": 8516.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.998344,
+      "measurement_wall_seconds": 20.000475,
+      "client_output_tokens": 4924,
+      "server_output_tokens": 4924,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 246.22038564672286,
+      "per_request_avg_tps": 123.11019282336143,
+      "ttft_avg": 0.11326407559681684,
+      "ttft_p50": 0.11326407559681684,
+      "ttft_p90": 0.14774754794780165,
+      "ttft_p99": 0.15550632922677324,
+      "time_to_second_token_avg": 0.015398422023281455,
+      "time_to_second_token_p50": 0.015398422023281455,
+      "time_to_second_token_p90": 0.017018200410529972,
+      "time_to_second_token_p99": 0.01738265054766089,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.007996202209365196,
+      "inter_token_latency_p50": 0.007996202209365196,
+      "inter_token_latency_p90": 0.008062418200969368,
+      "inter_token_latency_p99": 0.008077316799080308,
+      "output_tps_per_user_avg": 125.07276978039832,
+      "output_tps_per_user_p50": 125.07276978039832,
+      "output_tps_per_user_p90": 126.10848864503505,
+      "output_tps_per_user_p99": 126.34152538957832,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.02072697419690292,
+      "chunk_inter_token_latency_p50": 0.02072697419690292,
+      "chunk_inter_token_latency_p90": 0.020734919510937286,
+      "chunk_inter_token_latency_p99": 0.02073670720659502,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 3180.5,
+      "output_seq_len_p50": 3180.5,
+      "output_seq_len_p90": 3202.5,
+      "output_seq_len_p99": 3207.45,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.07015973515808582,
+          "time_to_second_token": 0.01337369903922081,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008078972198870412,
+          "chunk_inter_token_latency_avg": 0.020736905839445877,
+          "input_tokens": 78,
+          "output_tokens": 3153,
+          "output_tps_per_user": 123.77812119960238,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.15636841603554785,
+          "time_to_second_token": 0.0174231450073421,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007913432219859979,
+          "chunk_inter_token_latency_avg": 0.02071704255435996,
+          "input_tokens": 78,
+          "output_tokens": 3208,
+          "output_tps_per_user": 126.36741836119424,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4924,
+      "wall_time": 25.577398049877957,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 246.13329155338783,
+      "server_utilization": 0.005862646566164198,
+      "server_spec_accept_rate": 0.47959183673469385,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.537,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.327,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.58,
+        "cpu_temp_max_c": 75.75,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.89,
+        "mem_util_max_pct": 50.0,
+        "temp_avg_c": 67.36,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1176.5,
+        "power_total_max_w": 1178.11,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11309.78,
+        "pcie_rx_max_mb_s": 11397.0,
+        "pcie_tx_avg_mb_s": 11179.56,
+        "pcie_tx_max_mb_s": 11268.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.993715,
+      "measurement_wall_seconds": 20.000812,
+      "client_output_tokens": 6067,
+      "server_output_tokens": 6067,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 303.4453614548168,
+      "per_request_avg_tps": 75.8613403637042,
+      "ttft_avg": 0.15646358660887927,
+      "ttft_p50": 0.18395527265965939,
+      "ttft_p90": 0.18397697466425597,
+      "ttft_p99": 0.1839802248729393,
+      "time_to_second_token_avg": 0.025640497857239097,
+      "time_to_second_token_p50": 0.029807523358613253,
+      "time_to_second_token_p90": 0.029867575969547033,
+      "time_to_second_token_p99": 0.029868205869570376,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.012890471746641244,
+      "inter_token_latency_p50": 0.01291707436819987,
+      "inter_token_latency_p90": 0.013106307107593229,
+      "inter_token_latency_p99": 0.013131838856624441,
+      "output_tps_per_user_avg": 77.59775588315449,
+      "output_tps_per_user_p50": 77.42393606470978,
+      "output_tps_per_user_p90": 79.03458812531339,
+      "output_tps_per_user_p99": 79.37137995360604,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.033961205786559916,
+      "chunk_inter_token_latency_p50": 0.03394124054211881,
+      "chunk_inter_token_latency_p90": 0.03400282599744794,
+      "chunk_inter_token_latency_p99": 0.034024420723019345,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 1970.25,
+      "output_seq_len_p50": 1968.0,
+      "output_seq_len_p90": 2007.1,
+      "output_seq_len_p99": 2013.31,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.07396321510896087,
+          "time_to_second_token": 0.013078668853268027,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012794035052220763,
+          "chunk_inter_token_latency_avg": 0.03394683967189242,
+          "input_tokens": 78,
+          "output_tokens": 1991,
+          "output_tps_per_user": 78.16142412603614,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18394199712201953,
+          "time_to_second_token": 0.029868275858461857,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013040113684178978,
+          "chunk_inter_token_latency_avg": 0.034026820136971725,
+          "input_tokens": 78,
+          "output_tokens": 1945,
+          "output_tps_per_user": 76.68644800338343,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18398058600723743,
+          "time_to_second_token": 0.029865942895412445,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013134675717627909,
+          "chunk_inter_token_latency_avg": 0.0339356414123452,
+          "input_tokens": 78,
+          "output_tokens": 1931,
+          "output_tps_per_user": 76.13435013533761,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18396854819729924,
+          "time_to_second_token": 0.02974910382181406,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012593062532537325,
+          "chunk_inter_token_latency_avg": 0.033935521925030306,
+          "input_tokens": 78,
+          "output_tokens": 2014,
+          "output_tps_per_user": 79.40880126786078,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 6067,
+      "wall_time": 25.569467527093366,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 303.24736236685254,
+      "server_utilization": 0.02345058626465657,
+      "server_spec_accept_rate": 0.5431034482758621,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.541,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.826,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.47,
+        "cpu_temp_max_c": 75.88,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 35.31,
+        "mem_util_max_pct": 43.0,
+        "temp_avg_c": 68.16,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1172.73,
+        "power_total_max_w": 1173.45,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 7893.62,
+        "pcie_rx_max_mb_s": 8279.0,
+        "pcie_tx_avg_mb_s": 7755.88,
+        "pcie_tx_max_mb_s": 8153.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.988454,
+      "measurement_wall_seconds": 20.00057,
+      "client_output_tokens": 4735,
+      "server_output_tokens": 4735,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 236.8867590379722,
+      "per_request_avg_tps": 118.4433795189861,
+      "ttft_avg": 0.9413391120033339,
+      "ttft_p50": 0.9413391120033339,
+      "ttft_p90": 1.2234342327574268,
+      "ttft_p99": 1.2869056349270978,
+      "time_to_second_token_avg": 0.021329568000510335,
+      "time_to_second_token_p50": 0.021329568000510335,
+      "time_to_second_token_p90": 0.02503214799799025,
+      "time_to_second_token_p99": 0.02586522849742323,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.008409807194832745,
+      "inter_token_latency_p50": 0.008409807194832745,
+      "inter_token_latency_p90": 0.00844567690517398,
+      "inter_token_latency_p99": 0.00845374759000076,
+      "output_tps_per_user_avg": 118.91217038040975,
+      "output_tps_per_user_p50": 118.91217038040975,
+      "output_tps_per_user_p90": 119.41935740726734,
+      "output_tps_per_user_p99": 119.5334744883103,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.02114018666727937,
+      "chunk_inter_token_latency_p50": 0.02114018666727937,
+      "chunk_inter_token_latency_p90": 0.02137045316589387,
+      "chunk_inter_token_latency_p99": 0.021422263128082132,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 2805.0,
+      "output_seq_len_p50": 2805.0,
+      "output_seq_len_p90": 2826.6,
+      "output_seq_len_p99": 2831.46,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5887202110607177,
+          "time_to_second_token": 0.01670134300366044,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.00845464433275929,
+          "chunk_inter_token_latency_avg": 0.021428019790547495,
+          "input_tokens": 8192,
+          "output_tokens": 2832,
+          "output_tps_per_user": 118.27818659683774,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.29395801294595,
+          "time_to_second_token": 0.02595779299736023,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008364970056906203,
+          "chunk_inter_token_latency_avg": 0.020852353544011243,
+          "input_tokens": 8192,
+          "output_tokens": 2778,
+          "output_tps_per_user": 119.54615416398174,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4735,
+      "wall_time": 25.56432215613313,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 236.6879632349728,
+      "server_utilization": 0.012283640424343933,
+      "server_spec_accept_rate": 0.4791666666666667,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.554,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.882,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.54,
+        "cpu_temp_max_c": 75.62,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.84,
+        "mem_util_max_pct": 50.0,
+        "temp_avg_c": 67.75,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1177.72,
+        "power_total_max_w": 1178.14,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11234.38,
+        "pcie_rx_max_mb_s": 11286.0,
+        "pcie_tx_avg_mb_s": 11126.88,
+        "pcie_tx_max_mb_s": 11190.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.988009,
+      "measurement_wall_seconds": 20.000154,
+      "client_output_tokens": 5819,
+      "server_output_tokens": 5819,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 291.1245424388467,
+      "per_request_avg_tps": 72.78113560971167,
+      "ttft_avg": 3.0831757867126726,
+      "ttft_p50": 2.1997361014364287,
+      "ttft_p90": 5.800122947827914,
+      "ttft_p99": 7.188826314958277,
+      "time_to_second_token_avg": 0.03507792699383572,
+      "time_to_second_token_p50": 0.0406363804358989,
+      "time_to_second_token_p90": 0.0417290014680475,
+      "time_to_second_token_p99": 0.041733411899767814,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.014156019616279113,
+      "inter_token_latency_p50": 0.014055135648646407,
+      "inter_token_latency_p90": 0.01461799139621679,
+      "inter_token_latency_p99": 0.014785406939632685,
+      "output_tps_per_user_avg": 70.69962462194486,
+      "output_tps_per_user_p50": 71.15434738255996,
+      "output_tps_per_user_p90": 72.6003158823885,
+      "output_tps_per_user_p99": 72.90651063203018,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.03618710908102364,
+      "chunk_inter_token_latency_p50": 0.03679464568694915,
+      "chunk_inter_token_latency_p90": 0.03703470265231493,
+      "chunk_inter_token_latency_p99": 0.03705366284251491,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 1940.0,
+      "output_seq_len_p50": 2013.5,
+      "output_seq_len_p90": 2034.4,
+      "output_seq_len_p99": 2037.64,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5901042548939586,
+          "time_to_second_token": 0.01730504515580833,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014804008666678895,
+          "chunk_inter_token_latency_avg": 0.03705576953031491,
+          "input_tokens": 8192,
+          "output_tokens": 2026,
+          "output_tps_per_user": 67.54927145178024,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.1997808848973364,
+          "time_to_second_token": 0.04173390194773674,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.01418395109847188,
+          "chunk_inter_token_latency_avg": 0.03660374477025001,
+          "input_tokens": 8192,
+          "output_tokens": 2001,
+          "output_tps_per_user": 70.50221712254323,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.199691317975521,
+          "time_to_second_token": 0.04171756701543927,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013926320198820936,
+          "chunk_inter_token_latency_avg": 0.0369855466036483,
+          "input_tokens": 8192,
+          "output_tokens": 2038,
+          "output_tps_per_user": 71.80647764257671,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 7.343126689083874,
+          "time_to_second_token": 0.03955519385635853,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013709798501144739,
+          "chunk_inter_token_latency_avg": 0.03410337541988133,
+          "input_tokens": 8192,
+          "output_tokens": 1695,
+          "output_tps_per_user": 72.94053227087926,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5819,
+      "wall_time": 31.621210746001452,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 290.8514927893891,
+      "server_utilization": 0.024567280848687867,
+      "server_spec_accept_rate": 0.5,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 11.597,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.872,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.55,
+        "cpu_temp_max_c": 75.88,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 34.75,
+        "mem_util_max_pct": 43.0,
+        "temp_avg_c": 68.12,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1173.04,
+        "power_total_max_w": 1173.65,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8010.0,
+        "pcie_rx_max_mb_s": 8332.0,
+        "pcie_tx_avg_mb_s": 7806.5,
+        "pcie_tx_max_mb_s": 8129.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.989635,
+      "measurement_wall_seconds": 20.000757,
+      "client_output_tokens": 4633,
+      "server_output_tokens": 4633,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 231.7701102106262,
+      "per_request_avg_tps": 115.8850551053131,
+      "ttft_avg": 0.9621059750206769,
+      "ttft_p50": 0.9621059750206769,
+      "ttft_p90": 1.2434888477437198,
+      "ttft_p99": 1.3067999941064046,
+      "time_to_second_token_avg": 0.01357436552643776,
+      "time_to_second_token_p50": 0.01357436552643776,
+      "time_to_second_token_p90": 0.01572811957448721,
+      "time_to_second_token_p99": 0.016212714235298336,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.008537162788289372,
+      "inter_token_latency_p50": 0.008537162788289372,
+      "inter_token_latency_p90": 0.00858358805811946,
+      "inter_token_latency_p99": 0.00859403374383123,
+      "output_tps_per_user_avg": 117.14034665810209,
+      "output_tps_per_user_p50": 117.14034665810209,
+      "output_tps_per_user_p90": 117.77735831366668,
+      "output_tps_per_user_p99": 117.92068593616872,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.021224602006348896,
+      "chunk_inter_token_latency_p50": 0.021224602006348896,
+      "chunk_inter_token_latency_p90": 0.021463160367322105,
+      "chunk_inter_token_latency_p99": 0.021516835998541078,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 2760.5,
+      "output_seq_len_p50": 2760.5,
+      "output_seq_len_p90": 2778.5,
+      "output_seq_len_p99": 2782.55,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.6103773841168731,
+          "time_to_second_token": 0.010882172966375947,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008595194375576983,
+          "chunk_inter_token_latency_avg": 0.021522799957565408,
+          "input_tokens": 32768,
+          "output_tokens": 2783,
+          "output_tps_per_user": 116.34408208864636,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.3138345659244806,
+          "time_to_second_token": 0.016266558086499572,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.00847913120100176,
+          "chunk_inter_token_latency_avg": 0.020926404055132387,
+          "input_tokens": 32768,
+          "output_tokens": 2738,
+          "output_tps_per_user": 117.93661122755783,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4633,
+      "wall_time": 25.564056790899485,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 231.5835038411289,
+      "server_utilization": 0.013121161362367406,
+      "server_spec_accept_rate": 0.5,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.553,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.876,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.51,
+        "cpu_temp_max_c": 75.25,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.84,
+        "mem_util_max_pct": 51.0,
+        "temp_avg_c": 67.97,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1177.56,
+        "power_total_max_w": 1178.23,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11116.0,
+        "pcie_rx_max_mb_s": 11361.0,
+        "pcie_tx_avg_mb_s": 11002.38,
+        "pcie_tx_max_mb_s": 11184.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.997266,
+      "measurement_wall_seconds": 20.000362,
+      "client_output_tokens": 6078,
+      "server_output_tokens": 6078,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 303.94154153181165,
+      "per_request_avg_tps": 75.98538538295291,
+      "ttft_avg": 2.3015168351703323,
+      "ttft_p50": 2.210984098375775,
+      "ttft_p90": 3.5852746270596985,
+      "ttft_p99": 4.115192952086217,
+      "time_to_second_token_avg": 0.023911484284326434,
+      "time_to_second_token_p50": 0.024445773102343082,
+      "time_to_second_token_p90": 0.03160055840853602,
+      "time_to_second_token_p99": 0.03434951798757538,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.013270635558348034,
+      "inter_token_latency_p50": 0.01317594037834293,
+      "inter_token_latency_p90": 0.013953309896230092,
+      "inter_token_latency_p99": 0.014080148007312025,
+      "output_tps_per_user_avg": 75.5138067713092,
+      "output_tps_per_user_p50": 75.98396361855555,
+      "output_tps_per_user_p90": 78.96660843355386,
+      "output_tps_per_user_p99": 79.11936279225931,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.03530397237485847,
+      "chunk_inter_token_latency_p50": 0.03529481439149047,
+      "chunk_inter_token_latency_p90": 0.03563218072342417,
+      "chunk_inter_token_latency_p99": 0.03572440514104369,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 1907.25,
+      "output_seq_len_p50": 1856.0,
+      "output_seq_len_p90": 2040.9,
+      "output_seq_len_p99": 2110.29,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.6100263779517263,
+          "time_to_second_token": 0.012099432991817594,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012727410407705224,
+          "chunk_inter_token_latency_avg": 0.03573465229855697,
+          "input_tokens": 32768,
+          "output_tokens": 2118,
+          "output_tps_per_user": 78.57057861468788,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2114123029168695,
+          "time_to_second_token": 0.024417920038104057,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014094241130765572,
+          "chunk_inter_token_latency_avg": 0.035393080381447624,
+          "input_tokens": 32768,
+          "output_tokens": 1799,
+          "output_tps_per_user": 70.95096434934358,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2105558938346803,
+          "time_to_second_token": 0.024473626166582108,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013624470348980637,
+          "chunk_inter_token_latency_avg": 0.035196548401533315,
+          "input_tokens": 32768,
+          "output_tokens": 1861,
+          "output_tps_per_user": 73.39734862242322,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 4.174072765978053,
+          "time_to_second_token": 0.03465495794080198,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012636420345940702,
+          "chunk_inter_token_latency_avg": 0.03489160841789597,
+          "input_tokens": 32768,
+          "output_tokens": 1851,
+          "output_tps_per_user": 79.13633549878213,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 6078,
+      "wall_time": 28.609053145861253,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 303.824900318235,
+      "server_utilization": 0.02540480178671134,
+      "server_spec_accept_rate": 0.5316091954022989,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 8.576,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.268,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.48,
+        "cpu_temp_max_c": 76.25,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 35.08,
+        "mem_util_max_pct": 44.0,
+        "temp_avg_c": 68.44,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1172.67,
+        "power_total_max_w": 1174.13,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 7773.78,
+        "pcie_rx_max_mb_s": 8029.0,
+        "pcie_tx_avg_mb_s": 7899.0,
+        "pcie_tx_max_mb_s": 8283.0
+      }
+    }
+  ],
+  "summary_table": {
+    "0": {
+      "1": 185.68810232786203,
+      "2": 246.22038564672286,
+      "4": 303.4453614548168
+    },
+    "8192": {
+      "1": 166.89761277085296,
+      "2": 236.8867590379722,
+      "4": 291.1245424388467
+    },
+    "32768": {
+      "1": 173.8394059270179,
+      "2": 231.7701102106262,
+      "4": 303.94154153181165
+    }
+  },
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": false,
+      "mode": "skipped",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": true,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/decode-cap8192.log

@@ -0,0 +1,123 @@
+
+New version available: v0.6.2 (current: v0.4.29)
+Upgrade and restart? [Y/n]: Skipping update.
+
+╭──────────────────────────── NVIDIA P2P Override ─────────────────────────────╮
+│ Effective: yes                                                               │
+│ Configured file: yes (/etc/modprobe.d/nvidia-p2p-override.conf)              │
+│ Runtime: ForceP2P=0x11; RMForceP2PType=1; RMPcieP2PType=2;                   │
+│ GrdmaPciTopoCheckOverride=1; EnableResizableBar=1; DmaRemapPeerMmio=1        │
+╰──────────────────────────────────────────────────────────────────────────────╯
+╭─────────────────────────────── Configuration ────────────────────────────────╮
+│ LLM Inference Benchmark                                                      │
+│ Model: glm53-flash-trellismx-p8-k45 @ 127.0.0.1:8001                         │
+│ Decode concurrency: [1, 2, 4]                                                │
+│ Decode contexts: ['0', '8k', '32k']                                          │
+│ Duration: 20.0s per decode test | Max tokens: 8192                           │
+│ Pre-decode warmup: C=1 max-runnable context for 3s                           │
+│ Prefill: skipped | Sustained decode: 9 cells                                 │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Engine: vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f  
+Models: ['glm53-flash-trellismx-p8-k45']
+KV cache budget (vLLM metrics): 29,351,936 tokens (3583 blocks × 2048; local 
+7,337,984 × CP 4; CP source: local process)
+Model context length: 1,000,000 tokens
+Prefill tests: skipped
+Calibrating padding text (run=rumxqbvzafby, up to 32k)...
+  8k: 50,540 chars (8,192 prompt tokens via /tokenize)
+  32k: 205,139 chars (32,768 prompt tokens via /tokenize)
+  Token targeting: /tokenize exact
+Done.
+
+
+
+llm-decode-bench v0.4.29
+╭────────────────────────────────── Phase 2 ───────────────────────────────────╮
+│ Sustained Decode                                                             │
+│ Steady-state decode throughput after the engine has admitted the requested   │
+│ concurrency and passed warmup. Use this as the main tuning/regression signal │
+│ for kernels, NCCL, DCP, MTP, and scheduler changes.                          │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Aggregate tok/s  + TTFT/ITL                              
+╭────────────┬─────────────┬─────────────┬──────────────╮
+│ ctx \ conc │           1 │           2 │            4 │
+├────────────┼─────────────┼─────────────┼──────────────┤
+│ 0          │  185.7 72/5 │ 246.2 113/8 │ 303.4 184/13 │
+│ 8k         │ 166.9 580/6 │ 236.9 941/8 │  291.1 2k/14 │
+│ 32k        │ 173.8 595/6 │ 231.8 962/9 │  303.9 2k/13 │
+╰────────────┴─────────────┴─────────────┴──────────────╯
+Sustained Decode: aggregate tok/s uses OpenAI stream usage by default 
+(continuous completion_tokens when the server supports it). Prometheus is kept 
+as validation/scheduler data.
+Aggregate source(s): openai_continuous_usage
+Per-Request tok/s                    
+╭────────────┬───────┬───────┬──────╮
+│ ctx \ conc │     1 │     2 │    4 │
+├────────────┼───────┼───────┼──────┤
+│ 0          │ 185.7 │ 123.1 │ 75.9 │
+│ 8k         │ 166.9 │ 118.4 │ 72.8 │
+│ 32k        │ 173.8 │ 115.9 │ 76.0 │
+╰────────────┴───────┴───────┴──────╯
+Client request latency: p50 /   
+p90 ms                          
+╭────────────┬─────┬─────┬─────╮
+│ ctx \ conc │   1 │   2 │   4 │
+├────────────┼─────┼─────┼─────┤
+│ 0          │ —/— │ —/— │ —/— │
+│ 8k         │ —/— │ —/— │ —/— │
+│ 32k        │ —/— │ —/— │ —/— │
+╰────────────┴─────┴─────┴─────╯
+Aggregate cells show dim detail as TTFT ms / ITL ms for the same ctx/conc 
+coordinate. ITL is computed from observed generated tokens, including streams 
+stopped at the measurement boundary; a missing ITL means no stream produced at 
+least two measured output tokens. Per-request tok/s and request latency are 
+shown in separate per-cell matrices. Completion/sample counts and full 
+request-level distributions remain in JSON under request_samples.
+Sustained mode: client latency metrics explain request UX variance; aggregate 
+tok/s remains the primary throughput signal. 
+ITL=(last_token_time-first_token_time)/(output_tokens-1), user tok/s=1/ITL.
+Hardware Summary                                                                
+╭───┬─┬───────┬───────────┬───────┬─────────┬─────┬──────┬─────┬───────────────╮
+│ … │ │ mode  │ GPU avg/… │ Mem … │ W avg/… │ T … │ CPU… │ VR… │ PCIe rx/tx a… │
+├───┼─┼───────┼───────────┼───────┼─────────┼─────┼──────┼─────┼───────────────┤
+│ 0 │ │ sust… │    99/99% │   44% │ 1149/1… │ 80C │  75C │ 98… │     8502/8302 │
+│ … │ │ sust… │    99/99% │   44% │ 1152/1… │ 81C │  75C │ 98… │     8303/8387 │
+│ … │ │ sust… │    99/99% │   44% │ 1153/1… │ 82C │  75C │ 98… │     8358/8306 │
+│ 0 │ │ sust… │  100/100% │   41% │ 1176/1… │ 83C │  76C │ 98… │   11310/11180 │
+│ 0 │ │ sust… │  100/100% │   35% │ 1173/1… │ 83C │  76C │ 98… │     7894/7756 │
+│ … │ │ sust… │  100/100% │   41% │ 1178/1… │ 83C │  76C │ 98… │   11234/11127 │
+│ … │ │ sust… │  100/100% │   35% │ 1173/1… │ 83C │  76C │ 98… │     8010/7806 │
+│ … │ │ sust… │  100/100% │   41% │ 1178/1… │ 83C │  75C │ 98… │   11116/11002 │
+│ … │ │ sust… │  100/100% │   35% │ 1173/1… │ 84C │  76C │ 98… │     7774/7899 │
+╰───┴─┴───────┴───────────┴───────┴─────────┴─────┴──────┴─────┴───────────────╯
+╭───────────────────────── Whole-run GPU Power ─────────────────────────╮
+│ avg 1,095 W | max 1,178 W | limit 1,200 W | over 4m 32s | 114 samples │
+╰───────────────────────────────────────────────────────────────────────╯
+Hardware summary is sampled from nvidia-smi during the measured part of each 
+cell. Whole-run GPU power is the sampled sum of GPU power draw across the 
+complete benchmark run, not wall-outlet system power. PCIe rx/tx is MB/s and is 
+a coarse live diagnostic, not a per-kernel NCCL profiler.
+
+╭────────────────────────────────── Phase 3 ───────────────────────────────────╮
+│ Burst / E2E Decode                                                           │
+│ Not run. Re-run with --run-burst to append a finite client-facing request    │
+│ burst after Sustained Decode. This is intentionally disabled by default      │
+│ because it adds another full decode matrix.                                  │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+╭────────────────────────────── Primary Summary ───────────────────────────────╮
+│ Primary matrices repeated last so the important numbers are visible without  │
+│ scrolling back through diagnostics.                                          │
+╰─────────────────────────────────────��────────────────────────────────────────╯
+Aggregate decode tok/s                
+╭────────────┬───────┬───────┬───────╮
+│ ctx \ conc │     1 │     2 │     4 │
+├────────────┼───────┼───────┼───────┤
+│ 0          │ 185.7 │ 246.2 │ 303.4 │
+│ 8k         │ 166.9 │ 236.9 │ 291.1 │
+│ 32k        │ 173.8 │ 231.8 │ 303.9 │
+╰────────────┴───────┴───────┴───────╯
+
+Results saved to 
+<campaign>/candidate-speed-wi
+ndow-01/results-01/decode-warp-quant/rep-1/decode-cap8192.json

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/prefill-command.json

@@ -0,0 +1,31 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "8192",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/prefill.json",
+  "--contexts",
+  "0",
+  "--concurrency",
+  "1,2,4",
+  "--prefill-only",
+  "--prefill-contexts",
+  "8k,32k,64k,128k",
+  "--prefill-duration",
+  "20",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/prefill-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "67ce4838168f0c3c85e10e8484e79c3aa9878c8aa48713fceae52113000a0cf5"
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/prefill.json

@@ -0,0 +1,396 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T02:17:41.737509",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "standalone_cold",
+    "standalone_prefill": true,
+    "prefill_only": true,
+    "skip_prefill": false,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 0,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 29351936,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": true,
+      "prefill_only": true,
+      "skip_prefill": false,
+      "prefill_contexts": "8k,32k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 47,
+    "duration_seconds": 110.814,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 10.87,
+    "cpu_temp_max_c": 75.0,
+    "gpu_util_avg_pct": 85.39,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 19.4,
+    "mem_util_max_pct": 35.0,
+    "temp_avg_c": 60.56,
+    "temp_max_c": 81.0,
+    "power_total_avg_w": 1029.23,
+    "power_total_max_w": 1167.61,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 384658.34,
+    "vram_used_max_mb": 384770.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.24,
+    "vram_used_max_pct": 98.27,
+    "pcie_rx_avg_mb_s": 43654.87,
+    "pcie_rx_max_mb_s": 56329.0,
+    "pcie_tx_avg_mb_s": 42837.81,
+    "pcie_tx_max_mb_s": 56583.0
+  },
+  "event_log": [],
+  "prefill": {
+    "8192": {
+      "ttft_seconds": 1.095,
+      "prefill_seconds": 1.095,
+      "tok_per_sec": 7485.0,
+      "client_ttft_seconds": 1.095,
+      "client_tok_per_sec": 7485.0,
+      "prompt_tokens": 8194,
+      "samples": 14,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.243,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 10.9,
+        "cpu_temp_max_c": 73.38,
+        "gpu_util_avg_pct": 74.67,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 18.72,
+        "mem_util_max_pct": 34.0,
+        "temp_avg_c": 54.17,
+        "temp_max_c": 69.0,
+        "power_total_avg_w": 973.23,
+        "power_total_max_w": 1134.69,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 31974.11,
+        "pcie_rx_max_mb_s": 55296.0,
+        "pcie_tx_avg_mb_s": 32134.78,
+        "pcie_tx_max_mb_s": 49579.0
+      }
+    },
+    "32768": {
+      "ttft_seconds": 4.247,
+      "prefill_seconds": 4.247,
+      "tok_per_sec": 7716.0,
+      "client_ttft_seconds": 4.247,
+      "client_tok_per_sec": 7716.0,
+      "prompt_tokens": 32770,
+      "samples": 5,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 10,
+        "duration_seconds": 21.686,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.06,
+        "cpu_temp_max_c": 75.0,
+        "gpu_util_avg_pct": 86.97,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 19.82,
+        "mem_util_max_pct": 30.0,
+        "temp_avg_c": 58.9,
+        "temp_max_c": 75.0,
+        "power_total_avg_w": 997.14,
+        "power_total_max_w": 1146.5,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 48453.9,
+        "pcie_rx_max_mb_s": 55948.0,
+        "pcie_tx_avg_mb_s": 46628.3,
+        "pcie_tx_max_mb_s": 53063.0
+      }
+    },
+    "65536": {
+      "ttft_seconds": 8.558,
+      "prefill_seconds": 8.558,
+      "tok_per_sec": 7659.0,
+      "client_ttft_seconds": 8.558,
+      "client_tok_per_sec": 7659.0,
+      "prompt_tokens": 65538,
+      "samples": 3,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 11,
+        "duration_seconds": 24.072,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.24,
+        "cpu_temp_max_c": 74.88,
+        "gpu_util_avg_pct": 97.55,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 21.66,
+        "mem_util_max_pct": 35.0,
+        "temp_avg_c": 62.7,
+        "temp_max_c": 79.0,
+        "power_total_avg_w": 1097.1,
+        "power_total_max_w": 1167.61,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 51029.18,
+        "pcie_rx_max_mb_s": 56329.0,
+        "pcie_tx_avg_mb_s": 47833.18,
+        "pcie_tx_max_mb_s": 53350.0
+      }
+    },
+    "131072": {
+      "ttft_seconds": 17.398,
+      "prefill_seconds": 17.398,
+      "tok_per_sec": 7534.0,
+      "client_ttft_seconds": 17.398,
+      "client_tok_per_sec": 7534.0,
+      "prompt_tokens": 131074,
+      "samples": 2,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 15,
+        "duration_seconds": 33.754,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.3,
+        "cpu_temp_max_c": 75.0,
+        "gpu_util_avg_pct": 93.25,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 20.45,
+        "mem_util_max_pct": 28.0,
+        "temp_avg_c": 64.9,
+        "temp_max_c": 81.0,
+        "power_total_avg_w": 1108.87,
+        "power_total_max_w": 1149.51,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384770.0,
+        "vram_used_max_mb": 384770.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 47286.07,
+        "pcie_rx_max_mb_s": 55550.0,
+        "pcie_tx_avg_mb_s": 48228.93,
+        "pcie_tx_max_mb_s": 56583.0
+      }
+    }
+  },
+  "results": [],
+  "summary_table": {},
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": true,
+      "mode": "standalone_cold",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": false,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-1/prefill.log

@@ -0,0 +1,56 @@
+
+New version available: v0.6.2 (current: v0.4.29)
+Upgrade and restart? [Y/n]: Skipping update.
+
+╭──────────────────────────── NVIDIA P2P Override ─────────────────────────────╮
+│ Effective: yes                                                               │
+│ Configured file: yes (/etc/modprobe.d/nvidia-p2p-override.conf)              │
+│ Runtime: ForceP2P=0x11; RMForceP2PType=1; RMPcieP2PType=2;                   │
+│ GrdmaPciTopoCheckOverride=1; EnableResizableBar=1; DmaRemapPeerMmio=1        │
+╰──────────────────────────────────────────────────────────────────────────────╯
+╭─────────────────────────────── Configuration ────────────────────────────────╮
+│ LLM Inference Benchmark                                                      │
+│ Model: glm53-flash-trellismx-p8-k45 @ 127.0.0.1:8001                         │
+│ Decode concurrency: [1, 2, 4]                                                │
+│ Decode contexts: ['0']                                                       │
+│ Decode: skipped (--prefill-only) | Max tokens: 8192                          │
+│ Pre-decode warmup: C=1 max-runnable context for 3s                           │
+│ Prefill-only: standalone cold profile (client) | Sustained decode: 0 cells   │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Engine: vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f  
+Models: ['glm53-flash-trellismx-p8-k45']
+KV cache budget (vLLM metrics): 29,351,936 tokens (3583 blocks × 2048; local 
+7,337,984 × CP 4; CP source: local process)
+Model context length: 1,000,000 tokens
+Prefill tests: standalone cold profile ['8k', '32k', '64k', '128k']
+Calibrating padding text (run=cniaedazxusz, up to 128k)...
+  8k: 50,558 chars (8,192 prompt tokens via /tokenize)
+  32k: 205,152 chars (32,768 prompt tokens via /tokenize)
+  64k: 411,264 chars (65,536 prompt tokens via /tokenize)
+  128k: 823,408 chars (131,072 prompt tokens via /tokenize)
+  Token targeting: /tokenize exact
+Done.
+
+
+
+llm-decode-bench v0.4.29
+Prefill Speed (C=1, client ISL / TTFT)                                          
+                                                                                
+                                      Client                   PCIe rx/tx       
+  Context    Tokens   TTFT (s)         tok/s   Server tok/s           avg    N  
+ ────────────────────────────────────────────────────────────────────────────── 
+  8k          8,194       1.09         7,485              —   31974/32135   14  
+  32k        32,770       4.25         7,716              —   48454/46628    5  
+  64k        65,538       8.56         7,659              —   51029/47833    3  
+  128k      131,074      17.40         7,534              —   47286/48229    2  
+                                                                                
+Client tok/s = prompt_tokens / TTFT. Integrated scout rows come from the 
+prefix-cache scout request that decode needs anyway. Server tok/s is optional 
+Prometheus validation when the engine exports prefill counters and the exact 
+counter delta is uncontaminated; for vLLM this uses newly computed KV tokens, 
+not request prompt tokens.
+
+
+Results saved to 
+<campaign>/candidate-speed-wi
+ndow-01/results-01/decode-warp-quant/rep-1/prefill.json

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap512-command.json

@@ -0,0 +1,27 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "512",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap512.json",
+  "--contexts",
+  "0,8k,32k",
+  "--concurrency",
+  "1,2,4",
+  "--skip-prefill",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap512-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "46f539494020e5442078334b0cb382d3f3c063fdd3312f4e2fe67775555155ce"
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap512.json

@@ -0,0 +1,2342 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T02:37:59.150763",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "skipped",
+    "standalone_prefill": false,
+    "prefill_only": false,
+    "skip_prefill": true,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 32768,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 512,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 29351936,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0,
+      8192,
+      32768
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0,8k,32k",
+      "max_tokens": 512,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": false,
+      "prefill_only": false,
+      "skip_prefill": true,
+      "prefill_contexts": "8k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 113,
+    "duration_seconds": 269.93,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 10.96,
+    "cpu_temp_max_c": 77.38,
+    "gpu_util_avg_pct": 91.76,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 33.91,
+    "mem_util_max_pct": 56.0,
+    "temp_avg_c": 68.4,
+    "temp_max_c": 84.0,
+    "power_total_avg_w": 1097.05,
+    "power_total_max_w": 1178.78,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 384778.0,
+    "vram_used_max_mb": 384778.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.27,
+    "vram_used_max_pct": 98.27,
+    "pcie_rx_avg_mb_s": 17221.25,
+    "pcie_rx_max_mb_s": 74341.0,
+    "pcie_tx_avg_mb_s": 16568.58,
+    "pcie_tx_max_mb_s": 70985.0
+  },
+  "event_log": [
+    "02:33:26 benchmark start engine=vllm",
+    "02:33:26 startup server=http://127.0.0.1:8001 model=glm53-flash-trellismx-p8-k45",
+    "02:33:26 startup decode concurrency=1,2,4 contexts=0,8k,32k",
+    "02:33:26 startup NVIDIA P2P override: enabled: runtime NVIDIA P2P override matches expected RegistryDwords",
+    "02:33:26 startup engine vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f models=['glm53-flash-trellismx-p8-k45']",
+    "02:33:26 startup KV cache budget from vLLM metrics: 29,351,936 tokens (3583 blocks x 2048; local 7,337,984 \u00d7 CP 4; CP source: local process)",
+    "02:33:26 startup model context length: 1,000,000 tokens",
+    "02:33:26 startup prefill tests: skipped",
+    "02:33:26 startup calibrating padding text run=mkkehtszimwn up_to=32k",
+    "02:33:26 startup context 8k: 50,558 chars (8,192 prompt tokens via /tokenize)",
+    "02:33:26 startup context 32k: 205,152 chars (32,768 prompt tokens via /tokenize)",
+    "02:33:26 startup token targeting: /tokenize exact",
+    "02:33:26 startup startup preparation done",
+    "02:33:26 hardware monitor interval=2s",
+    "02:33:26 decode warmup start",
+    "02:33:26 decode warmup start C=1 ctx=32k 3s",
+    "02:33:26 cell start C=1 ctx=32k",
+    "02:33:35 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:33:38 cell done C=1 ctx=32k 180.4 tok/s",
+    "02:33:38 decode warmup done C=1 ctx=32k",
+    "02:33:40 cell start C=1 ctx=0",
+    "02:33:45 ready C=1 ctx=0 running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:34:05 cell done C=1 ctx=0 191.1 tok/s",
+    "02:34:07 cell start C=1 ctx=8k",
+    "02:34:13 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:34:33 cell done C=1 ctx=8k 165.2 tok/s",
+    "02:34:35 cell start C=1 ctx=32k",
+    "02:34:44 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:35:04 cell done C=1 ctx=32k 162.4 tok/s",
+    "02:35:06 cell start C=2 ctx=0",
+    "02:35:12 ready C=2 ctx=0 running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:35:32 cell done C=2 ctx=0 257.8 tok/s",
+    "02:35:34 cell start C=4 ctx=0",
+    "02:35:39 ready C=4 ctx=0 running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:35:59 cell done C=4 ctx=0 305.6 tok/s",
+    "02:36:01 cell start C=2 ctx=8k",
+    "02:36:07 ready C=2 ctx=8k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:36:27 cell done C=2 ctx=8k 183.5 tok/s",
+    "02:36:29 cell start C=4 ctx=8k",
+    "02:36:38 ready C=4 ctx=8k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:36:58 cell done C=4 ctx=8k 214.9 tok/s",
+    "02:37:00 cell start C=2 ctx=32k",
+    "02:37:06 ready C=2 ctx=32k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:37:26 cell done C=2 ctx=32k 186.4 tok/s",
+    "02:37:28 cell start C=4 ctx=32k",
+    "02:37:36 ready C=4 ctx=32k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:37:57 cell done C=4 ctx=32k 213.0 tok/s"
+  ],
+  "prefill": {},
+  "results": [
+    {
+      "concurrency": 1,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 20.000282,
+      "measurement_wall_seconds": 20.000333,
+      "client_output_tokens": 3823,
+      "server_output_tokens": 3823,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 191.1473049181911,
+      "per_request_avg_tps": 191.1473049181911,
+      "ttft_avg": 0.07954429925885051,
+      "ttft_p50": 0.08016932185273618,
+      "ttft_p90": 0.08546189323533326,
+      "ttft_p99": 0.08585712626809254,
+      "time_to_second_token_avg": 0.013534792908467352,
+      "time_to_second_token_p50": 0.013614983530715108,
+      "time_to_second_token_p90": 0.01384659220930189,
+      "time_to_second_token_p99": 0.014208161800634117,
+      "request_latency_avg": 2.673719661180965,
+      "request_latency_p50": 2.642661032034084,
+      "request_latency_p90": 2.810000071441755,
+      "request_latency_p99": 2.8247497128229586,
+      "inter_token_latency_avg": 0.005110297649696225,
+      "inter_token_latency_p50": 0.005072282791254339,
+      "inter_token_latency_p90": 0.00536784812823367,
+      "inter_token_latency_p99": 0.005399117829612438,
+      "output_tps_per_user_avg": 196.0011309531592,
+      "output_tps_per_user_p50": 197.17019837912926,
+      "output_tps_per_user_p90": 204.7882193380745,
+      "output_tps_per_user_p99": 209.5508687453994,
+      "e2e_output_tps_per_user_avg": 191.73898029102156,
+      "e2e_output_tps_per_user_p50": 193.74410633584293,
+      "e2e_output_tps_per_user_p90": 198.93980762708756,
+      "e2e_output_tps_per_user_p99": 202.87404403324115,
+      "chunk_inter_token_latency_avg": 0.01418378739450653,
+      "chunk_inter_token_latency_p50": 0.014161504070741708,
+      "chunk_inter_token_latency_p90": 0.014271326914315551,
+      "chunk_inter_token_latency_p99": 0.014287257237805287,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 10,
+      "completed_request_count": 9,
+      "request_samples": [
+        {
+          "ttft": 0.0710762650705874,
+          "time_to_second_token": 0.012838942930102348,
+          "latency": 2.636708308942616,
+          "inter_token_latency_avg": 0.0050208063480861615,
+          "chunk_inter_token_latency_avg": 0.014174762673326124,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 199.17119495779428,
+          "e2e_output_tps_per_user": 194.18150967382675,
+          "completed": true
+        },
+        {
+          "ttft": 0.07547624688595533,
+          "time_to_second_token": 0.013636301038786769,
+          "latency": 2.739387070061639,
+          "inter_token_latency_avg": 0.005213132726371201,
+          "chunk_inter_token_latency_avg": 0.014169738421147254,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 191.8232380582583,
+          "e2e_output_tps_per_user": 186.90312354744358,
+          "completed": true
+        },
+        {
+          "ttft": 0.07437980198301375,
+          "time_to_second_token": 0.013298804871737957,
+          "latency": 2.6143259189557284,
+          "inter_token_latency_avg": 0.004970540346326251,
+          "chunk_inter_token_latency_avg": 0.01426936020771188,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 201.18537026645492,
+          "e2e_output_tps_per_user": 195.84398268312097,
+          "completed": true
+        },
+        {
+          "ttft": 0.08492515003308654,
+          "time_to_second_token": 0.013621869031339884,
+          "latency": 2.642661032034084,
+          "inter_token_latency_avg": 0.005005353976518586,
+          "chunk_inter_token_latency_avg": 0.01428902727374859,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 199.78607001448037,
+          "e2e_output_tps_per_user": 193.74410633584293,
+          "completed": true
+        },
+        {
+          "ttft": 0.07432189281098545,
+          "time_to_second_token": 0.013672125991433859,
+          "latency": 2.8059029488358647,
+          "inter_token_latency_avg": 0.005345559796526182,
+          "chunk_inter_token_latency_avg": 0.014153269720336162,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 187.0711465335868,
+          "e2e_output_tps_per_user": 182.4724551547382,
+          "completed": true
+        },
+        {
+          "ttft": 0.08590104104951024,
+          "time_to_second_token": 0.013801953988149762,
+          "latency": 2.5183071410283446,
+          "inter_token_latency_avg": 0.004760090215222768,
+          "chunk_inter_token_latency_avg": 0.014141895930109503,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 210.08005201287995,
+          "e2e_output_tps_per_user": 203.31118141170265,
+          "completed": true
+        },
+        {
+          "ttft": 0.07369623705744743,
+          "time_to_second_token": 0.01322799688205123,
+          "latency": 2.6919372058473527,
+          "inter_token_latency_avg": 0.0051237592344225156,
+          "chunk_inter_token_latency_avg": 0.014152653885350839,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 195.1692018004642,
+          "e2e_output_tps_per_user": 190.19760152199967,
+          "completed": true
+        },
+        {
+          "ttft": 0.08539086184464395,
+          "time_to_second_token": 0.013393500121310353,
+          "latency": 2.826388561865315,
+          "inter_token_latency_avg": 0.005363987671273328,
+          "chunk_inter_token_latency_avg": 0.01420206062186876,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 186.42846726801207,
+          "e2e_output_tps_per_user": 181.14989810958562,
+          "completed": true
+        },
+        {
+          "ttft": 0.08541309903375804,
+          "time_to_second_token": 0.013608098030090332,
+          "latency": 2.5878587630577385,
+          "inter_token_latency_avg": 0.004897153941338514,
+          "chunk_inter_token_latency_avg": 0.014138111096180682,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 204.20023792976278,
+          "e2e_output_tps_per_user": 197.84696418093378,
+          "completed": true
+        },
+        {
+          "ttft": 0.08486239681951702,
+          "time_to_second_token": 0.01424833619967103,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005402592240876745,
+          "chunk_inter_token_latency_avg": 0.0141469941152855,
+          "input_tokens": 78,
+          "output_tokens": 255,
+          "output_tps_per_user": 185.09633068989817,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3823,
+      "wall_time": 25.55206400505267,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 191.09785171771475,
+      "server_utilization": 0.005862646566164198,
+      "server_spec_accept_rate": 0.6,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.537,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.265,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.57,
+        "cpu_temp_max_c": 75.75,
+        "gpu_util_avg_pct": 98.97,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 43.47,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 68.11,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1152.88,
+        "power_total_max_w": 1155.06,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8388.89,
+        "pcie_rx_max_mb_s": 8661.0,
+        "pcie_tx_avg_mb_s": 8246.33,
+        "pcie_tx_max_mb_s": 8504.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.993423,
+      "measurement_wall_seconds": 20.000549,
+      "client_output_tokens": 3302,
+      "server_output_tokens": 3302,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 165.15431387835127,
+      "per_request_avg_tps": 165.15431387835127,
+      "ttft_avg": 0.6249137402628548,
+      "ttft_p50": 0.6289092639926821,
+      "ttft_p90": 0.6339087889529764,
+      "ttft_p99": 0.6347594925714657,
+      "time_to_second_token_avg": 0.017741062707500532,
+      "time_to_second_token_p50": 0.01763475441839546,
+      "time_to_second_token_p90": 0.018197339260950685,
+      "time_to_second_token_p99": 0.018255747086368502,
+      "request_latency_avg": 3.151980838239459,
+      "request_latency_p50": 3.1515419960487634,
+      "request_latency_p90": 3.2448623499367386,
+      "request_latency_p99": 3.291342785186134,
+      "inter_token_latency_avg": 0.004930662211393978,
+      "inter_token_latency_p50": 0.004938842342353614,
+      "inter_token_latency_p90": 0.005095802539678877,
+      "inter_token_latency_p99": 0.0051998018340067296,
+      "output_tps_per_user_avg": 203.0459222201492,
+      "output_tps_per_user_p50": 202.51955283196384,
+      "output_tps_per_user_p90": 210.3772401347902,
+      "output_tps_per_user_p99": 215.44168752783455,
+      "e2e_output_tps_per_user_avg": 162.56228618550645,
+      "e2e_output_tps_per_user_p50": 162.46015462967605,
+      "e2e_output_tps_per_user_p90": 167.22336677689123,
+      "e2e_output_tps_per_user_p99": 170.41814081496065,
+      "chunk_inter_token_latency_avg": 0.01426418191410213,
+      "chunk_inter_token_latency_p50": 0.014250208142300946,
+      "chunk_inter_token_latency_p90": 0.014323315611798752,
+      "chunk_inter_token_latency_p99": 0.014336108877223197,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 8,
+      "completed_request_count": 7,
+      "request_samples": [
+        {
+          "ttft": 0.591038762126118,
+          "time_to_second_token": 0.017253336030989885,
+          "latency": 3.1515419960487634,
+          "inter_token_latency_avg": 0.005010769538009091,
+          "chunk_inter_token_latency_avg": 0.01422501796623692,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 199.57014434899074,
+          "e2e_output_tps_per_user": 162.46015462967605,
+          "completed": true
+        },
+        {
+          "ttft": 0.6249128989875317,
+          "time_to_second_token": 0.017716374015435576,
+          "latency": 3.1119065389502794,
+          "inter_token_latency_avg": 0.004866915146698137,
+          "chunk_inter_token_latency_avg": 0.014211392228358558,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 205.46896131493693,
+          "e2e_output_tps_per_user": 164.52936281714614,
+          "completed": true
+        },
+        {
+          "ttft": 0.6324374838732183,
+          "time_to_second_token": 0.018169526010751724,
+          "latency": 2.998129991814494,
+          "inter_token_latency_avg": 0.004629535240589581,
+          "chunk_inter_token_latency_avg": 0.014337530351159247,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 216.00440390483948,
+          "e2e_output_tps_per_user": 170.77311570807947,
+          "completed": true
+        },
+        {
+          "ttft": 0.6335036919917911,
+          "time_to_second_token": 0.01815098780207336,
+          "latency": 3.2965072779916227,
+          "inter_token_latency_avg": 0.005211357311154269,
+          "chunk_inter_token_latency_avg": 0.014317223580644255,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 191.88858876738755,
+          "e2e_output_tps_per_user": 155.3159015962898,
+          "completed": true
+        },
+        {
+          "ttft": 0.6260347329080105,
+          "time_to_second_token": 0.017283352091908455,
+          "latency": 3.1057244250550866,
+          "inter_token_latency_avg": 0.004852621706745746,
+          "chunk_inter_token_latency_avg": 0.01425109018475331,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 206.07417194912927,
+          "e2e_output_tps_per_user": 164.85686748943237,
+          "completed": true
+        },
+        {
+          "ttft": 0.6317837950773537,
+          "time_to_second_token": 0.01826223684474826,
+          "latency": 3.2104323979001492,
+          "inter_token_latency_avg": 0.005046279066189424,
+          "chunk_inter_token_latency_avg": 0.014246677363661853,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 198.165814233323,
+          "e2e_output_tps_per_user": 159.48007512473532,
+          "completed": true
+        },
+        {
+          "ttft": 0.6247445419430733,
+          "time_to_second_token": 0.017539554042741656,
+          "latency": 3.189623239915818,
+          "inter_token_latency_avg": 0.005019332089966232,
+          "chunk_inter_token_latency_avg": 0.014249326099848582,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 199.22969472353194,
+          "e2e_output_tps_per_user": 160.52052593318606,
+          "completed": true
+        },
+        {
+          "ttft": 0.6348540151957422,
+          "time_to_second_token": 0.017553134821355343,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.004808487591799348,
+          "chunk_inter_token_latency_avg": 0.014275197538154316,
+          "input_tokens": 8192,
+          "output_tokens": 381,
+          "output_tps_per_user": 207.96559851905482,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3302,
+      "wall_time": 26.068795799044892,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 165.0529013540346,
+      "server_utilization": 0.006141820212172022,
+      "server_spec_accept_rate": 0.6956521739130435,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 0.9,
+      "max_running_reqs": 1,
+      "effective_concurrency": 0.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 6.061,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.92,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.54,
+        "cpu_temp_max_c": 76.12,
+        "gpu_util_avg_pct": 98.81,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 39.97,
+        "mem_util_max_pct": 56.0,
+        "temp_avg_c": 68.56,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1154.06,
+        "power_total_max_w": 1157.95,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11118.0,
+        "pcie_rx_max_mb_s": 30706.0,
+        "pcie_tx_avg_mb_s": 9898.38,
+        "pcie_tx_max_mb_s": 21763.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.993472,
+      "measurement_wall_seconds": 20.000595,
+      "client_output_tokens": 3246,
+      "server_output_tokens": 3246,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 162.35299164317425,
+      "per_request_avg_tps": 162.35299164317425,
+      "ttft_avg": 0.6279244296310935,
+      "ttft_p50": 0.629666926455684,
+      "ttft_p90": 0.6346158204367385,
+      "ttft_p99": 0.6394360246905125,
+      "time_to_second_token_avg": 0.011822661821497604,
+      "time_to_second_token_p50": 0.012242211028933525,
+      "time_to_second_token_p90": 0.012837472674436867,
+      "time_to_second_token_p99": 0.012860276207793503,
+      "request_latency_avg": 3.184498446295038,
+      "request_latency_p50": 3.200756788952276,
+      "request_latency_p90": 3.258722552191466,
+      "request_latency_p99": 3.2730169181805104,
+      "inter_token_latency_avg": 0.005025481984653069,
+      "inter_token_latency_p50": 0.005032082509756118,
+      "inter_token_latency_p90": 0.005174755995160009,
+      "inter_token_latency_p99": 0.005182216347690941,
+      "output_tps_per_user_avg": 199.13235326934893,
+      "output_tps_per_user_p50": 198.72498796426024,
+      "output_tps_per_user_p90": 206.07410309939215,
+      "output_tps_per_user_p99": 208.73558083264146,
+      "e2e_output_tps_per_user_avg": 160.84431793668682,
+      "e2e_output_tps_per_user_p50": 159.96216949916902,
+      "e2e_output_tps_per_user_p90": 164.8638332468683,
+      "e2e_output_tps_per_user_p99": 166.41261273335627,
+      "chunk_inter_token_latency_avg": 0.014293934351271118,
+      "chunk_inter_token_latency_p50": 0.014270523142799533,
+      "chunk_inter_token_latency_p90": 0.014406589656055605,
+      "chunk_inter_token_latency_p99": 0.01443256986090686,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 8,
+      "completed_request_count": 7,
+      "request_samples": [
+        {
+          "ttft": 0.6056491718627512,
+          "time_to_second_token": 0.01198451709933579,
+          "latency": 3.248134132940322,
+          "inter_token_latency_avg": 0.005171203446335755,
+          "chunk_inter_token_latency_avg": 0.014283702492311194,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 193.37858399452193,
+          "e2e_output_tps_per_user": 157.62895836340357,
+          "completed": true
+        },
+        {
+          "ttft": 0.6288057561032474,
+          "time_to_second_token": 0.012826613849028945,
+          "latency": 3.2020828151144087,
+          "inter_token_latency_avg": 0.005035767238769396,
+          "chunk_inter_token_latency_avg": 0.014295983661173118,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 198.5794721211882,
+          "e2e_output_tps_per_user": 159.89592698329588,
+          "completed": true
+        },
+        {
+          "ttft": 0.6289016020018607,
+          "time_to_second_token": 0.01273016701452434,
+          "latency": 3.073511565104127,
+          "inter_token_latency_avg": 0.0047839725305328104,
+          "chunk_inter_token_latency_avg": 0.014212848622687594,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 209.03130058078028,
+          "e2e_output_tps_per_user": 166.58469934296605,
+          "completed": true
+        },
+        {
+          "ttft": 0.6260690451599658,
+          "time_to_second_token": 0.011439014924690127,
+          "latency": 3.274605181068182,
+          "inter_token_latency_avg": 0.005183045275749934,
+          "chunk_inter_token_latency_avg": 0.014394218129935958,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 192.9367672473805,
+          "e2e_output_tps_per_user": 156.35472726913133,
+          "completed": true
+        },
+        {
+          "ttft": 0.6312455229926854,
+          "time_to_second_token": 0.011678099865093827,
+          "latency": 3.200756788952276,
+          "inter_token_latency_avg": 0.005028397780742839,
+          "chunk_inter_token_latency_avg": 0.014435456550334779,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 198.87050380733228,
+          "e2e_output_tps_per_user": 159.96216949916902,
+          "completed": true
+        },
+        {
+          "ttft": 0.6304322509095073,
+          "time_to_second_token": 0.01249990495853126,
+          "latency": 3.165042991982773,
+          "inter_token_latency_avg": 0.004960099297599346,
+          "chunk_inter_token_latency_avg": 0.014239386185804863,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 201.6088670813492,
+          "e2e_output_tps_per_user": 161.76715491603875,
+          "completed": true
+        },
+        {
+          "ttft": 0.6323204850777984,
+          "time_to_second_token": 0.01286280993372202,
+          "latency": 3.127355648903176,
+          "inter_token_latency_avg": 0.004882651984002696,
+          "chunk_inter_token_latency_avg": 0.014257343793287873,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 204.80673275022582,
+          "e2e_output_tps_per_user": 163.71658918280312,
+          "completed": true
+        },
+        {
+          "ttft": 0.6399716029409319,
+          "time_to_second_token": 0.008560166927054524,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005158718323491779,
+          "chunk_inter_token_latency_avg": 0.014232535374633568,
+          "input_tokens": 32768,
+          "output_tokens": 310,
+          "output_tps_per_user": 193.84659857201325,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3246,
+      "wall_time": 29.109418482985348,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 162.2569315918152,
+      "server_utilization": 0.006979341150195384,
+      "server_spec_accept_rate": 0.6163522012578616,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 0.9,
+      "max_running_reqs": 1,
+      "effective_concurrency": 0.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 9.102,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.9,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.49,
+        "cpu_temp_max_c": 76.62,
+        "gpu_util_avg_pct": 95.94,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 38.56,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 68.72,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1151.98,
+        "power_total_max_w": 1158.51,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 12769.0,
+        "pcie_rx_max_mb_s": 39144.0,
+        "pcie_tx_avg_mb_s": 12689.38,
+        "pcie_tx_max_mb_s": 43337.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.987074,
+      "measurement_wall_seconds": 20.001238,
+      "client_output_tokens": 5152,
+      "server_output_tokens": 5152,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 257.76658819245483,
+      "per_request_avg_tps": 128.88329409622742,
+      "ttft_avg": 0.12264618660057229,
+      "ttft_p50": 0.12349911453202367,
+      "ttft_p90": 0.14274162366054954,
+      "ttft_p99": 0.16984736640239134,
+      "time_to_second_token_avg": 0.019459182529577186,
+      "time_to_second_token_p50": 0.02056812052614987,
+      "time_to_second_token_p90": 0.02087695982772857,
+      "time_to_second_token_p99": 0.020941995084285736,
+      "request_latency_avg": 4.002757317425373,
+      "request_latency_p50": 4.017580935033038,
+      "request_latency_p90": 4.2898590574972335,
+      "request_latency_p99": 4.336852466568817,
+      "inter_token_latency_avg": 0.007545030826191364,
+      "inter_token_latency_p50": 0.00755755253999135,
+      "inter_token_latency_p90": 0.008070413317256143,
+      "inter_token_latency_p99": 0.008259788365663738,
+      "output_tps_per_user_avg": 132.88272402904317,
+      "output_tps_per_user_p50": 132.32326777905013,
+      "output_tps_per_user_p90": 141.37995688471477,
+      "output_tps_per_user_p99": 145.6980556167255,
+      "e2e_output_tps_per_user_avg": 128.22354203464204,
+      "e2e_output_tps_per_user_p50": 127.43988690683825,
+      "e2e_output_tps_per_user_p90": 134.35973544624332,
+      "e2e_output_tps_per_user_p99": 140.69704751927173,
+      "chunk_inter_token_latency_avg": 0.02104700313164791,
+      "chunk_inter_token_latency_p50": 0.02098676121404283,
+      "chunk_inter_token_latency_p90": 0.0212750823967716,
+      "chunk_inter_token_latency_p99": 0.02139749144743477,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 14,
+      "completed_request_count": 12,
+      "request_samples": [
+        {
+          "ttft": 0.07193002197891474,
+          "time_to_second_token": 0.01364690694026649,
+          "latency": 4.016205613967031,
+          "inter_token_latency_avg": 0.0077187389275697,
+          "chunk_inter_token_latency_avg": 0.020869183026392152,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 129.55484171490914,
+          "e2e_output_tps_per_user": 127.48351285089433,
+          "completed": true
+        },
+        {
+          "ttft": 0.10589846200309694,
+          "time_to_second_token": 0.01401821500621736,
+          "latency": 3.8386515881866217,
+          "inter_token_latency_avg": 0.007304800638323923,
+          "chunk_inter_token_latency_avg": 0.02108900071290127,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 136.89627541011834,
+          "e2e_output_tps_per_user": 133.3801696344806,
+          "completed": true
+        },
+        {
+          "ttft": 0.12403434701263905,
+          "time_to_second_token": 0.020574419992044568,
+          "latency": 3.8075810340233147,
+          "inter_token_latency_avg": 0.007208506236811498,
+          "chunk_inter_token_latency_avg": 0.021048838211489576,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 138.72499615708523,
+          "e2e_output_tps_per_user": 134.46857609199472,
+          "completed": true
+        },
+        {
+          "ttft": 0.11724809114821255,
+          "time_to_second_token": 0.020686850883066654,
+          "latency": 3.954718264983967,
+          "inter_token_latency_avg": 0.007509726367584646,
+          "chunk_inter_token_latency_avg": 0.02096978237068718,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 133.16064408371113,
+          "e2e_output_tps_per_user": 129.46560682549045,
+          "completed": true
+        },
+        {
+          "ttft": 0.11513775610364974,
+          "time_to_second_token": 0.020763778826221824,
+          "latency": 4.340469955932349,
+          "inter_token_latency_avg": 0.008268751858764578,
+          "chunk_inter_token_latency_avg": 0.021126660999143496,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 120.93723660845332,
+          "e2e_output_tps_per_user": 117.95957700391926,
+          "completed": true
+        },
+        {
+          "ttft": 0.12337096291594207,
+          "time_to_second_token": 0.020409638062119484,
+          "latency": 3.619222234003246,
+          "inter_token_latency_avg": 0.006841196225219772,
+          "chunk_inter_token_latency_avg": 0.02093324114423535,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 146.1732666450267,
+          "e2e_output_tps_per_user": 141.46685859455317,
+          "completed": true
+        },
+        {
+          "ttft": 0.12318649212829769,
+          "time_to_second_token": 0.02092546597123146,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007490993097976402,
+          "chunk_inter_token_latency_avg": 0.021415427327156067,
+          "input_tokens": 78,
+          "output_tokens": 244,
+          "output_tps_per_user": 133.4936485617825,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.15055592195130885,
+          "time_to_second_token": 0.018002169905230403,
+          "latency": 4.036904443986714,
+          "inter_token_latency_avg": 0.007605378712398053,
+          "chunk_inter_token_latency_avg": 0.02089434689266347,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 131.48589147438915,
+          "e2e_output_tps_per_user": 126.82985369214379,
+          "completed": true
+        },
+        {
+          "ttft": 0.17272999603301287,
+          "time_to_second_token": 0.020944464951753616,
+          "latency": 4.130337374052033,
+          "inter_token_latency_avg": 0.0077448285284129545,
+          "chunk_inter_token_latency_avg": 0.021277459021607634,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 129.11841706131574,
+          "e2e_output_tps_per_user": 123.96081812021731,
+          "completed": true
+        },
+        {
+          "ttft": 0.12362726614810526,
+          "time_to_second_token": 0.020660570822656155,
+          "latency": 4.093334136996418,
+          "inter_token_latency_avg": 0.0077685065965720414,
+          "chunk_inter_token_latency_avg": 0.02100374005739848,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 128.72486977629183,
+          "e2e_output_tps_per_user": 125.08140866694363,
+          "completed": true
+        },
+        {
+          "ttft": 0.12366992980241776,
+          "time_to_second_token": 0.02057293802499771,
+          "latency": 3.8691232178825885,
+          "inter_token_latency_avg": 0.007329654184109923,
+          "chunk_inter_token_latency_avg": 0.02092432004514062,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 136.43208463612325,
+          "e2e_output_tps_per_user": 132.32972205010222,
+          "completed": true
+        },
+        {
+          "ttft": 0.11748491204343736,
+          "time_to_second_token": 0.020382387097924948,
+          "latency": 4.307583688991144,
+          "inter_token_latency_avg": 0.008199801911835043,
+          "chunk_inter_token_latency_avg": 0.021269536938820846,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 121.9541655703496,
+          "e2e_output_tps_per_user": 118.86013992218285,
+          "completed": true
+        },
+        {
+          "ttft": 0.12366419215686619,
+          "time_to_second_token": 0.02027744590304792,
+          "latency": 4.018956256099045,
+          "inter_token_latency_avg": 0.007622880751354558,
+          "chunk_inter_token_latency_avg": 0.02094243045130204,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 131.18400151049244,
+          "e2e_output_tps_per_user": 127.39626096278218,
+          "completed": true
+        },
+        {
+          "ttft": 0.1245082609821111,
+          "time_to_second_token": 0.020563303027302027,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007016667529746,
+          "chunk_inter_token_latency_avg": 0.020894076644132533,
+          "input_tokens": 78,
+          "output_tokens": 135,
+          "output_tps_per_user": 142.517797196556,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5152,
+      "wall_time": 25.544181779026985,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 257.5183083679263,
+      "server_utilization": 0.011725293132328285,
+      "server_spec_accept_rate": 0.5490196078431373,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.536,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.327,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.5,
+        "cpu_temp_max_c": 76.5,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.53,
+        "mem_util_max_pct": 50.0,
+        "temp_avg_c": 68.83,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1175.08,
+        "power_total_max_w": 1176.23,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11424.78,
+        "pcie_rx_max_mb_s": 12660.0,
+        "pcie_tx_avg_mb_s": 11473.0,
+        "pcie_tx_max_mb_s": 12876.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.982261,
+      "measurement_wall_seconds": 20.001434,
+      "client_output_tokens": 6107,
+      "server_output_tokens": 6107,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 305.62106309879454,
+      "per_request_avg_tps": 76.40526577469863,
+      "ttft_avg": 0.18411949259461835,
+      "ttft_p50": 0.18233595008496195,
+      "ttft_p90": 0.22579583001788706,
+      "ttft_p99": 0.24202772771241143,
+      "time_to_second_token_avg": 0.032063150822068565,
+      "time_to_second_token_p50": 0.03419885353650898,
+      "time_to_second_token_p90": 0.034818764543160796,
+      "time_to_second_token_p99": 0.03491839349735528,
+      "request_latency_avg": 6.813750679527099,
+      "request_latency_p50": 6.7830935755046085,
+      "request_latency_p90": 7.136929157585837,
+      "request_latency_p99": 7.218653435369488,
+      "inter_token_latency_avg": 0.01272961827279658,
+      "inter_token_latency_p50": 0.012860421986189724,
+      "inter_token_latency_p90": 0.013537522864770526,
+      "inter_token_latency_p99": 0.013757404152471607,
+      "output_tps_per_user_avg": 78.81967025679022,
+      "output_tps_per_user_p50": 77.75794916087145,
+      "output_tps_per_user_p90": 85.34425839032679,
+      "output_tps_per_user_p99": 89.87592923645526,
+      "e2e_output_tps_per_user_avg": 75.21735206235604,
+      "e2e_output_tps_per_user_p50": 75.48202324847628,
+      "e2e_output_tps_per_user_p90": 78.42642900563833,
+      "e2e_output_tps_per_user_p99": 78.66327604603737,
+      "chunk_inter_token_latency_avg": 0.03457812773162968,
+      "chunk_inter_token_latency_p50": 0.03461816136042967,
+      "chunk_inter_token_latency_p90": 0.03496178867127844,
+      "chunk_inter_token_latency_p99": 0.035020915078215,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 16,
+      "completed_request_count": 12,
+      "request_samples": [
+        {
+          "ttft": 0.071592987049371,
+          "time_to_second_token": 0.013179793022572994,
+          "latency": 6.610689978115261,
+          "inter_token_latency_avg": 0.012796667301498806,
+          "chunk_inter_token_latency_avg": 0.03405779682846818,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 78.14534647492752,
+          "e2e_output_tps_per_user": 77.45031179725261,
+          "completed": true
+        },
+        {
+          "ttft": 0.22638577106408775,
+          "time_to_second_token": 0.034233805956318974,
+          "latency": 6.729027182096615,
+          "inter_token_latency_avg": 0.012725325657597902,
+          "chunk_inter_token_latency_avg": 0.03458851814379004,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 78.5834505856383,
+          "e2e_output_tps_per_user": 76.08826449122355,
+          "completed": true
+        },
+        {
+          "ttft": 0.18156861700117588,
+          "time_to_second_token": 0.034193265018984675,
+          "latency": 6.771002053981647,
+          "inter_token_latency_avg": 0.012895173066497987,
+          "chunk_inter_token_latency_avg": 0.03468122861568669,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 77.5483969732851,
+          "e2e_output_tps_per_user": 75.61657726848887,
+          "completed": true
+        },
+        {
+          "ttft": 0.16493634996004403,
+          "time_to_second_token": 0.03424763702787459,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.01179988086244578,
+          "chunk_inter_token_latency_avg": 0.03492764735283951,
+          "input_tokens": 78,
+          "output_tokens": 445,
+          "output_tps_per_user": 84.74661834786765,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18636853992938995,
+          "time_to_second_token": 0.029447735054418445,
+          "latency": 7.226989238988608,
+          "inter_token_latency_avg": 0.013778122698746023,
+          "chunk_inter_token_latency_avg": 0.03468286058649861,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 72.57882818034507,
+          "e2e_output_tps_per_user": 70.84554619755497,
+          "completed": true
+        },
+        {
+          "ttft": 0.18095432897098362,
+          "time_to_second_token": 0.03420444205403328,
+          "latency": 6.79518509702757,
+          "inter_token_latency_avg": 0.012943700133183144,
+          "chunk_inter_token_latency_avg": 0.034995929989717386,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 77.25766123369529,
+          "e2e_output_tps_per_user": 75.34746922846371,
+          "completed": true
+        },
+        {
+          "ttft": 0.18133209715597332,
+          "time_to_second_token": 0.034592753974720836,
+          "latency": 6.89423265401274,
+          "inter_token_latency_avg": 0.01313679169639289,
+          "chunk_inter_token_latency_avg": 0.03460258018998333,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 76.12208696850851,
+          "e2e_output_tps_per_user": 74.26497272354074,
+          "completed": true
+        },
+        {
+          "ttft": 0.18180311610922217,
+          "time_to_second_token": 0.03340313700027764,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.011041162894689477,
+          "chunk_inter_token_latency_avg": 0.034236164014541014,
+          "input_tokens": 78,
+          "output_tokens": 401,
+          "output_tps_per_user": 90.57016996651457,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18593904399313033,
+          "time_to_second_token": 0.029689945047721267,
+          "latency": 6.507442394969985,
+          "inter_token_latency_avg": 0.01237084804496449,
+          "chunk_inter_token_latency_avg": 0.033804830753886926,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 80.83520194939638,
+          "e2e_output_tps_per_user": 78.67914442020374,
+          "completed": true
+        },
+        {
+          "ttft": 0.1811696880031377,
+          "time_to_second_token": 0.034721852047368884,
+          "latency": 7.151209206087515,
+          "inter_token_latency_avg": 0.01363999905691659,
+          "chunk_inter_token_latency_avg": 0.03502532421147928,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 73.31378806019188,
+          "e2e_output_tps_per_user": 71.5962832641166,
+          "completed": true
+        },
+        {
+          "ttft": 0.18142080190591514,
+          "time_to_second_token": 0.03491567703895271,
+          "latency": 6.5193956850562245,
+          "inter_token_latency_avg": 0.01240308196311215,
+          "chunk_inter_token_latency_avg": 0.034633742530876005,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 80.62512228606465,
+          "e2e_output_tps_per_user": 78.53488647323674,
+          "completed": true
+        },
+        {
+          "ttft": 0.24478807300329208,
+          "time_to_second_token": 0.03329922794364393,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.01343504667262446,
+          "chunk_inter_token_latency_avg": 0.034875908828251166,
+          "input_tokens": 78,
+          "output_tokens": 380,
+          "output_tps_per_user": 74.43219397500282,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.18580231512896717,
+          "time_to_second_token": 0.029555089073255658,
+          "latency": 7.008408721070737,
+          "inter_token_latency_avg": 0.013351480246461388,
+          "chunk_inter_token_latency_avg": 0.034457608110817016,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 74.89806235267697,
+          "e2e_output_tps_per_user": 73.05510000589366,
+          "completed": true
+        },
+        {
+          "ttft": 0.18286878406070173,
+          "time_to_second_token": 0.0349188728723675,
+          "latency": 6.753246791893616,
+          "inter_token_latency_avg": 0.012857882598498854,
+          "chunk_inter_token_latency_avg": 0.03458093688333113,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 77.7733030566595,
+          "e2e_output_tps_per_user": 75.81538418151527,
+          "completed": true
+        },
+        {
+          "ttft": 0.22520588897168636,
+          "time_to_second_token": 0.03444401710294187,
+          "latency": 6.798179151024669,
+          "inter_token_latency_avg": 0.012862961373880592,
+          "chunk_inter_token_latency_avg": 0.03477763630715864,
+          "input_tokens": 78,
+          "output_tokens": 512,
+          "output_tps_per_user": 77.7425952650834,
+          "e2e_output_tps_per_user": 75.31428469678204,
+          "completed": true
+        },
+        {
+          "ttft": 0.18377547920681536,
+          "time_to_second_token": 0.033963162917643785,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.011635768097234754,
+          "chunk_inter_token_latency_avg": 0.03432133035874999,
+          "input_tokens": 78,
+          "output_tokens": 411,
+          "output_tps_per_user": 85.94189843278592,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 6107,
+      "wall_time": 25.55054651084356,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 305.247463685858,
+      "server_utilization": 0.02345058626465657,
+      "server_spec_accept_rate": 0.5805555555555556,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 3.9,
+      "max_running_reqs": 4,
+      "effective_concurrency": 3.9,
+      "avg_queue_reqs": 0.1,
+      "max_queue_reqs": 1,
+      "queue_fraction": 0.05,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.534,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": true,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.89,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.53,
+        "cpu_temp_max_c": 77.38,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 35.53,
+        "mem_util_max_pct": 45.0,
+        "temp_avg_c": 69.31,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1171.59,
+        "power_total_max_w": 1174.02,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8123.88,
+        "pcie_rx_max_mb_s": 11132.0,
+        "pcie_tx_avg_mb_s": 8456.62,
+        "pcie_tx_max_mb_s": 11467.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.996131,
+      "measurement_wall_seconds": 20.000268,
+      "client_output_tokens": 3670,
+      "server_output_tokens": 3670,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 183.53550127953065,
+      "per_request_avg_tps": 91.76775063976532,
+      "ttft_avg": 1.2004322615684941,
+      "ttft_p50": 1.2173257099930197,
+      "ttft_p90": 1.3659410797525195,
+      "ttft_p99": 1.9023066108068454,
+      "time_to_second_token_avg": 0.02541845981031656,
+      "time_to_second_token_p50": 0.027192589943297207,
+      "time_to_second_token_p90": 0.027629395388066767,
+      "time_to_second_token_p99": 0.027753254855051635,
+      "request_latency_avg": 5.37285651997081,
+      "request_latency_p50": 5.446895623463206,
+      "request_latency_p90": 5.665212600212544,
+      "request_latency_p99": 5.969511628868058,
+      "inter_token_latency_avg": 0.00809208952899205,
+      "inter_token_latency_p50": 0.00815642245274234,
+      "inter_token_latency_p90": 0.008585907641263088,
+      "inter_token_latency_p99": 0.009444491741223483,
+      "output_tps_per_user_avg": 124.44614817528017,
+      "output_tps_per_user_p50": 122.61098226109307,
+      "output_tps_per_user_p90": 140.4927553359884,
+      "output_tps_per_user_p99": 142.21413373382023,
+      "e2e_output_tps_per_user_avg": 95.63015924091229,
+      "e2e_output_tps_per_user_p50": 93.9996349349982,
+      "e2e_output_tps_per_user_p90": 103.25204134689379,
+      "e2e_output_tps_per_user_p99": 103.55866690719752,
+      "chunk_inter_token_latency_avg": 0.023212736865606577,
+      "chunk_inter_token_latency_p50": 0.02336988428975669,
+      "chunk_inter_token_latency_p90": 0.0253701743817706,
+      "chunk_inter_token_latency_p99": 0.02550767397899725,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 10,
+      "completed_request_count": 8,
+      "request_samples": [
+        {
+          "ttft": 0.5909663320053369,
+          "time_to_second_token": 0.018111236859112978,
+          "latency": 5.465850109001622,
+          "inter_token_latency_avg": 0.009539889974552416,
+          "chunk_inter_token_latency_avg": 0.025522951712022433,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 104.82301186570206,
+          "e2e_output_tps_per_user": 93.67252847947574,
+          "completed": true
+        },
+        {
+          "ttft": 1.9619027809239924,
+          "time_to_second_token": 0.027767017018049955,
+          "latency": 6.003322632052004,
+          "inter_token_latency_avg": 0.007908845109839554,
+          "chunk_inter_token_latency_avg": 0.023634034217122877,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 126.44071114199465,
+          "e2e_output_tps_per_user": 85.28610427605696,
+          "completed": true
+        },
+        {
+          "ttft": 1.2156082668807358,
+          "time_to_second_token": 0.026928086066618562,
+          "latency": 5.520308300852776,
+          "inter_token_latency_avg": 0.008424070516579334,
+          "chunk_inter_token_latency_avg": 0.023395108880282824,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 118.70745835186321,
+          "e2e_output_tps_per_user": 92.74844303911547,
+          "completed": true
+        },
+        {
+          "ttft": 1.2087725747842342,
+          "time_to_second_token": 0.027614104095846415,
+          "latency": 5.18546292395331,
+          "inter_token_latency_avg": 0.007782172894655725,
+          "chunk_inter_token_latency_avg": 0.0235307121252608,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 128.4988156311373,
+          "e2e_output_tps_per_user": 98.73756837309712,
+          "completed": true
+        },
+        {
+          "ttft": 1.2152251708321273,
+          "time_to_second_token": 0.02757256105542183,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007022205717217775,
+          "chunk_inter_token_latency_avg": 0.021066617151653325,
+          "input_tokens": 8192,
+          "output_tokens": 163,
+          "output_tps_per_user": 142.40539800024598,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.2997231129556894,
+          "time_to_second_token": 0.02719552395865321,
+          "latency": 4.942431465024129,
+          "inter_token_latency_avg": 0.007128587773128061,
+          "chunk_inter_token_latency_avg": 0.020935105471657695,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 140.2802394844042,
+          "e2e_output_tps_per_user": 103.59273641389794,
+          "completed": true
+        },
+        {
+          "ttft": 0.8319369831588119,
+          "time_to_second_token": 0.017664924962446094,
+          "latency": 4.9657619839999825,
+          "inter_token_latency_avg": 0.008089677105364327,
+          "chunk_inter_token_latency_avg": 0.022106016047278985,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 123.61432810920134,
+          "e2e_output_tps_per_user": 103.10602917532059,
+          "completed": true
+        },
+        {
+          "ttft": 1.2190431531053036,
+          "time_to_second_token": 0.027477154973894358,
+          "latency": 5.471773606957868,
+          "inter_token_latency_avg": 0.008322368794232024,
+          "chunk_inter_token_latency_avg": 0.023238964228702537,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 120.15809737884591,
+          "e2e_output_tps_per_user": 93.57112277981393,
+          "completed": true
+        },
+        {
+          "ttft": 1.2259023920632899,
+          "time_to_second_token": 0.027189655927941203,
+          "latency": 5.42794113792479,
+          "inter_token_latency_avg": 0.008223167800120354,
+          "chunk_inter_token_latency_avg": 0.02334465969923056,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 121.6076364129848,
+          "e2e_output_tps_per_user": 94.32674139052064,
+          "completed": true
+        },
+        {
+          "ttft": 1.23524184897542,
+          "time_to_second_token": 0.02666433318518102,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.00847990960423094,
+          "chunk_inter_token_latency_avg": 0.02535319912285373,
+          "input_tokens": 8192,
+          "output_tokens": 294,
+          "output_tps_per_user": 117.9257853764223,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3670,
+      "wall_time": 25.57071568304673,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 183.44782855657667,
+      "server_utilization": 0.012283640424343933,
+      "server_spec_accept_rate": 0.6352201257861635,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1.9,
+      "max_running_reqs": 2,
+      "effective_concurrency": 1.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.553,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.958,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.59,
+        "cpu_temp_max_c": 76.75,
+        "gpu_util_avg_pct": 99.81,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 37.69,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 68.81,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1163.89,
+        "power_total_max_w": 1178.06,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 37319.5,
+        "pcie_rx_max_mb_s": 67250.0,
+        "pcie_tx_avg_mb_s": 33538.0,
+        "pcie_tx_max_mb_s": 67525.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.987172,
+      "measurement_wall_seconds": 20.001291,
+      "client_output_tokens": 4295,
+      "server_output_tokens": 4295,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 214.8878284941506,
+      "per_request_avg_tps": 53.72195712353765,
+      "ttft_avg": 1.7763419252975534,
+      "ttft_p50": 1.3619820600142702,
+      "ttft_p90": 2.3120769894914703,
+      "ttft_p99": 4.039579109621701,
+      "time_to_second_token_avg": 0.03882854864544546,
+      "time_to_second_token_p50": 0.04027673741802573,
+      "time_to_second_token_p90": 0.043103844858706,
+      "time_to_second_token_p99": 0.043685059298295525,
+      "request_latency_avg": 9.80318702897057,
+      "request_latency_p50": 9.880283242091537,
+      "request_latency_p90": 10.619067505188287,
+      "request_latency_p99": 11.727601487580687,
+      "inter_token_latency_avg": 0.015157077239068617,
+      "inter_token_latency_p50": 0.01507125252269121,
+      "inter_token_latency_p90": 0.016277488439941666,
+      "inter_token_latency_p99": 0.01681107227458143,
+      "output_tps_per_user_avg": 66.20683975505182,
+      "output_tps_per_user_p50": 66.35252027526646,
+      "output_tps_per_user_p90": 70.24470636649606,
+      "output_tps_per_user_p99": 74.73678565140237,
+      "e2e_output_tps_per_user_avg": 52.8389794149083,
+      "e2e_output_tps_per_user_p50": 51.82037674980822,
+      "e2e_output_tps_per_user_p90": 58.62424331130355,
+      "e2e_output_tps_per_user_p99": 64.67365860038261,
+      "chunk_inter_token_latency_avg": 0.04252216480840284,
+      "chunk_inter_token_latency_p50": 0.042666952384107254,
+      "chunk_inter_token_latency_p90": 0.044934111945601436,
+      "chunk_inter_token_latency_p99": 0.04507588112417429,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 12,
+      "completed_request_count": 9,
+      "request_samples": [
+        {
+          "ttft": 0.5923555630724877,
+          "time_to_second_token": 0.017299477010965347,
+          "latency": 7.835237701190636,
+          "inter_token_latency_avg": 0.01417393764798072,
+          "chunk_inter_token_latency_avg": 0.04069034909055139,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 70.55202476797012,
+          "e2e_output_tps_per_user": 65.34581585472473,
+          "completed": true
+        },
+        {
+          "ttft": 1.2594965409953147,
+          "time_to_second_token": 0.03946907399222255,
+          "latency": 8.991313345031813,
+          "inter_token_latency_avg": 0.01513075695506164,
+          "chunk_inter_token_latency_avg": 0.042717219911803855,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 66.09054675651726,
+          "e2e_output_tps_per_user": 56.943850175448254,
+          "completed": true
+        },
+        {
+          "ttft": 1.2591414290945977,
+          "time_to_second_token": 0.04004574380815029,
+          "latency": 9.880283242091537,
+          "inter_token_latency_avg": 0.0168711190078218,
+          "chunk_inter_token_latency_avg": 0.044901780276025725,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 59.27289111862582,
+          "e2e_output_tps_per_user": 51.82037674980822,
+          "completed": true
+        },
+        {
+          "ttft": 2.212953792186454,
+          "time_to_second_token": 0.04086994403041899,
+          "latency": 9.785697993123904,
+          "inter_token_latency_avg": 0.014819460275807142,
+          "chunk_inter_token_latency_avg": 0.04160848462053544,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 67.47884075322945,
+          "e2e_output_tps_per_user": 52.32125499476542,
+          "completed": true
+        },
+        {
+          "ttft": 1.4577314089983702,
+          "time_to_second_token": 0.043352056061849,
+          "latency": 9.128734683152288,
+          "inter_token_latency_avg": 0.015011748090320779,
+          "chunk_inter_token_latency_avg": 0.04261668485641065,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 66.61449379401566,
+          "e2e_output_tps_per_user": 56.08663388420428,
+          "completed": true
+        },
+        {
+          "ttft": 1.2618389299605042,
+          "time_to_second_token": 0.04062130395323038,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.015226825442038138,
+          "chunk_inter_token_latency_avg": 0.04493770435333207,
+          "input_tokens": 8192,
+          "output_tokens": 485,
+          "output_tps_per_user": 65.67357088360686,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2129524589981884,
+          "time_to_second_token": 0.04079655185341835,
+          "latency": 10.311141398968175,
+          "inter_token_latency_avg": 0.015847727866868857,
+          "chunk_inter_token_latency_avg": 0.04307547308494674,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 63.100528252418606,
+          "e2e_output_tps_per_user": 49.65502655712153,
+          "completed": true
+        },
+        {
+          "ttft": 2.3230906780809164,
+          "time_to_second_token": 0.04372621700167656,
+          "latency": 10.149147124029696,
+          "inter_token_latency_avg": 0.01531517895488998,
+          "chunk_inter_token_latency_avg": 0.04372098573155743,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 65.29469900060882,
+          "e2e_output_tps_per_user": 50.447588722776494,
+          "completed": true
+        },
+        {
+          "ttft": 1.2644218259956688,
+          "time_to_second_token": 0.03977838600985706,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.015003678621124169,
+          "chunk_inter_token_latency_avg": 0.04158162360711556,
+          "input_tokens": 8192,
+          "output_tokens": 389,
+          "output_tps_per_user": 66.65032124802163,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 4.251729365205392,
+          "time_to_second_token": 0.03976707882247865,
+          "latency": 11.850771930068731,
+          "inter_token_latency_avg": 0.014870924784468375,
+          "chunk_inter_token_latency_avg": 0.04175298112562274,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 67.24531355604925,
+          "e2e_output_tps_per_user": 43.203936673602875,
+          "completed": true
+        },
+        {
+          "ttft": 1.9541583999525756,
+          "time_to_second_token": 0.03970902017317712,
+          "latency": 10.296355843078345,
+          "inter_token_latency_avg": 0.016325239614727535,
+          "chunk_inter_token_latency_avg": 0.04509295915203119,
+          "input_tokens": 8192,
+          "output_tokens": 512,
+          "output_tps_per_user": 61.254843640877844,
+          "e2e_output_tps_per_user": 49.726331121722886,
+          "completed": true
+        },
+        {
+          "ttft": 1.2662327110301703,
+          "time_to_second_token": 0.04050773102790117,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013288329607714268,
+          "chunk_inter_token_latency_avg": 0.037569731890901244,
+          "input_tokens": 8192,
+          "output_tokens": 312,
+          "output_tps_per_user": 75.25400328868051,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4295,
+      "wall_time": 28.98594278888777,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 214.68362406405674,
+      "server_utilization": 0.008933556672250154,
+      "server_spec_accept_rate": 0.603448275862069,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 3.9,
+      "max_running_reqs": 4,
+      "effective_concurrency": 3.9,
+      "avg_queue_reqs": 0.1,
+      "max_queue_reqs": 1,
+      "queue_fraction": 0.05,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 8.579,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": true,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.84,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.51,
+        "cpu_temp_max_c": 76.62,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 31.03,
+        "mem_util_max_pct": 46.0,
+        "temp_avg_c": 69.19,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1159.62,
+        "power_total_max_w": 1173.08,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 9737.0,
+        "pcie_rx_max_mb_s": 24076.0,
+        "pcie_tx_avg_mb_s": 7863.75,
+        "pcie_tx_max_mb_s": 8476.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.995457,
+      "measurement_wall_seconds": 20.000576,
+      "client_output_tokens": 3727,
+      "server_output_tokens": 3727,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 186.39233603677485,
+      "per_request_avg_tps": 93.19616801838743,
+      "ttft_avg": 1.1562239681370556,
+      "ttft_p50": 1.213384915026836,
+      "ttft_p90": 1.3303045589243994,
+      "ttft_p99": 1.4650318499957211,
+      "time_to_second_token_avg": 0.01826943955384195,
+      "time_to_second_token_p50": 0.020064805983565748,
+      "time_to_second_token_p90": 0.021722023980692028,
+      "time_to_second_token_p99": 0.021918757599778474,
+      "request_latency_avg": 5.276185235736193,
+      "request_latency_p50": 5.3531620495487005,
+      "request_latency_p90": 5.51592188810464,
+      "request_latency_p99": 5.72075475651538,
+      "inter_token_latency_avg": 0.008087649016423303,
+      "inter_token_latency_p50": 0.00809587391185412,
+      "inter_token_latency_p90": 0.008533725732085115,
+      "inter_token_latency_p99": 0.009295477393811626,
+      "output_tps_per_user_avg": 124.29876816215655,
+      "output_tps_per_user_p50": 123.51971941775874,
+      "output_tps_per_user_p90": 131.3783276316068,
+      "output_tps_per_user_p99": 142.50807229575008,
+      "e2e_output_tps_per_user_avg": 97.32706467298749,
+      "e2e_output_tps_per_user_p50": 95.64440520102602,
+      "e2e_output_tps_per_user_p90": 105.49679616444378,
+      "e2e_output_tps_per_user_p99": 106.19561216563397,
+      "chunk_inter_token_latency_avg": 0.02302849329169416,
+      "chunk_inter_token_latency_p50": 0.02311558809813452,
+      "chunk_inter_token_latency_p90": 0.02417786236213047,
+      "chunk_inter_token_latency_p99": 0.02524273630673258,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 10,
+      "completed_request_count": 8,
+      "request_samples": [
+        {
+          "ttft": 0.6123396621551365,
+          "time_to_second_token": 0.007417730987071991,
+          "latency": 5.405579176964238,
+          "inter_token_latency_avg": 0.009380116467336793,
+          "chunk_inter_token_latency_avg": 0.02536105563391059,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 106.60848439165707,
+          "e2e_output_tps_per_user": 94.71695506410806,
+          "completed": true
+        },
+        {
+          "ttft": 1.4800015490036458,
+          "time_to_second_token": 0.0212986059486866,
+          "latency": 5.743513964116573,
+          "inter_token_latency_avg": 0.008343468522725885,
+          "chunk_inter_token_latency_avg": 0.023046013054664475,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 119.85423056085207,
+          "e2e_output_tps_per_user": 89.14403328672888,
+          "completed": true
+        },
+        {
+          "ttft": 1.2117794880177826,
+          "time_to_second_token": 0.018087500939145684,
+          "latency": 5.4183824269566685,
+          "inter_token_latency_avg": 0.00823209968481191,
+          "chunk_inter_token_latency_avg": 0.02311320296120267,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 121.4756912923423,
+          "e2e_output_tps_per_user": 94.49314567624086,
+          "completed": true
+        },
+        {
+          "ttft": 1.2173947461415082,
+          "time_to_second_token": 0.021697735879570246,
+          "latency": 5.353260674979538,
+          "inter_token_latency_avg": 0.008093671093616497,
+          "chunk_inter_token_latency_avg": 0.023105396250491784,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 123.55332807985033,
+          "e2e_output_tps_per_user": 95.64264307042306,
+          "completed": true
+        },
+        {
+          "ttft": 1.2256406019441783,
+          "time_to_second_token": 0.02125983708538115,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.0076920541456072695,
+          "chunk_inter_token_latency_avg": 0.020978329488019826,
+          "input_tokens": 32768,
+          "output_tokens": 181,
+          "output_tps_per_user": 130.00428508047798,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.3136715600267053,
+          "time_to_second_token": 0.019344910979270935,
+          "latency": 4.868584974901751,
+          "inter_token_latency_avg": 0.006956777719912027,
+          "chunk_inter_token_latency_avg": 0.020911255381617914,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 143.744710591766,
+          "e2e_output_tps_per_user": 105.16402664006749,
+          "completed": true
+        },
+        {
+          "ttft": 0.8629559089895338,
+          "time_to_second_token": 0.013778487918898463,
+          "latency": 4.81776890787296,
+          "inter_token_latency_avg": 0.007739360076092811,
+          "chunk_inter_token_latency_avg": 0.02340126034842264,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 129.2096491399902,
+          "e2e_output_tps_per_user": 106.27325838798845,
+          "completed": true
+        },
+        {
+          "ttft": 1.2149462150409818,
+          "time_to_second_token": 0.017084267921745777,
+          "latency": 5.353063424117863,
+          "inter_token_latency_avg": 0.008098076730091745,
+          "chunk_inter_token_latency_avg": 0.023117973235066376,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 123.48611075566714,
+          "e2e_output_tps_per_user": 95.64616733162899,
+          "completed": true
+        },
+        {
+          "ttft": 1.2118236150126904,
+          "time_to_second_token": 0.02194061689078808,
+          "latency": 5.249328335979953,
+          "inter_token_latency_avg": 0.007901183406980945,
+          "chunk_inter_token_latency_avg": 0.023204050120501512,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 126.56331950432494,
+          "e2e_output_tps_per_user": 97.53628792671415,
+          "completed": true
+        },
+        {
+          "ttft": 1.2116863350383937,
+          "time_to_second_token": 0.02078470098786056,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008439682317057152,
+          "chunk_inter_token_latency_avg": 0.02404639644304379,
+          "input_tokens": 32768,
+          "output_tokens": 360,
+          "output_tps_per_user": 118.48787222463746,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3727,
+      "wall_time": 25.575839941157028,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 186.28349244400528,
+      "server_utilization": 0.013121161362367406,
+      "server_spec_accept_rate": 0.55,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1.9,
+      "max_running_reqs": 2,
+      "effective_concurrency": 1.9,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.559,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.296,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.52,
+        "cpu_temp_max_c": 76.75,
+        "gpu_util_avg_pct": 99.89,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 38.5,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 68.69,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1159.62,
+        "power_total_max_w": 1178.78,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 18465.67,
+        "pcie_rx_max_mb_s": 56770.0,
+        "pcie_tx_avg_mb_s": 18393.11,
+        "pcie_tx_max_mb_s": 51076.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.996813,
+      "measurement_wall_seconds": 20.000936,
+      "client_output_tokens": 4259,
+      "server_output_tokens": 4259,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 212.98394100240475,
+      "per_request_avg_tps": 53.24598525060119,
+      "ttft_avg": 1.6594751148368232,
+      "ttft_p50": 1.2507346520433202,
+      "ttft_p90": 2.214244986465201,
+      "ttft_p99": 3.976403836400715,
+      "time_to_second_token_avg": 0.03094565647188574,
+      "time_to_second_token_p50": 0.03346484643407166,
+      "time_to_second_token_p90": 0.03496504717040807,
+      "time_to_second_token_p99": 0.035161831779405475,
+      "request_latency_avg": 9.732150099524814,
+      "request_latency_p50": 9.580612420104444,
+      "request_latency_p90": 11.209756568027661,
+      "request_latency_p99": 12.458838488040492,
+      "inter_token_latency_avg": 0.015378300425214705,
+      "inter_token_latency_p50": 0.01540227101955366,
+      "inter_token_latency_p90": 0.016775147308305694,
+      "inter_token_latency_p99": 0.016912279111026093,
+      "output_tps_per_user_avg": 65.35975868873867,
+      "output_tps_per_user_p50": 64.9280008387403,
+      "output_tps_per_user_p90": 70.36340913782152,
+      "output_tps_per_user_p99": 74.69250108569062,
+      "e2e_output_tps_per_user_avg": 53.42647790415391,
+      "e2e_output_tps_per_user_p50": 53.44126007285225,
+      "e2e_output_tps_per_user_p90": 58.48582251783244,
+      "e2e_output_tps_per_user_p99": 64.43718529934995,
+      "chunk_inter_token_latency_avg": 0.043139480462465546,
+      "chunk_inter_token_latency_p50": 0.04417947091825035,
+      "chunk_inter_token_latency_p90": 0.04521715045374133,
+      "chunk_inter_token_latency_p99": 0.04545031231990535,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 512.0,
+      "output_seq_len_p50": 512.0,
+      "output_seq_len_p90": 512.0,
+      "output_seq_len_p99": 512.0,
+      "request_count": 12,
+      "completed_request_count": 9,
+      "request_samples": [
+        {
+          "ttft": 0.6141200510319322,
+          "time_to_second_token": 0.011225768830627203,
+          "latency": 7.865010872948915,
+          "inter_token_latency_avg": 0.014189610219015622,
+          "chunk_inter_token_latency_avg": 0.04028272678842768,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 70.474099327964,
+          "e2e_output_tps_per_user": 65.09844783062967,
+          "completed": true
+        },
+        {
+          "ttft": 1.275223245844245,
+          "time_to_second_token": 0.030873473035171628,
+          "latency": 9.194723297841847,
+          "inter_token_latency_avg": 0.015498043154594134,
+          "chunk_inter_token_latency_avg": 0.04525428601141487,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 64.52427509879315,
+          "e2e_output_tps_per_user": 55.68411179052825,
+          "completed": true
+        },
+        {
+          "ttft": 1.2505115061067045,
+          "time_to_second_token": 0.035012715961784124,
+          "latency": 9.598736566957086,
+          "inter_token_latency_avg": 0.016337035344129905,
+          "chunk_inter_token_latency_avg": 0.044882930434679474,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 61.21061618192019,
+          "e2e_output_tps_per_user": 53.3403533296789,
+          "completed": true
+        },
+        {
+          "ttft": 2.2142701919656247,
+          "time_to_second_token": 0.02940852497704327,
+          "latency": 10.862789368024096,
+          "inter_token_latency_avg": 0.016924695060779787,
+          "chunk_inter_token_latency_avg": 0.044351380390043445,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 59.08525952218403,
+          "e2e_output_tps_per_user": 47.13338192003727,
+          "completed": true
+        },
+        {
+          "ttft": 1.250957797979936,
+          "time_to_second_token": 0.03281243494711816,
+          "latency": 9.841799243818969,
+          "inter_token_latency_avg": 0.01681182279029165,
+          "chunk_inter_token_latency_avg": 0.044282687865149654,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 59.4819498440985,
+          "e2e_output_tps_per_user": 52.023007919162325,
+          "completed": true
+        },
+        {
+          "ttft": 1.2194926510564983,
+          "time_to_second_token": 0.03447063802741468,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014444936651048233,
+          "chunk_inter_token_latency_avg": 0.041717839432504976,
+          "input_tokens": 32768,
+          "output_tokens": 388,
+          "output_tps_per_user": 69.22841021441465,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2140181369613856,
+          "time_to_second_token": 0.02934236405417323,
+          "latency": 9.580612420104444,
+          "inter_token_latency_avg": 0.014416035779144928,
+          "chunk_inter_token_latency_avg": 0.04138536114125314,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 69.36719742653926,
+          "e2e_output_tps_per_user": 53.44126007285225,
+          "completed": true
+        },
+        {
+          "ttft": 1.2175294221378863,
+          "time_to_second_token": 0.029971704818308353,
+          "latency": 9.039150352124125,
+          "inter_token_latency_avg": 0.015306498884513187,
+          "chunk_inter_token_latency_avg": 0.04547454029061766,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 65.33172657868745,
+          "e2e_output_tps_per_user": 56.64249183328213,
+          "completed": true
+        },
+        {
+          "ttft": 1.2166177490726113,
+          "time_to_second_token": 0.03411725792102516,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.015614057580944196,
+          "chunk_inter_token_latency_avg": 0.044076253971351044,
+          "input_tokens": 32768,
+          "output_tokens": 495,
+          "output_tps_per_user": 64.04485155866378,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 4.194195635151118,
+          "time_to_second_token": 0.035180261824280024,
+          "latency": 12.597625368041918,
+          "inter_token_latency_avg": 0.016445067970432093,
+          "chunk_inter_token_latency_avg": 0.0444625911793164,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 60.80850512737194,
+          "e2e_output_tps_per_user": 40.64258025158129,
+          "completed": true
+        },
+        {
+          "ttft": 1.2128918368835002,
+          "time_to_second_token": 0.03453602804802358,
+          "latency": 9.008903405861929,
+          "inter_token_latency_avg": 0.015256382718157395,
+          "chunk_inter_token_latency_avg": 0.04355313725686273,
+          "input_tokens": 32768,
+          "output_tokens": 512,
+          "output_tps_per_user": 65.54633680039039,
+          "e2e_output_tps_per_user": 56.83266618963313,
+          "completed": true
+        },
+        {
+          "ttft": 2.033873153850436,
+          "time_to_second_token": 0.03439670521765947,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013295418949525321,
+          "chunk_inter_token_latency_avg": 0.03795003078796548,
+          "input_tokens": 32768,
+          "output_tokens": 295,
+          "output_tps_per_user": 75.21387658383661,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4259,
+      "wall_time": 28.930821696063504,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 212.86426983114268,
+      "server_utilization": 0.02819653824678947,
+      "server_spec_accept_rate": 0.5884057971014492,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 3.8,
+      "max_running_reqs": 4,
+      "effective_concurrency": 3.8,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": true,
+      "warmup_timed_out": false,
+      "warmup_duration": 8.576,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.299,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.51,
+        "cpu_temp_max_c": 76.75,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 32.06,
+        "mem_util_max_pct": 45.0,
+        "temp_avg_c": 69.14,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1161.68,
+        "power_total_max_w": 1174.21,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 35105.33,
+        "pcie_rx_max_mb_s": 74341.0,
+        "pcie_tx_avg_mb_s": 34509.67,
+        "pcie_tx_max_mb_s": 70985.0
+      }
+    }
+  ],
+  "summary_table": {
+    "0": {
+      "1": 191.1473049181911,
+      "2": 257.76658819245483,
+      "4": 305.62106309879454
+    },
+    "8192": {
+      "1": 165.15431387835127,
+      "2": 183.53550127953065,
+      "4": 214.8878284941506
+    },
+    "32768": {
+      "1": 162.35299164317425,
+      "2": 186.39233603677485,
+      "4": 212.98394100240475
+    }
+  },
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": false,
+      "mode": "skipped",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": true,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap512.log

@@ -0,0 +1,126 @@
+
+New version available: v0.6.2 (current: v0.4.29)
+Upgrade and restart? [Y/n]: Skipping update.
+
+╭──────────────────────────── NVIDIA P2P Override ─────────────────────────────╮
+│ Effective: yes                                                               │
+│ Configured file: yes (/etc/modprobe.d/nvidia-p2p-override.conf)              │
+│ Runtime: ForceP2P=0x11; RMForceP2PType=1; RMPcieP2PType=2;                   │
+│ GrdmaPciTopoCheckOverride=1; EnableResizableBar=1; DmaRemapPeerMmio=1        │
+╰──────────────────────────────────────────────────────────────────────────────╯
+╭─────────────────────────────── Configuration ────────────────────────────────╮
+│ LLM Inference Benchmark                                                      │
+│ Model: glm53-flash-trellismx-p8-k45 @ 127.0.0.1:8001                         │
+│ Decode concurrency: [1, 2, 4]                                                │
+│ Decode contexts: ['0', '8k', '32k']                                          │
+│ Duration: 20.0s per decode test | Max tokens: 512                            │
+│ Pre-decode warmup: C=1 max-runnable context for 3s                           │
+│ Prefill: skipped | Sustained decode: 9 cells                                 │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Engine: vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f  
+Models: ['glm53-flash-trellismx-p8-k45']
+KV cache budget (vLLM metrics): 29,351,936 tokens (3583 blocks × 2048; local 
+7,337,984 × CP 4; CP source: local process)
+Model context length: 1,000,000 tokens
+Prefill tests: skipped
+Calibrating padding text (run=mkkehtszimwn, up to 32k)...
+  8k: 50,558 chars (8,192 prompt tokens via /tokenize)
+  32k: 205,152 chars (32,768 prompt tokens via /tokenize)
+  Token targeting: /tokenize exact
+Done.
+
+
+
+llm-decode-bench v0.4.29
+╭────────────────────────────────── Phase 2 ───────────────────────────────────╮
+│ Sustained Decode                                                             │
+│ Steady-state decode throughput after the engine has admitted the requested   │
+│ concurrency and passed warmup. Use this as the main tuning/regression signal │
+│ for kernels, NCCL, DCP, MTP, and scheduler changes.                          │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Aggregate tok/s  + TTFT/ITL                                        
+╭────────────┬─────────────┬──────────────────┬───────────────────╮
+│ ctx \ conc │           1 │                2 │                 4 │
+├────────────┼─────────────┼──────────────────┼───────────────────┤
+│ 0          │  191.1 80/5 │      257.8 123/8 │   ∅ (4/4)* 182/13 │
+│ 8k         │ 165.2 629/5 │ 183.5 (2/2) 1k/8 │    ∅ (4/4)* 1k/15 │
+│ 32k        │ 162.4 630/5 │ 186.4 (2/2) 1k/8 │ 213.0 (4/4) 1k/15 │
+╰────────────┴─────────────┴──────────────────┴───────────────────╯
+Sustained Decode: aggregate tok/s uses OpenAI stream usage by default 
+(continuous completion_tokens when the server supports it). Prometheus is kept 
+as validation/scheduler data.
+Aggregate source(s): openai_continuous_usage
+∅ = skipped/hidden because the cell does not fit in KV cache; exact deficit is 
+kept in JSON timeout_reason
+(X/Y) = avg running / requested concurrency from Prometheus; * = 
+capacity-limited or warmup timed out
+Per-Request tok/s                               
+╭────────────┬───────┬────────────┬────────────╮
+│ ctx \ conc │     1 │          2 │          4 │
+├────────────┼───────┼──────���─────┼────────────┤
+│ 0          │ 191.1 │      128.9 │   ∅ (4/4)* │
+│ 8k         │ 165.2 │ 91.8 (2/2) │   ∅ (4/4)* │
+│ 32k        │ 162.4 │ 93.2 (2/2) │ 53.2 (4/4) │
+╰────────────┴───────┴────────────┴────────────╯
+Client request latency: p50 / p90 ms               
+╭────────────┬───────────┬───────────┬────────────╮
+│ ctx \ conc │         1 │         2 │          4 │
+├────────────┼───────────┼───────────┼────────────┤
+│ 0          │ 2.6k/2.8k │   4k/4.3k │  6.8k/7.1k │
+│ 8k         │ 3.2k/3.2k │ 5.4k/5.7k │ 9.9k/10.6k │
+│ 32k        │ 3.2k/3.3k │ 5.4k/5.5k │ 9.6k/11.2k │
+╰────────────┴───────────┴───────────┴────────────╯
+Aggregate cells show dim detail as TTFT ms / ITL ms for the same ctx/conc 
+coordinate. ITL is computed from observed generated tokens, including streams 
+stopped at the measurement boundary; a missing ITL means no stream produced at 
+least two measured output tokens. Per-request tok/s and request latency are 
+shown in separate per-cell matrices. Completion/sample counts and full 
+request-level distributions remain in JSON under request_samples.
+Sustained mode: client latency metrics explain request UX variance; aggregate 
+tok/s remains the primary throughput signal. 
+ITL=(last_token_time-first_token_time)/(output_tokens-1), user tok/s=1/ITL.
+Hardware Summary                                                                
+╭───┬─┬───────┬───────────┬───────┬─────────┬─────┬──────┬─────┬───────────────╮
+│ … │ │ mode  │ GPU avg/… │ Mem … │ W avg/… │ T … │ CPU… │ VR… │ PCIe rx/tx a… │
+├───┼─┼───────┼───────────┼───────┼─────────┼─────┼──────┼─────┼───────────────┤
+│ 0 │ │ sust… │   99/100% │   43% │ 1153/1… │ 83C │  76C │ 98… │     8389/8246 │
+│ … │ │ sust… │   99/100% │   40% │ 1154/1… │ 84C │  76C │ 98… │    11118/9898 │
+│ … │ │ sust… │   96/100% │   39% │ 1152/1… │ 84C │  77C │ 98… │   12769/12689 │
+│ 0 │ │ sust… │  100/100% │   41% │ 1175/1… │ 84C │  76C │ 98… │   11425/11473 │
+│ 0 │ │ sust… │  100/100% │   36% │ 1172/1… │ 84C │  77C │ 98… │     8124/8457 │
+│ … │ │ sust… │  100/100% │   38% │ 1164/1… │ 84C │  77C │ 98… │   37320/33538 │
+│ … │ │ sust… │  100/100% │   31% │ 1160/1… │ 84C │  77C │ 98… │     9737/7864 │
+│ … │ │ sust… │  100/100% │   38% │ 1160/1… │ 84C │  77C │ 98… │   18466/18393 │
+│ … │ │ sust… │  100/100% │   32% │ 1162/1… │ 84C │  77C │ 98… │   35105/34510 │
+╰───┴─┴───────┴───────────┴───────┴─────────┴─────┴──────┴─────┴───────────────╯
+╭───────────────────────── Whole-run GPU Power ─────────────────────────╮
+│ avg 1,097 W | max 1,179 W | limit 1,200 W | over 4m 29s | 113 samples │
+╰───────────────────────────────────────────────────────────────────────╯
+Hardware summary is sampled from nvidia-smi during the measured part of each 
+cell. Whole-run GPU power is the sampled sum of GPU power draw across the 
+complete benchmark run, not wall-outlet system power. PCIe rx/tx is MB/s and is 
+a coarse live diagnostic, not a per-kernel NCCL profiler.
+
+╭────────────────────────────────── Phase 3 ───────────────────────────────────╮
+│ Burst / E2E Decode                                                           │
+│ Not run. Re-run with --run-burst to append a finite client-facing request    │
+│ burst after Sustained Decode. This is intentionally disabled by default      │
+│ because it adds another full decode matrix.                                  │
+╰───────���──────────────────────────────────────────────────────────────────────╯
+
+╭────────────────────────────── Primary Summary ───────────────────────────────╮
+│ Primary matrices repeated last so the important numbers are visible without  │
+│ scrolling back through diagnostics.                                          │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Aggregate decode tok/s                            
+╭────────────┬───────┬─────────────┬─────────────╮
+│ ctx \ conc │     1 │           2 │           4 │
+├────────────┼───────┼─────────────┼─────────────┤
+│ 0          │ 191.1 │       257.8 │    ∅ (4/4)* │
+│ 8k         │ 165.2 │ 183.5 (2/2) │    ∅ (4/4)* │
+│ 32k        │ 162.4 │ 186.4 (2/2) │ 213.0 (4/4) │
+╰────────────┴───────┴─────────────┴─────────────╯
+
+Results saved to 
+<campaign>/candidate-speed-wi
+ndow-01/results-01/decode-warp-quant/rep-2/decode-cap512.json

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap8192-command.json

@@ -0,0 +1,27 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "8192",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap8192.json",
+  "--contexts",
+  "0,8k,32k",
+  "--concurrency",
+  "1,2,4",
+  "--skip-prefill",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap8192-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "c97b1e6c81137e30eeaae4e137a8605c633dbe6eaa2b5037e53dc6f63dbeae19"
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap8192.json

@@ -0,0 +1,1394 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T02:33:25.169970",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "skipped",
+    "standalone_prefill": false,
+    "prefill_only": false,
+    "skip_prefill": true,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 32768,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 29351936,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0,
+      8192,
+      32768
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0,8k,32k",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": false,
+      "prefill_only": false,
+      "skip_prefill": true,
+      "prefill_contexts": "8k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 113,
+    "duration_seconds": 269.983,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 11.42,
+    "cpu_temp_max_c": 77.38,
+    "gpu_util_avg_pct": 89.04,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 34.99,
+    "mem_util_max_pct": 56.0,
+    "temp_avg_c": 67.89,
+    "temp_max_c": 85.0,
+    "power_total_avg_w": 1105.19,
+    "power_total_max_w": 1178.45,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 384778.0,
+    "vram_used_max_mb": 384778.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.27,
+    "vram_used_max_pct": 98.27,
+    "pcie_rx_avg_mb_s": 12268.5,
+    "pcie_rx_max_mb_s": 68298.0,
+    "pcie_tx_avg_mb_s": 12130.98,
+    "pcie_tx_max_mb_s": 63483.0
+  },
+  "event_log": [
+    "02:28:53 benchmark start engine=vllm",
+    "02:28:53 startup server=http://127.0.0.1:8001 model=glm53-flash-trellismx-p8-k45",
+    "02:28:53 startup decode concurrency=1,2,4 contexts=0,8k,32k",
+    "02:28:53 startup NVIDIA P2P override: enabled: runtime NVIDIA P2P override matches expected RegistryDwords",
+    "02:28:53 startup engine vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f models=['glm53-flash-trellismx-p8-k45']",
+    "02:28:53 startup KV cache budget from vLLM metrics: 29,351,936 tokens (3583 blocks x 2048; local 7,337,984 \u00d7 CP 4; CP source: local process)",
+    "02:28:53 startup model context length: 1,000,000 tokens",
+    "02:28:53 startup prefill tests: skipped",
+    "02:28:53 startup calibrating padding text run=lijekedoguqy up_to=32k",
+    "02:28:53 startup context 8k: 50,540 chars (8,192 prompt tokens via /tokenize)",
+    "02:28:53 startup context 32k: 205,139 chars (32,768 prompt tokens via /tokenize)",
+    "02:28:53 startup token targeting: /tokenize exact",
+    "02:28:53 startup startup preparation done",
+    "02:28:53 hardware monitor interval=2s",
+    "02:28:53 decode warmup start",
+    "02:28:53 decode warmup start C=1 ctx=32k 3s",
+    "02:28:53 cell start C=1 ctx=32k",
+    "02:29:01 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:29:04 cell done C=1 ctx=32k 184.6 tok/s",
+    "02:29:04 decode warmup done C=1 ctx=32k",
+    "02:29:06 cell start C=1 ctx=0",
+    "02:29:12 ready C=1 ctx=0 running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:29:32 cell done C=1 ctx=0 175.2 tok/s",
+    "02:29:34 cell start C=1 ctx=8k",
+    "02:29:40 ready C=1 ctx=8k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:30:00 cell done C=1 ctx=8k 174.2 tok/s",
+    "02:30:02 cell start C=1 ctx=32k",
+    "02:30:11 ready C=1 ctx=32k running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+    "02:30:31 cell done C=1 ctx=32k 179.3 tok/s",
+    "02:30:33 cell start C=2 ctx=0",
+    "02:30:39 ready C=2 ctx=0 running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:30:59 cell done C=2 ctx=0 236.4 tok/s",
+    "02:31:01 cell start C=4 ctx=0",
+    "02:31:06 ready C=4 ctx=0 running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:31:26 cell done C=4 ctx=0 296.7 tok/s",
+    "02:31:28 cell start C=2 ctx=8k",
+    "02:31:34 ready C=2 ctx=8k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:31:54 cell done C=2 ctx=8k 246.8 tok/s",
+    "02:31:56 cell start C=4 ctx=8k",
+    "02:32:04 ready C=4 ctx=8k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:32:24 cell done C=4 ctx=8k 291.4 tok/s",
+    "02:32:26 cell start C=2 ctx=32k",
+    "02:32:32 ready C=2 ctx=32k running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+    "02:32:52 cell done C=2 ctx=32k 239.9 tok/s",
+    "02:32:54 cell start C=4 ctx=32k",
+    "02:33:03 ready C=4 ctx=32k running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+    "02:33:23 cell done C=4 ctx=32k 302.2 tok/s"
+  ],
+  "prefill": {},
+  "results": [
+    {
+      "concurrency": 1,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.996529,
+      "measurement_wall_seconds": 20.000654,
+      "client_output_tokens": 3503,
+      "server_output_tokens": 3503,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 175.1804006140207,
+      "per_request_avg_tps": 175.1804006140207,
+      "ttft_avg": 0.06969588715583086,
+      "ttft_p50": 0.06969588715583086,
+      "ttft_p90": 0.06969588715583086,
+      "ttft_p99": 0.06969588715583086,
+      "time_to_second_token_avg": 0.013065006816759706,
+      "time_to_second_token_p50": 0.013065006816759706,
+      "time_to_second_token_p90": 0.013065006816759706,
+      "time_to_second_token_p99": 0.013065006816759706,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005635859511267837,
+      "inter_token_latency_p50": 0.005635859511267837,
+      "inter_token_latency_p90": 0.005635859511267837,
+      "inter_token_latency_p99": 0.005635859511267837,
+      "output_tps_per_user_avg": 177.43522492011888,
+      "output_tps_per_user_p50": 177.43522492011888,
+      "output_tps_per_user_p90": 177.43522492011888,
+      "output_tps_per_user_p99": 177.43522492011888,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.01424891621259546,
+      "chunk_inter_token_latency_p50": 0.01424891621259546,
+      "chunk_inter_token_latency_p90": 0.01424891621259546,
+      "chunk_inter_token_latency_p99": 0.01424891621259546,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 4519.0,
+      "output_seq_len_p50": 4519.0,
+      "output_seq_len_p90": 4519.0,
+      "output_seq_len_p99": 4519.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.06969588715583086,
+          "time_to_second_token": 0.013065006816759706,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005635859511267837,
+          "chunk_inter_token_latency_avg": 0.01424891621259546,
+          "input_tokens": 78,
+          "output_tokens": 4519,
+          "output_tps_per_user": 177.43522492011888,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3503,
+      "wall_time": 25.547412615967914,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 175.1028092847603,
+      "server_utilization": 0.005862646566164198,
+      "server_spec_accept_rate": 0.4835680751173709,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.536,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.909,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.56,
+        "cpu_temp_max_c": 75.88,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 44.09,
+        "mem_util_max_pct": 56.0,
+        "temp_avg_c": 67.31,
+        "temp_max_c": 82.0,
+        "power_total_avg_w": 1153.01,
+        "power_total_max_w": 1155.24,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8331.0,
+        "pcie_rx_max_mb_s": 8554.0,
+        "pcie_tx_avg_mb_s": 8194.0,
+        "pcie_tx_max_mb_s": 8520.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.992808,
+      "measurement_wall_seconds": 20.000923,
+      "client_output_tokens": 3482,
+      "server_output_tokens": 3482,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 174.16263081468745,
+      "per_request_avg_tps": 174.16263081468745,
+      "ttft_avg": 0.5914782441686839,
+      "ttft_p50": 0.5914782441686839,
+      "ttft_p90": 0.5914782441686839,
+      "ttft_p99": 0.5914782441686839,
+      "time_to_second_token_avg": 0.01696636783890426,
+      "time_to_second_token_p50": 0.01696636783890426,
+      "time_to_second_token_p90": 0.01696636783890426,
+      "time_to_second_token_p99": 0.01696636783890426,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005593080112173436,
+      "inter_token_latency_p50": 0.005593080112173436,
+      "inter_token_latency_p90": 0.005593080112173436,
+      "inter_token_latency_p99": 0.005593080112173436,
+      "output_tps_per_user_avg": 178.79236126503582,
+      "output_tps_per_user_p50": 178.79236126503582,
+      "output_tps_per_user_p90": 178.79236126503582,
+      "output_tps_per_user_p99": 178.79236126503582,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.014262687604284941,
+      "chunk_inter_token_latency_p50": 0.014262687604284941,
+      "chunk_inter_token_latency_p90": 0.014262687604284941,
+      "chunk_inter_token_latency_p99": 0.014262687604284941,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 4280.0,
+      "output_seq_len_p50": 4280.0,
+      "output_seq_len_p90": 4280.0,
+      "output_seq_len_p99": 4280.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5914782441686839,
+          "time_to_second_token": 0.01696636783890426,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005593080112173436,
+          "chunk_inter_token_latency_avg": 0.014262687604284941,
+          "input_tokens": 8192,
+          "output_tokens": 4280,
+          "output_tps_per_user": 178.79236126503582,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3482,
+      "wall_time": 26.0623870131094,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 174.05206214713238,
+      "server_utilization": 0.006141820212172022,
+      "server_spec_accept_rate": 0.5047619047619047,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 6.055,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.885,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.53,
+        "cpu_temp_max_c": 76.62,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 43.56,
+        "mem_util_max_pct": 55.0,
+        "temp_avg_c": 67.78,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1153.83,
+        "power_total_max_w": 1155.28,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8357.0,
+        "pcie_rx_max_mb_s": 8584.0,
+        "pcie_tx_avg_mb_s": 8151.12,
+        "pcie_tx_max_mb_s": 8441.0
+      }
+    },
+    {
+      "concurrency": 1,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.987041,
+      "measurement_wall_seconds": 20.00121,
+      "client_output_tokens": 3584,
+      "server_output_tokens": 3586,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 179.316188111569,
+      "per_request_avg_tps": 179.316188111569,
+      "ttft_avg": 0.6033541450742632,
+      "ttft_p50": 0.6033541450742632,
+      "ttft_p90": 0.6033541450742632,
+      "ttft_p99": 0.6033541450742632,
+      "time_to_second_token_avg": 0.012713581090793014,
+      "time_to_second_token_p50": 0.012713581090793014,
+      "time_to_second_token_p90": 0.012713581090793014,
+      "time_to_second_token_p99": 0.012713581090793014,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.005507921420528074,
+      "inter_token_latency_p50": 0.005507921420528074,
+      "inter_token_latency_p90": 0.005507921420528074,
+      "inter_token_latency_p99": 0.005507921420528074,
+      "output_tps_per_user_avg": 181.55669328777836,
+      "output_tps_per_user_p50": 181.55669328777836,
+      "output_tps_per_user_p90": 181.55669328777836,
+      "output_tps_per_user_p99": 181.55669328777836,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.014389527561933152,
+      "chunk_inter_token_latency_p50": 0.014389527561933152,
+      "chunk_inter_token_latency_p90": 0.014389527561933152,
+      "chunk_inter_token_latency_p99": 0.014389527561933152,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 4343.0,
+      "output_seq_len_p50": 4343.0,
+      "output_seq_len_p90": 4343.0,
+      "output_seq_len_p99": 4343.0,
+      "request_count": 1,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.6033541450742632,
+          "time_to_second_token": 0.012713581090793014,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.005507921420528074,
+          "chunk_inter_token_latency_avg": 0.014389527561933152,
+          "input_tokens": 32768,
+          "output_tokens": 4343,
+          "output_tps_per_user": 181.55669328777836,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 3584,
+      "wall_time": 29.107660971814767,
+      "num_completed": 1,
+      "num_errors": 0,
+      "server_gen_throughput": 179.23767853872448,
+      "server_utilization": 0.006979341150195384,
+      "server_spec_accept_rate": 0.46190476190476193,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 1,
+      "max_running_reqs": 1,
+      "effective_concurrency": 1,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 9.101,
+      "ready_reason": "running_reqs=1/1, queue_reqs=0, active_streams=1/1, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.891,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.56,
+        "cpu_temp_max_c": 75.88,
+        "gpu_util_avg_pct": 99.0,
+        "gpu_util_max_pct": 99.0,
+        "mem_util_avg_pct": 44.0,
+        "mem_util_max_pct": 56.0,
+        "temp_avg_c": 68.16,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1155.58,
+        "power_total_max_w": 1156.94,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8340.12,
+        "pcie_rx_max_mb_s": 8569.0,
+        "pcie_tx_avg_mb_s": 8143.62,
+        "pcie_tx_max_mb_s": 8438.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 20.00082,
+      "measurement_wall_seconds": 20.000837,
+      "client_output_tokens": 4728,
+      "server_output_tokens": 4728,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 236.39031276129717,
+      "per_request_avg_tps": 118.19515638064858,
+      "ttft_avg": 0.11200506216846406,
+      "ttft_p50": 0.11200506216846406,
+      "ttft_p90": 0.1434264814015478,
+      "ttft_p99": 0.15049630072899162,
+      "time_to_second_token_avg": 0.01528907532338053,
+      "time_to_second_token_p50": 0.01528907532338053,
+      "time_to_second_token_p90": 0.017035022121854128,
+      "time_to_second_token_p99": 0.017427860151510686,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.00843379404097247,
+      "inter_token_latency_p50": 0.00843379404097247,
+      "inter_token_latency_p90": 0.008573998497973905,
+      "inter_token_latency_p99": 0.008605544500799228,
+      "output_tps_per_user_avg": 118.62182033895986,
+      "output_tps_per_user_p50": 118.62182033895986,
+      "output_tps_per_user_p90": 120.59380445765525,
+      "output_tps_per_user_p99": 121.03750088436172,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.020858210011846075,
+      "chunk_inter_token_latency_p50": 0.020858210011846075,
+      "chunk_inter_token_latency_p90": 0.02088407432588644,
+      "chunk_inter_token_latency_p99": 0.02088989379654552,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 3017.0,
+      "output_seq_len_p50": 3017.0,
+      "output_seq_len_p90": 3063.4,
+      "output_seq_len_p99": 3073.84,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.07272828812710941,
+          "time_to_second_token": 0.013106641825288534,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008609049612224263,
+          "chunk_inter_token_latency_avg": 0.02089054040439653,
+          "input_tokens": 78,
+          "output_tokens": 2959,
+          "output_tps_per_user": 116.15684019059063,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.15128183620981872,
+          "time_to_second_token": 0.017471508821472526,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008258538469720678,
+          "chunk_inter_token_latency_avg": 0.020825879619295624,
+          "input_tokens": 78,
+          "output_tokens": 3075,
+          "output_tps_per_user": 121.0868004873291,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4728,
+      "wall_time": 25.560176193946972,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 236.33660483401786,
+      "server_utilization": 0.011725293132328285,
+      "server_spec_accept_rate": 0.3958333333333333,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.538,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.86,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.55,
+        "cpu_temp_max_c": 77.12,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.34,
+        "mem_util_max_pct": 50.0,
+        "temp_avg_c": 68.5,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1176.31,
+        "power_total_max_w": 1178.12,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11179.25,
+        "pcie_rx_max_mb_s": 11225.0,
+        "pcie_tx_avg_mb_s": 11087.75,
+        "pcie_tx_max_mb_s": 11179.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 0,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.975466,
+      "measurement_wall_seconds": 20.000669,
+      "client_output_tokens": 5926,
+      "server_output_tokens": 5926,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 296.6639108412111,
+      "per_request_avg_tps": 74.16597771030277,
+      "ttft_avg": 0.1473790240706876,
+      "ttft_p50": 0.17257179808802903,
+      "ttft_p90": 0.17268561611417682,
+      "ttft_p99": 0.17268778499448673,
+      "time_to_second_token_avg": 0.025300663721282035,
+      "time_to_second_token_p50": 0.02924731746315956,
+      "time_to_second_token_p90": 0.029298453708179295,
+      "time_to_second_token_p99": 0.029313364170957357,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.013187405590978741,
+      "inter_token_latency_p50": 0.01321511895063593,
+      "inter_token_latency_p90": 0.01351998806064279,
+      "inter_token_latency_p99": 0.01357511053376785,
+      "output_tps_per_user_avg": 75.87497211232,
+      "output_tps_per_user_p50": 75.68227161840176,
+      "output_tps_per_user_p90": 77.93597879757434,
+      "output_tps_per_user_p99": 78.44750412533332,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.034238496269222964,
+      "chunk_inter_token_latency_p50": 0.03419213106811758,
+      "chunk_inter_token_latency_p90": 0.03441501889490805,
+      "chunk_inter_token_latency_p99": 0.0344653942706813,
+      "input_seq_len_avg": 78.0,
+      "output_seq_len_avg": 1925.25,
+      "output_seq_len_p50": 1918.5,
+      "output_seq_len_p90": 1975.6,
+      "output_seq_len_p99": 1988.56,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.0716844741255045,
+          "time_to_second_token": 0.01339299906976521,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013581235253003969,
+          "chunk_inter_token_latency_avg": 0.03409873140600058,
+          "input_tokens": 78,
+          "output_tokens": 1874,
+          "output_tps_per_user": 73.63100493961437,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.17267999309115112,
+          "time_to_second_token": 0.029315020889043808,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013053159956138491,
+          "chunk_inter_token_latency_avg": 0.034284416068829246,
+          "input_tokens": 78,
+          "output_tokens": 1942,
+          "output_tps_per_user": 76.60980202190285,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.17268802598118782,
+          "time_to_second_token": 0.029259796952828765,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012738149209639133,
+          "chunk_inter_token_latency_avg": 0.034470991534656104,
+          "input_tokens": 78,
+          "output_tokens": 1990,
+          "output_tps_per_user": 78.50434027286211,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 0.17246360308490694,
+          "time_to_second_token": 0.029234837973490357,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.01337707794513337,
+          "chunk_inter_token_latency_avg": 0.034099846067405924,
+          "input_tokens": 78,
+          "output_tokens": 1895,
+          "output_tps_per_user": 74.75474121490065,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5926,
+      "wall_time": 25.54523406806402,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 296.1937254544746,
+      "server_utilization": 0.02345058626465657,
+      "server_spec_accept_rate": 0.5028735632183908,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.534,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.831,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.49,
+        "cpu_temp_max_c": 77.38,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 35.0,
+        "mem_util_max_pct": 43.0,
+        "temp_avg_c": 69.0,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1172.77,
+        "power_total_max_w": 1173.84,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 8004.38,
+        "pcie_rx_max_mb_s": 8287.0,
+        "pcie_tx_avg_mb_s": 7682.5,
+        "pcie_tx_max_mb_s": 7820.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.987602,
+      "measurement_wall_seconds": 20.001775,
+      "client_output_tokens": 4932,
+      "server_output_tokens": 4932,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 246.75295838278893,
+      "per_request_avg_tps": 123.37647919139447,
+      "ttft_avg": 0.9467372725484893,
+      "ttft_p50": 0.9467372725484893,
+      "ttft_p90": 1.2301197802415118,
+      "ttft_p99": 1.293880844472442,
+      "time_to_second_token_avg": 0.02229496242944151,
+      "time_to_second_token_p50": 0.02229496242944151,
+      "time_to_second_token_p90": 0.02584285002667457,
+      "time_to_second_token_p99": 0.026641124736052006,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.008086084075571213,
+      "inter_token_latency_p50": 0.008086084075571213,
+      "inter_token_latency_p90": 0.008137498278031636,
+      "inter_token_latency_p99": 0.008149066473585232,
+      "output_tps_per_user_avg": 123.67706846442331,
+      "output_tps_per_user_p50": 123.67706846442331,
+      "output_tps_per_user_p90": 124.46345131405901,
+      "output_tps_per_user_p99": 124.64038745522704,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.021277465417892594,
+      "chunk_inter_token_latency_p50": 0.021277465417892594,
+      "chunk_inter_token_latency_p90": 0.021564048664052024,
+      "chunk_inter_token_latency_p99": 0.021628529894437896,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 2917.0,
+      "output_seq_len_p50": 2917.0,
+      "output_seq_len_p90": 2970.6,
+      "output_seq_len_p99": 2982.66,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5925091379322112,
+          "time_to_second_token": 0.01786010293290019,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008021816322495684,
+          "chunk_inter_token_latency_avg": 0.021635694475591882,
+          "input_tokens": 8192,
+          "output_tokens": 2984,
+          "output_tps_per_user": 124.66004702646794,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.3009654071647674,
+          "time_to_second_token": 0.026729821925982833,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008150351828646742,
+          "chunk_inter_token_latency_avg": 0.020919236360193307,
+          "input_tokens": 8192,
+          "output_tokens": 2850,
+          "output_tps_per_user": 122.6940899023787,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4932,
+      "wall_time": 25.56168243405409,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 246.50722758411473,
+      "server_utilization": 0.012283640424343933,
+      "server_spec_accept_rate": 0.5763888888888888,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.552,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.901,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.55,
+        "cpu_temp_max_c": 77.25,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.5,
+        "mem_util_max_pct": 50.0,
+        "temp_avg_c": 68.62,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1177.05,
+        "power_total_max_w": 1177.91,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11224.88,
+        "pcie_rx_max_mb_s": 11341.0,
+        "pcie_tx_avg_mb_s": 11103.0,
+        "pcie_tx_max_mb_s": 11202.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 8192,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.99956,
+      "measurement_wall_seconds": 20.000651,
+      "client_output_tokens": 5828,
+      "server_output_tokens": 5828,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 291.40641309659526,
+      "per_request_avg_tps": 72.85160327414881,
+      "ttft_avg": 2.3201283884700388,
+      "ttft_p50": 2.213119035004638,
+      "ttft_p90": 3.644294320815243,
+      "ttft_p99": 4.196257012102286,
+      "time_to_second_token_avg": 0.034855086996685714,
+      "time_to_second_token_p50": 0.040691820555366576,
+      "time_to_second_token_p90": 0.041562757641077044,
+      "time_to_second_token_p99": 0.0415682226838544,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.013789638304529176,
+      "inter_token_latency_p50": 0.013799817255296618,
+      "inter_token_latency_p90": 0.013947021664650167,
+      "inter_token_latency_p99": 0.013999457133027752,
+      "output_tps_per_user_avg": 72.52803020492948,
+      "output_tps_per_user_p50": 72.46477597135552,
+      "output_tps_per_user_p90": 73.40383210307152,
+      "output_tps_per_user_p99": 73.74323179562586,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.0354126605281524,
+      "chunk_inter_token_latency_p50": 0.03551991527773096,
+      "chunk_inter_token_latency_p90": 0.035908490963787884,
+      "chunk_inter_token_latency_p99": 0.03602955673091524,
+      "input_seq_len_avg": 8192.0,
+      "output_seq_len_avg": 1830.25,
+      "output_seq_len_p50": 1837.5,
+      "output_seq_len_p90": 1899.9,
+      "output_seq_len_p99": 1923.3899999999999,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.5966892838478088,
+          "time_to_second_token": 0.01646787696518004,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.014005283296180816,
+          "chunk_inter_token_latency_avg": 0.03604300848281828,
+          "input_tokens": 8192,
+          "output_tokens": 1926,
+          "output_tps_per_user": 71.40162600443048,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2132799359969795,
+          "time_to_second_token": 0.04156882991082966,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013811077857745319,
+          "chunk_inter_token_latency_avg": 0.03544521380274498,
+          "input_tokens": 8192,
+          "output_tokens": 1836,
+          "output_tps_per_user": 72.40564496848414,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.212958134012297,
+          "time_to_second_token": 0.04154858901165426,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013788556652847917,
+          "chunk_inter_token_latency_avg": 0.035594616752716954,
+          "input_tokens": 8192,
+          "output_tokens": 1839,
+          "output_tps_per_user": 72.52390697422692,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 4.25758620002307,
+          "time_to_second_token": 0.039835052099078894,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013553635411342652,
+          "chunk_inter_token_latency_avg": 0.034567803074329405,
+          "input_tokens": 8192,
+          "output_tokens": 1720,
+          "output_tps_per_user": 73.78094287257635,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 5828,
+      "wall_time": 28.611056566005573,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 291.3185246321431,
+      "server_utilization": 0.024567280848687867,
+      "server_spec_accept_rate": 0.4511494252873563,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 8.576,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.82,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.51,
+        "cpu_temp_max_c": 76.88,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 34.94,
+        "mem_util_max_pct": 43.0,
+        "temp_avg_c": 69.09,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1172.67,
+        "power_total_max_w": 1173.43,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 7728.12,
+        "pcie_rx_max_mb_s": 7936.0,
+        "pcie_tx_avg_mb_s": 7832.62,
+        "pcie_tx_max_mb_s": 8312.0
+      }
+    },
+    {
+      "concurrency": 2,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.985443,
+      "measurement_wall_seconds": 20.000566,
+      "client_output_tokens": 4794,
+      "server_output_tokens": 4794,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 239.87458866483988,
+      "per_request_avg_tps": 119.93729433241994,
+      "ttft_avg": 0.9611447914503515,
+      "ttft_p50": 0.9611447914503515,
+      "ttft_p90": 1.2436655103228986,
+      "ttft_p99": 1.3072326720692218,
+      "time_to_second_token_avg": 0.014196591568179429,
+      "time_to_second_token_p50": 0.014196591568179429,
+      "time_to_second_token_p90": 0.017423492693342268,
+      "time_to_second_token_p99": 0.018149545446503906,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.00824110461840657,
+      "inter_token_latency_p50": 0.00824110461840657,
+      "inter_token_latency_p90": 0.008593042280594048,
+      "inter_token_latency_p99": 0.008672228254586231,
+      "output_tps_per_user_avg": 121.68972103528682,
+      "output_tps_per_user_p50": 121.68972103528682,
+      "output_tps_per_user_p90": 126.88649961248755,
+      "output_tps_per_user_p99": 128.0557747923577,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.021374294281339835,
+      "chunk_inter_token_latency_p50": 0.021374294281339835,
+      "chunk_inter_token_latency_p90": 0.02164637385802789,
+      "chunk_inter_token_latency_p99": 0.0217075917627827,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 2865.0,
+      "output_seq_len_p50": 2865.0,
+      "output_seq_len_p90": 2953.0,
+      "output_seq_len_p99": 2972.8,
+      "request_count": 2,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.6079938928596675,
+          "time_to_second_token": 0.010162965161725879,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.008681026696140919,
+          "chunk_inter_token_latency_avg": 0.0217143937521999,
+          "input_tokens": 32768,
+          "output_tokens": 2755,
+          "output_tps_per_user": 115.1937478137859,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 1.3142956900410354,
+          "time_to_second_token": 0.01823021797463298,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.007801182540672222,
+          "chunk_inter_token_latency_avg": 0.021034194810479773,
+          "input_tokens": 32768,
+          "output_tokens": 2975,
+          "output_tps_per_user": 128.18569425678774,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 4794,
+      "wall_time": 25.558490577852353,
+      "num_completed": 2,
+      "num_errors": 0,
+      "server_gen_throughput": 239.6320204448516,
+      "server_utilization": 0.013121161362367406,
+      "server_spec_accept_rate": 0.5555555555555556,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 2,
+      "max_running_reqs": 2,
+      "effective_concurrency": 2,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 5.551,
+      "ready_reason": "running_reqs=2/2, queue_reqs=0, active_streams=2/2, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.321,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.53,
+        "cpu_temp_max_c": 76.88,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 40.97,
+        "mem_util_max_pct": 50.0,
+        "temp_avg_c": 68.61,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1177.52,
+        "power_total_max_w": 1178.4,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 11127.78,
+        "pcie_rx_max_mb_s": 11295.0,
+        "pcie_tx_avg_mb_s": 10971.33,
+        "pcie_tx_max_mb_s": 11178.0
+      }
+    },
+    {
+      "concurrency": 4,
+      "context_tokens": 32768,
+      "benchmark_mode": "duration",
+      "request_count_target": 0,
+      "warmup_request_count": 0,
+      "measurement_seconds": 19.994025,
+      "measurement_wall_seconds": 20.000158,
+      "client_output_tokens": 6042,
+      "server_output_tokens": 6042,
+      "aggregate_source": "openai_continuous_usage",
+      "aggregate_tps": 302.19028317254777,
+      "per_request_avg_tps": 75.54757079313694,
+      "ttft_avg": 2.305535151215736,
+      "ttft_p50": 2.2079672454856336,
+      "ttft_p90": 3.5996002558851616,
+      "ttft_p99": 4.136311722563113,
+      "time_to_second_token_avg": 0.026320659555494785,
+      "time_to_second_token_p50": 0.03037197550293058,
+      "time_to_second_token_p90": 0.03217388866469264,
+      "time_to_second_token_p99": 0.032832831228151914,
+      "request_latency_avg": 0.0,
+      "request_latency_p50": 0.0,
+      "request_latency_p90": 0.0,
+      "request_latency_p99": 0.0,
+      "inter_token_latency_avg": 0.013299206291348701,
+      "inter_token_latency_p50": 0.013299051093053463,
+      "inter_token_latency_p90": 0.013574522517023369,
+      "inter_token_latency_p99": 0.01365233632768284,
+      "output_tps_per_user_avg": 75.22142959253767,
+      "output_tps_per_user_p50": 75.19564602998912,
+      "output_tps_per_user_p90": 76.78903624677658,
+      "output_tps_per_user_p99": 77.24282693804788,
+      "e2e_output_tps_per_user_avg": 0.0,
+      "e2e_output_tps_per_user_p50": 0.0,
+      "e2e_output_tps_per_user_p90": 0.0,
+      "e2e_output_tps_per_user_p99": 0.0,
+      "chunk_inter_token_latency_avg": 0.035561150767445995,
+      "chunk_inter_token_latency_p50": 0.03564201678101365,
+      "chunk_inter_token_latency_p90": 0.036069497146972274,
+      "chunk_inter_token_latency_p99": 0.0361956293509813,
+      "input_seq_len_avg": 32768.0,
+      "output_seq_len_avg": 1899.0,
+      "output_seq_len_p50": 1876.0,
+      "output_seq_len_p90": 1995.4,
+      "output_seq_len_p99": 2033.74,
+      "request_count": 4,
+      "completed_request_count": 0,
+      "request_samples": [
+        {
+          "ttft": 0.6102597839199007,
+          "time_to_second_token": 0.011632640147581697,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013225319178200705,
+          "chunk_inter_token_latency_avg": 0.03620964404031564,
+          "input_tokens": 32768,
+          "output_tokens": 2038,
+          "output_tps_per_user": 75.61254186199908,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2081260830163956,
+          "time_to_second_token": 0.030465519055724144,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013372783007906223,
+          "chunk_inter_token_latency_avg": 0.03574248772917108,
+          "input_tokens": 32768,
+          "output_tokens": 1896,
+          "output_tps_per_user": 74.77875019797916,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 2.2078084079548717,
+          "time_to_second_token": 0.03027843195013702,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.013660982306645003,
+          "chunk_inter_token_latency_avg": 0.035541545832856215,
+          "input_tokens": 32768,
+          "output_tokens": 1856,
+          "output_tps_per_user": 73.20117818420553,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        },
+        {
+          "ttft": 4.195946329971775,
+          "time_to_second_token": 0.03290604706853628,
+          "latency": 0.0,
+          "inter_token_latency_avg": 0.012937740672642875,
+          "chunk_inter_token_latency_avg": 0.03475092546744106,
+          "input_tokens": 32768,
+          "output_tokens": 1806,
+          "output_tps_per_user": 77.29324812596693,
+          "e2e_output_tps_per_user": 0.0,
+          "completed": false
+        }
+      ],
+      "total_tokens": 6042,
+      "wall_time": 28.607491098809987,
+      "num_completed": 4,
+      "num_errors": 0,
+      "server_gen_throughput": 302.0169777555693,
+      "server_utilization": 0.02540480178671134,
+      "server_spec_accept_rate": 0.5201149425287356,
+      "server_spec_accept_length": 0.0,
+      "avg_running_reqs": 4,
+      "max_running_reqs": 4,
+      "effective_concurrency": 4,
+      "avg_queue_reqs": 0,
+      "max_queue_reqs": 0,
+      "queue_fraction": 0.0,
+      "underfilled": false,
+      "warmup_timed_out": false,
+      "warmup_duration": 8.578,
+      "ready_reason": "running_reqs=4/4, queue_reqs=0, active_streams=4/4, stable=3.0s",
+      "timeout_reason": "",
+      "capacity_limited": false,
+      "hardware_summary": {
+        "samples": 8,
+        "duration_seconds": 16.85,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.53,
+        "cpu_temp_max_c": 76.88,
+        "gpu_util_avg_pct": 100.0,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 35.12,
+        "mem_util_max_pct": 44.0,
+        "temp_avg_c": 69.22,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1172.46,
+        "power_total_max_w": 1173.31,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 7907.88,
+        "pcie_rx_max_mb_s": 8173.0,
+        "pcie_tx_avg_mb_s": 7679.75,
+        "pcie_tx_max_mb_s": 8296.0
+      }
+    }
+  ],
+  "summary_table": {
+    "0": {
+      "1": 175.1804006140207,
+      "2": 236.39031276129717,
+      "4": 296.6639108412111
+    },
+    "8192": {
+      "1": 174.16263081468745,
+      "2": 246.75295838278893,
+      "4": 291.40641309659526
+    },
+    "32768": {
+      "1": 179.316188111569,
+      "2": 239.87458866483988,
+      "4": 302.19028317254777
+    }
+  },
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": false,
+      "mode": "skipped",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": true,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/decode-cap8192.log

@@ -0,0 +1,123 @@
+
+New version available: v0.6.2 (current: v0.4.29)
+Upgrade and restart? [Y/n]: Skipping update.
+
+╭──────────────────────────── NVIDIA P2P Override ─────────────────────────────╮
+│ Effective: yes                                                               │
+│ Configured file: yes (/etc/modprobe.d/nvidia-p2p-override.conf)              │
+│ Runtime: ForceP2P=0x11; RMForceP2PType=1; RMPcieP2PType=2;                   │
+│ GrdmaPciTopoCheckOverride=1; EnableResizableBar=1; DmaRemapPeerMmio=1        │
+╰──────────────────────────────────────────────────────────────────────────────╯
+╭─────────────────────────────── Configuration ────────────────────────────────╮
+│ LLM Inference Benchmark                                                      │
+│ Model: glm53-flash-trellismx-p8-k45 @ 127.0.0.1:8001                         │
+│ Decode concurrency: [1, 2, 4]                                                │
+│ Decode contexts: ['0', '8k', '32k']                                          │
+│ Duration: 20.0s per decode test | Max tokens: 8192                           │
+│ Pre-decode warmup: C=1 max-runnable context for 3s                           │
+│ Prefill: skipped | Sustained decode: 9 cells                                 │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Engine: vLLM 0.26.1rc0+glm53.flash.nvfp4.luke.clean.r1.vllme75bcfd.b12x58a046f  
+Models: ['glm53-flash-trellismx-p8-k45']
+KV cache budget (vLLM metrics): 29,351,936 tokens (3583 blocks × 2048; local 
+7,337,984 × CP 4; CP source: local process)
+Model context length: 1,000,000 tokens
+Prefill tests: skipped
+Calibrating padding text (run=lijekedoguqy, up to 32k)...
+  8k: 50,540 chars (8,192 prompt tokens via /tokenize)
+  32k: 205,139 chars (32,768 prompt tokens via /tokenize)
+  Token targeting: /tokenize exact
+Done.
+
+
+
+llm-decode-bench v0.4.29
+╭────────────────────────────────── Phase 2 ───────────────────────────────────╮
+│ Sustained Decode                                                             │
+│ Steady-state decode throughput after the engine has admitted the requested   │
+│ concurrency and passed warmup. Use this as the main tuning/regression signal │
+│ for kernels, NCCL, DCP, MTP, and scheduler changes.                          │
+╰──────────────────────────────────────────────────────────────────────────────╯
+Aggregate tok/s  + TTFT/ITL                              
+╭────────────┬─────────────┬─────────────┬──────────────╮
+│ ctx \ conc │           1 │           2 │            4 │
+├────────────┼─────────────┼─────────────┼──────────────┤
+│ 0          │  175.2 70/6 │ 236.4 112/8 │ 296.7 173/13 │
+│ 8k         │ 174.2 591/6 │ 246.8 947/8 │  291.4 2k/14 │
+│ 32k        │ 179.3 603/6 │ 239.9 961/8 │  302.2 2k/13 │
+╰────────────┴─────────────┴─────────────┴──────────────╯
+Sustained Decode: aggregate tok/s uses OpenAI stream usage by default 
+(continuous completion_tokens when the server supports it). Prometheus is kept 
+as validation/scheduler data.
+Aggregate source(s): openai_continuous_usage
+Per-Request tok/s                    
+╭────────────┬───────┬───────┬──────╮
+│ ctx \ conc │     1 │     2 │    4 │
+├────────────┼───────┼───────┼──────┤
+│ 0          │ 175.2 │ 118.2 │ 74.2 │
+│ 8k         │ 174.2 │ 123.4 │ 72.9 │
+│ 32k        │ 179.3 │ 119.9 │ 75.5 │
+╰────────────┴───────┴───────┴──────╯
+Client request latency: p50 /   
+p90 ms                          
+╭────────────┬─────┬─────┬─────╮
+│ ctx \ conc │   1 │   2 │   4 │
+├────────────┼─────┼─────┼─────┤
+│ 0          │ —/— │ —/— │ —/— │
+│ 8k         │ —/— │ —/— │ —/— │
+│ 32k        │ —/— │ —/— │ —/— │
+╰────────────┴─────┴─────┴─────╯
+Aggregate cells show dim detail as TTFT ms / ITL ms for the same ctx/conc 
+coordinate. ITL is computed from observed generated tokens, including streams 
+stopped at the measurement boundary; a missing ITL means no stream produced at 
+least two measured output tokens. Per-request tok/s and request latency are 
+shown in separate per-cell matrices. Completion/sample counts and full 
+request-level distributions remain in JSON under request_samples.
+Sustained mode: client latency metrics explain request UX variance; aggregate 
+tok/s remains the primary throughput signal. 
+ITL=(last_token_time-first_token_time)/(output_tokens-1), user tok/s=1/ITL.
+Hardware Summary                                                                
+╭───┬─┬───────┬───────────┬───────┬─────────┬─────┬──────┬─────┬───────────────╮
+│ … │ │ mode  │ GPU avg/… │ Mem … │ W avg/… │ T … │ CPU… │ VR… │ PCIe rx/tx a… │
+├───┼─┼───────┼───────────┼───────┼─────────┼─────┼──────┼─────┼───────────────┤
+│ 0 │ │ sust… │    99/99% │   44% │ 1153/1… │ 82C │  76C │ 98… │     8331/8194 │
+│ … │ │ sust… │    99/99% │   44% │ 1154/1… │ 83C │  77C │ 98… │     8357/8151 │
+│ … │ │ sust… │    99/99% │   44% │ 1156/1… │ 83C │  76C │ 98… │     8340/8144 │
+│ 0 │ │ sust… │  100/100% │   40% │ 1176/1… │ 84C │  77C │ 98… │   11179/11088 │
+│ 0 │ │ sust… │  100/100% │   35% │ 1173/1… │ 84C │  77C │ 98… │     8004/7682 │
+│ … │ │ sust… │  100/100% │   40% │ 1177/1… │ 84C │  77C │ 98… │   11225/11103 │
+│ … │ │ sust… │  100/100% │   35% │ 1173/1… │ 84C │  77C │ 98… │     7728/7833 │
+│ … │ │ sust… │  100/100% │   41% │ 1178/1… │ 84C │  77C │ 98… │   11128/10971 │
+│ … │ │ sust… │  100/100% │   35% │ 1172/1… │ 84C │  77C │ 98… │     7908/7680 │
+╰───┴─┴───────┴───────────┴───────┴─────────┴─────┴──────┴─────┴───────────────╯
+╭───────────────────────── Whole-run GPU Power ─────────────────────────╮
+│ avg 1,105 W | max 1,178 W | limit 1,200 W | over 4m 29s | 113 samples │
+╰───────────────────────────────────────────────────────────────────────╯
+Hardware summary is sampled from nvidia-smi during the measured part of each 
+cell. Whole-run GPU power is the sampled sum of GPU power draw across the 
+complete benchmark run, not wall-outlet system power. PCIe rx/tx is MB/s and is 
+a coarse live diagnostic, not a per-kernel NCCL profiler.
+
+╭────────────────────────────────── Phase 3 ───────────────────────────────────╮
+│ Burst / E2E Decode                                                           │
+│ Not run. Re-run with --run-burst to append a finite client-facing request    │
+│ burst after Sustained Decode. This is intentionally disabled by default      │
+│ because it adds another full decode matrix.                                  │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+╭────────────────────────────── Primary Summary ───────────────────────────────╮
+│ Primary matrices repeated last so the important numbers are visible without  │
+│ scrolling back through diagnostics.                                          │
+╰─────────────────────────────────────��────────────────────────────────────────╯
+Aggregate decode tok/s                
+╭────────────┬───────┬───────┬───────╮
+│ ctx \ conc │     1 │     2 │     4 │
+├────────────┼───────┼───────┼───────┤
+│ 0          │ 175.2 │ 236.4 │ 296.7 │
+│ 8k         │ 174.2 │ 246.8 │ 291.4 │
+│ 32k        │ 179.3 │ 239.9 │ 302.2 │
+╰────────────┴───────┴───────┴───────╯
+
+Results saved to 
+<campaign>/candidate-speed-wi
+ndow-01/results-01/decode-warp-quant/rep-2/decode-cap8192.json

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/prefill-command.json

@@ -0,0 +1,31 @@
+[
+  "/usr/bin/python3",
+  "<workspace>/trellismx-performance-audit-20260908/llm_decode_bench.py",
+  "--host",
+  "127.0.0.1",
+  "--port",
+  "8001",
+  "--model",
+  "glm53-flash-trellismx-p8-k45",
+  "--duration",
+  "20",
+  "--max-tokens",
+  "8192",
+  "--token-targeting",
+  "exact",
+  "--display-mode",
+  "plain",
+  "--output",
+  "<campaign>/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/prefill.json",
+  "--contexts",
+  "0",
+  "--concurrency",
+  "1,2,4",
+  "--prefill-only",
+  "--prefill-contexts",
+  "8k,32k,64k,128k",
+  "--prefill-duration",
+  "20",
+  "--cell-warmup-timeout-seconds",
+  "180"
+]

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/prefill-receipt.json

@@ -0,0 +1,5 @@
+{
+  "exit_code": 0,
+  "result_exists": true,
+  "sha256": "d0342998106c3361f6c09187c2c66554d2169d2910240fa71372ec70b061bdbb"
+}

results/speed-20260909/evidence/candidate-speed-window-01/results-01/decode-warp-quant/rep-2/prefill.json

@@ -0,0 +1,396 @@
+{
+  "metadata": {
+    "version": "0.4.29",
+    "engine": "vllm",
+    "model": "glm53-flash-trellismx-p8-k45",
+    "server": "127.0.0.1:8001",
+    "timestamp": "2026-09-09T02:28:52.054869",
+    "decode_mode": "duration",
+    "primary_decode_layer": "sustained_decode",
+    "duration_per_test": 20.0,
+    "request_count": 0,
+    "warmup_request_count": 0,
+    "run_burst": false,
+    "prefill_mode": "standalone_cold",
+    "standalone_prefill": true,
+    "prefill_only": true,
+    "skip_prefill": false,
+    "burst_e2e_status": "not_run_use_--run-burst",
+    "burst_request_count": 0,
+    "burst_warmup_request_count": 0,
+    "burst_requests_per_concurrency": 5,
+    "decode_warmup_seconds": 3.0,
+    "decode_warmup_context": 0,
+    "decode_warmup_concurrency": 1,
+    "cell_warmup_timeout_seconds": 180.0,
+    "cell_warmup_timeout_policy": "<=32k:60s,64k:120s,>=128k:180s when override is 0",
+    "show_capacity_limited_values": false,
+    "max_tokens": 8192,
+    "temperature": null,
+    "ignore_eos": true,
+    "max_total_tokens": 29351936,
+    "dcp_size": 0,
+    "metrics_available": true,
+    "metrics_warning": "",
+    "concurrency_levels": [
+      1,
+      2,
+      4
+    ],
+    "context_lengths": [
+      0
+    ],
+    "startup_diagnostics_available": true,
+    "nvidia_p2p_override_effective": true,
+    "p2pmark_status": "not_run",
+    "amd_fabric_status": "not_run"
+  },
+  "startup_diagnostics": {
+    "version": "0.4.29",
+    "server_url": "http://127.0.0.1:8001",
+    "hostname": "<host>",
+    "uname": "Linux <host> 6.18.7-76061807-generic #202601231045~1769703228~24.04~cb87b5b SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux",
+    "env": {},
+    "args": {
+      "concurrency": "1,2,4",
+      "contexts": "0",
+      "max_tokens": 8192,
+      "duration": 20.0,
+      "request_count": 0,
+      "run_burst": false,
+      "standalone_prefill": true,
+      "prefill_only": true,
+      "skip_prefill": false,
+      "prefill_contexts": "8k,32k,64k,128k",
+      "prefill_metric": "client",
+      "dcp_size": 0,
+      "kv_budget": 0
+    },
+    "nvidia_p2p_override": {
+      "effective": true,
+      "configured": true,
+      "params_path": "/proc/driver/nvidia/params",
+      "params_available": true,
+      "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+      "modprobe_available": true,
+      "runtime": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1",
+        "DmaRemapPeerMmio": "1"
+      },
+      "expected": {
+        "ForceP2P": "0x11",
+        "RMForceP2PType": "1",
+        "RMPcieP2PType": "2",
+        "GrdmaPciTopoCheckOverride": "1",
+        "EnableResizableBar": "1"
+      },
+      "missing": [],
+      "mismatched": {},
+      "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+      "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+      "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+    },
+    "p2pmark": {
+      "status": "not_run"
+    },
+    "amd_fabric": {
+      "status": "not_run"
+    },
+    "nvidia_smi_query": {
+      "cmd": [
+        "nvidia-smi",
+        "--query-gpu=index,name,driver_version,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,power.limit",
+        "--format=csv,noheader,nounits"
+      ],
+      "returncode": 0,
+      "stdout": "0, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:01:00.0, 5, 16, 300.00\n1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:21:00.0, 5, 16, 300.00\n2, NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 610.57.04, 00000000:81:00.0, 5, 16, 300.00\n3, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 610.57.04, 00000000:C1:00.0, 5, 16, 300.00",
+      "stderr": ""
+    },
+    "nvidia_smi_topo": {
+      "cmd": [
+        "nvidia-smi",
+        "topo",
+        "-m"
+      ],
+      "returncode": 0,
+      "stdout": "\u001b[4mGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity\tGPU NUMA ID\u001b[0m\nGPU0\t X \tNODE\tNODE\tNODE\t0-47\t0\t\tN/A\nGPU1\tNODE\t X \tNODE\tNODE\t0-47\t0\t\tN/A\nGPU2\tNODE\tNODE\t X \tNODE\t0-47\t0\t\tN/A\nGPU3\tNODE\tNODE\tNODE\t X \t0-47\t0\t\tN/A\n\nLegend:\n\n  X    = Self\n  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n  PIX  = Connection traversing at most a single PCIe bridge\n  NV#  = Connection traversing a bonded set of # NVLinks",
+      "stderr": ""
+    }
+  },
+  "nvidia_p2p_override": {
+    "effective": true,
+    "configured": true,
+    "params_path": "/proc/driver/nvidia/params",
+    "params_available": true,
+    "modprobe_path": "/etc/modprobe.d/nvidia-p2p-override.conf",
+    "modprobe_available": true,
+    "runtime": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1",
+      "DmaRemapPeerMmio": "1"
+    },
+    "expected": {
+      "ForceP2P": "0x11",
+      "RMForceP2PType": "1",
+      "RMPcieP2PType": "2",
+      "GrdmaPciTopoCheckOverride": "1",
+      "EnableResizableBar": "1"
+    },
+    "missing": [],
+    "mismatched": {},
+    "registry_dwords": "ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1",
+    "suggested_modprobe_line": "options nvidia NVreg_RegistryDwords=\"ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1\"",
+    "suggested_reload": "stop GPU workloads, then reload NVIDIA modules or reboot; the modprobe file alone is not enough until the nvidia module is reloaded"
+  },
+  "p2pmark": {
+    "status": "not_run"
+  },
+  "amd_fabric": {
+    "status": "not_run"
+  },
+  "hardware_run_summary": {
+    "samples": 48,
+    "duration_seconds": 113.123,
+    "gpu_count": 4,
+    "cpu_util_avg_pct": 10.91,
+    "cpu_temp_max_c": 77.25,
+    "gpu_util_avg_pct": 88.84,
+    "gpu_util_max_pct": 100.0,
+    "mem_util_avg_pct": 19.54,
+    "mem_util_max_pct": 34.0,
+    "temp_avg_c": 67.36,
+    "temp_max_c": 84.0,
+    "power_total_avg_w": 1049.68,
+    "power_total_max_w": 1145.97,
+    "power_limit_total_w": 1200.0,
+    "vram_used_avg_mb": 384778.0,
+    "vram_used_max_mb": 384778.0,
+    "vram_total_mb": 391548.0,
+    "vram_used_avg_pct": 98.27,
+    "vram_used_max_pct": 98.27,
+    "pcie_rx_avg_mb_s": 39825.88,
+    "pcie_rx_max_mb_s": 55605.0,
+    "pcie_tx_avg_mb_s": 40385.83,
+    "pcie_tx_max_mb_s": 53259.0
+  },
+  "event_log": [],
+  "prefill": {
+    "8192": {
+      "ttft_seconds": 1.155,
+      "prefill_seconds": 1.155,
+      "tok_per_sec": 7097.0,
+      "client_ttft_seconds": 1.155,
+      "client_tok_per_sec": 7097.0,
+      "prompt_tokens": 8194,
+      "samples": 14,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 9,
+        "duration_seconds": 19.191,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.01,
+        "cpu_temp_max_c": 75.75,
+        "gpu_util_avg_pct": 76.69,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 18.22,
+        "mem_util_max_pct": 34.0,
+        "temp_avg_c": 65.86,
+        "temp_max_c": 80.0,
+        "power_total_avg_w": 987.33,
+        "power_total_max_w": 1139.37,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 29085.56,
+        "pcie_rx_max_mb_s": 53572.0,
+        "pcie_tx_avg_mb_s": 29491.11,
+        "pcie_tx_max_mb_s": 53259.0
+      }
+    },
+    "32768": {
+      "ttft_seconds": 4.454,
+      "prefill_seconds": 4.454,
+      "tok_per_sec": 7357.0,
+      "client_ttft_seconds": 4.454,
+      "client_tok_per_sec": 7357.0,
+      "prompt_tokens": 32770,
+      "samples": 5,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 10,
+        "duration_seconds": 21.644,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.16,
+        "cpu_temp_max_c": 76.38,
+        "gpu_util_avg_pct": 90.4,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 19.95,
+        "mem_util_max_pct": 30.0,
+        "temp_avg_c": 67.0,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1056.58,
+        "power_total_max_w": 1144.59,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 45582.6,
+        "pcie_rx_max_mb_s": 51598.0,
+        "pcie_tx_avg_mb_s": 44600.2,
+        "pcie_tx_max_mb_s": 48299.0
+      }
+    },
+    "65536": {
+      "ttft_seconds": 8.891,
+      "prefill_seconds": 8.891,
+      "tok_per_sec": 7371.0,
+      "client_ttft_seconds": 8.891,
+      "client_tok_per_sec": 7371.0,
+      "prompt_tokens": 65538,
+      "samples": 3,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 12,
+        "duration_seconds": 26.544,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.23,
+        "cpu_temp_max_c": 76.75,
+        "gpu_util_avg_pct": 97.67,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 21.02,
+        "mem_util_max_pct": 29.0,
+        "temp_avg_c": 68.04,
+        "temp_max_c": 83.0,
+        "power_total_avg_w": 1074.72,
+        "power_total_max_w": 1145.97,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 42799.33,
+        "pcie_rx_max_mb_s": 54328.0,
+        "pcie_tx_avg_mb_s": 43858.08,
+        "pcie_tx_max_mb_s": 53032.0
+      }
+    },
+    "131072": {
+      "ttft_seconds": 17.957,
+      "prefill_seconds": 17.957,
+      "tok_per_sec": 7299.0,
+      "client_ttft_seconds": 17.957,
+      "client_tok_per_sec": 7299.0,
+      "prompt_tokens": 131073,
+      "samples": 2,
+      "method": "client",
+      "server_validation": {
+        "method": "",
+        "tok_per_sec": 0.0,
+        "prefill_seconds": 0.0,
+        "prompt_tokens": 0,
+        "request_prompt_tokens": 0,
+        "cached_tokens": 0,
+        "token_source": "",
+        "samples": 0,
+        "invalid_reason": ""
+      },
+      "hardware_summary": {
+        "samples": 15,
+        "duration_seconds": 33.726,
+        "gpu_count": 4,
+        "cpu_util_avg_pct": 11.25,
+        "cpu_temp_max_c": 77.25,
+        "gpu_util_avg_pct": 99.87,
+        "gpu_util_max_pct": 100.0,
+        "mem_util_avg_pct": 21.47,
+        "mem_util_max_pct": 29.0,
+        "temp_avg_c": 68.65,
+        "temp_max_c": 84.0,
+        "power_total_avg_w": 1123.91,
+        "power_total_max_w": 1145.94,
+        "power_limit_total_w": 1200.0,
+        "vram_used_avg_mb": 384778.0,
+        "vram_used_max_mb": 384778.0,
+        "vram_total_mb": 391548.0,
+        "vram_used_avg_pct": 98.27,
+        "vram_used_max_pct": 98.27,
+        "pcie_rx_avg_mb_s": 43837.13,
+        "pcie_rx_max_mb_s": 55605.0,
+        "pcie_tx_avg_mb_s": 44351.67,
+        "pcie_tx_max_mb_s": 52140.0
+      }
+    }
+  },
+  "results": [],
+  "summary_table": {},
+  "burst_results": [],
+  "burst_summary_table": {},
+  "methodology": {
+    "prefill": {
+      "name": "Prefill",
+      "present": true,
+      "mode": "standalone_cold",
+      "formula": "prompt_tokens / TTFT",
+      "notes": "Default mode records the required decode scout request for each non-zero decode context, so normal runs do not pay for a separate prefill phase. Standalone mode repeats cold-prefill samples. Prometheus prefill counters, when available and uncontaminated, are stored as validation."
+    },
+    "sustained_decode": {
+      "name": "Sustained Decode",
+      "present": false,
+      "formula": "OpenAI stream usage completion_tokens per measured window; client chunk fallback only when continuous usage is unavailable",
+      "notes": "Duration-based steady-state cell after warmup. This is the main tuning/regression signal for kernels, NCCL, DCP, MTP, and scheduling. Prometheus metrics are stored as validation and scheduler state, not the default headline."
+    },
+    "burst_e2e_decode": {
+      "name": "Burst / E2E Decode",
+      "present": false,
+      "status": "not run; use --run-burst",
+      "formula": "sum(completion_tokens) / profiling_wall_time",
+      "notes": "Finite client-facing request burst using OpenAI stream usage. It includes request admission, scheduling, prefill/cache behavior, and completion."
+    }
+  }
+}