Verify vLLM usage on GPU and update latency and effort charts

README.md

@@ -37,7 +37,7 @@ We introduce **APUS-OpenJev-v1**, a family of decision models for browser agents
 
 **Decision-oriented inference.** The runtime scores candidate labels and maps them back to application actions. This removes the need to generate a structured answer token by token for bounded-choice tasks. Choose `effort="low"` or `effort="high"` to balance compute cost and decision quality.
 
-On our frozen 80-question development panel, **APUS-OpenJev 9B achieves 85.0% accuracy**, compared with **82.5% for the Jev API**. Its historical local reference implementation records **77.76 ms median model-forward latency**. Accuracy and latency are reported with their evaluation settings below.
+On our frozen 80-question development panel, **APUS-OpenJev 9B achieves 85.0% accuracy**, compared with **82.5% for the Jev API**. The optimized 9B decision service reports **25.58 ms median HTTP latency** in the run snapshot below. The 85.0% result uses `9B-3000`; the optimized latency measurement uses `9B-5949`. Accuracy and latency are reported with their evaluation settings below.
 
 ![Decision accuracy on the same 80-question panel: APUS 9B 85%, APUS 4B and Jev 82.5%, Laya complete-input 68.75%.](assets/accuracy.svg)
 
@@ -66,18 +66,29 @@ The 4B model leads this panel's Browser subset, while 9B's gains come from princ
 
 ### Response latency
 
-![P50 and P95 latency bars, with local model, local pipeline and remote API timing boundaries labeled.](assets/latency.svg)
+![APUS 9B vLLM decision API: P50 25.58 ms, P95 222.36 ms, P99 275.58 ms; comparison paths are labeled separately.](assets/latency-vllm.svg)
 
-*Figure 3. Latency observations with explicit measurement boundaries. P50 and P95 use separately labeled linear scales. These measurements are not streaming first-token latency or text-generation throughput.*
+*Figure 3. vLLM HTTP API full-response latency for the optimized decision service. The supplied run snapshot reports 333/333 successful requests and 0% errors. This is full-response latency, not streaming first-token latency.*
 
-| Measured path | P50 | P95 | Timing boundary |
-| --- | ---: | ---: | --- |
-| APUS-OpenJev 9B reference | 77.76 ms | 505.16 ms | Local model forward |
-| APUS-OpenJev 4B reference | 85.43 ms | 406.30 ms | Local model forward |
-| Laya typed · complete input | 8.68 ms | 37.66 ms | Local `system_one`, including tokenization and output processing |
-| Jev API | 436.96 ms | 4874.61 ms | Complete public HTTP response |
+| Decision API metric | Run snapshot |
+| --- | ---: |
+| **P50 latency** | **25.58 ms** |
+| P95 latency | 222.36 ms |
+| P99 latency | 275.58 ms |
+| Successful requests | 333 /333 |
+| Error rate | 0% |
 
-APUS figures come from the historical RTX PRO 6000 reference implementation, not a new end-to-end test of this merged release. Laya's measured local path is faster; the displayed statistics are the medians of three per-run P50/P95 values. Public API times include network, queuing, and service overhead, so their ratio to a local forward time is not a model speedup. Exact values and aggregation are recorded in [chart-data.json](chart-data.json).
+The chart distinguishes the supplied 333-request snapshot from the independently archived three-run C1 validation: **25.34 / 222.83 / 276.78 ms** median per-run P50/P95/P99, with **3,120/3,120** successful serial requests on an RTX PRO 6000 96GB. The exact 333-request raw trace is not part of that archive. Accuracy uses 80 fixed questions; replay counts do not increase the number of independent questions.
+
+The optimized path is the candidate-scoring `/decide` service. The standard Chat Completions launcher below has a separate generation path. Historical 4B model-forward, Laya local-pipeline and Jev public-API measurements remain labeled in the chart; different network and processing boundaries do not establish a model speedup ratio. See [measurement sources](latency-vllm-data.json).
+
+### Selectable effort: measured quality–latency trade-off
+
+![Same-checkpoint 4B effort comparison: 1.66x lower median latency and 2.05x lower P95 latency, with a 7.58% relative accuracy reduction.](assets/effort-tradeoff.svg)
+
+**Around 2× faster at P95, with less than 8% relative accuracy reduction.** On the same 4B checkpoint and frozen panel, selecting `effort="low"` reduces P95 model-forward latency from **406.30 to 198.32 ms**. Accuracy changes from **82.50% to 76.25%**: **6.25 percentage points**, or **7.58% relative**. P50 changes from **85.43 to 51.48 ms**, a **1.66×** speed ratio.
+
+These paired measurements use the native effort runtime. They are separate from the full-depth vLLM Chat API measurements and do not imply a measured vLLM low-effort endpoint. Deployment-specific low/high performance must be measured using the same serving path. See [effort measurement data](effort-tradeoff-data.json).
 
 ## 3. Evaluation Set
 
@@ -137,14 +148,21 @@ On Linux with Python 3.12 and an NVIDIA GPU with sufficient memory (the existing
 ```bash
 python3.12 -m venv .venv
 source .venv/bin/activate
-pip install vllm==0.29.0 huggingface-hub openai ninja
-curl -L https://huggingface.co/apus-ailab/APUS-OpenJev-v1/resolve/main/deployment/serve_vllm.py -o serve_vllm.py
+pip install vllm==0.29.0 transformers==5.17.0 huggingface-hub==1.32.0 openai==3.16.2 ninja==1.13.2
+# Blackwell: use the CUDA 13 compiler used in the GPU verification.
+pip install nvidia-cuda-nvcc==13.4.92 nvidia-cuda-crt==13.4.92 nvidia-cuda-cccl==13.3.4.3.1
+export CUDA_HOME="$(python -c 'import sysconfig; print(sysconfig.get_path("purelib") + "/nvidia/cu13")')"
+export PATH="$CUDA_HOME/bin:$PATH"
+"$CUDA_HOME/bin/nvcc" --version
+curl -fL https://huggingface.co/apus-ailab/APUS-OpenJev-v1/resolve/main/deployment/serve_vllm.py -o serve_vllm.py
 CUDA_VISIBLE_DEVICES=0 python serve_vllm.py
 # Optional: --revision <40-character-commit-SHA>
 # Inspect the command without downloading or using a GPU:
 # python serve_vllm.py --dry-run
 ```
 
+The first launch downloads approximately 18 GB and compiles GPU kernels; allow several minutes before sending requests. Check readiness with `curl -f http://127.0.0.1:8000/health`.
+
 In a second terminal using the same environment:
 
 ```python
@@ -159,10 +177,15 @@ response = client.chat.completions.create(
         "Reply with only A, B, or C."}],
     temperature=0,
     max_tokens=16,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
 )
-print(response.choices[0].message.content)
+print(response.choices[0].message.content)  # A
 ```
 
+Keep `enable_thinking=False` for these short decision outputs. The original example without this setting reached its 16-token limit before producing a decision; the corrected example returned `A` in the real GPU check.
+
+**GPU verification:** anonymous download from this public repository into a fresh cache, followed by 240/240 successful Chat Completions requests (80 fixed questions replayed three times), with no invalid answers or truncated completions. Local HTTP P50/P95/P99: **45.47 / 191.36 / 232.29 ms**. Test environment: Linux, Python 3.12.3, RTX PRO 6000 Blackwell 96GB, vLLM 0.29.0, PyTorch 2.13.0, Transformers 5.17.0, and the CUDA 13 compiler specified above. This reused an installed Python environment; a clean OS installation was not tested.
+
 This example returns standard chat-completion text through `choices[0].message.content`. It demonstrates **full-depth text serving**, not the specialized candidate-probability `/decide` gateway or `effort` routing. It does not enforce the three candidate labels; application-specific decision serving requires the corresponding prompt/compiler and output constraints. The server binds to localhost by default. Blackwell JIT compilation requires a compatible CUDA toolchain; the launcher disables the FlashInfer sampler as in the tested engine configuration.
 
 ## 6. Reproducing the Evaluation

README.zh-CN.md

@@ -39,7 +39,7 @@ tags:
 
 **面向决策的推理。** runtime对候选标签评分，再映射回应用动作。对于有界选择任务，无需逐token生成结构化答案。通过选择`effort="low"`或`effort="high"`，在计算成本与决策质量之间取得平衡。
 
-在冻结的80题开发回归面板上，**APUS-OpenJev 9B的准确率为85.0%**，**Jev API为82.5%**。其历史本地参考实现测得的**模型前向计算中位延迟为77.76 ms**。下文分别说明准确率和延迟的评测条件。
+在冻结的80题开发回归面板上，**APUS-OpenJev 9B的准确率为85.0%**，**Jev API为82.5%**。优化后的9B决策服务在下方运行快照中达到 **25.58 ms HTTP中位延迟**。85.0%对应`9B-3000`；优化延迟对应`9B-5949`。下文分别说明准确率和延迟的评测条件。
 
 ![同一80题面板上的决策准确率：APUS 9B为85%，APUS 4B与Jev为82.5%，Laya完整输入配置为68.75%。](assets/accuracy.svg)
 
@@ -68,18 +68,27 @@ tags:
 
 ### 响应延迟
 
-![P50与P95延迟柱状图，分别标注本地模型、本地处理流程和远程API的计时边界。](assets/latency.svg)
+![9B vLLM决策接口：P50 25.58 ms、P95 222.36 ms、P99 275.58 ms。](assets/latency-vllm.svg)
 
-*图3. 明确标注测量边界的延迟观测。P50与P95使用分别标注的线性刻度。这些测量不是流式首token延迟，也不是文本生成吞吐量。*
+| 决策接口指标 | 提供的运行快照 |
+| --- | ---: |
+| **P50延迟** | **25.58 ms** |
+| P95延迟 | 222.36 ms |
+| P99延迟 | 275.58 ms |
+| 成功请求 | 333 /333 |
+| 错误率 | 0% |
 
-| 测量路径 | P50 | P95 | 计时边界 |
-| --- | ---: | ---: | --- |
-| APUS-OpenJev 9B参考实现 | 77.76 ms | 505.16 ms | 本地模型前向计算 |
-| APUS-OpenJev 4B参考实现 | 85.43 ms | 406.30 ms | 本地模型前向计算 |
-| Laya typed · 完整输入 | 8.68 ms | 37.66 ms | 本地`system_one`，包含分词和输出处理 |
-| Jev API | 436.96 ms | 4874.61 ms | 公网完整HTTP响应 |
+333次请求数据来自提供的运行截图。独立归档可核验的C1三轮结果为：各轮P50/P95/P99的中位数 **25.34 / 222.83 / 276.78 ms**，成功 **3,120/3,120**，硬件为RTX PRO 6000 96GB；归档不含这333次请求的精确原始明细。准确率对应80道固定题，重复请求不增加独立题数。
 
-APUS数字来自历史RTX PRO 6000参考实现，不是对本次合并发布模型进行的新一轮端到端测试。Laya已测的本地路径更快；表中数字取三轮各自P50/P95的中位数。公网API时间包含网络、排队和服务开销，因此不能把它与本地前向耗时相除，作为模型加速倍数。精确数据及聚合方式记录在[chart-data.json](chart-data.json)中。
+这是候选打分`/decide`服务的完整HTTP响应耗时，非流式首token。下方标准Chat Completions脚本走独立生成路径。图中4B本地forward、Laya本地流程与Jev公网API保留各自计时边界，不直接相除宣称模型加速倍数。[数据来源](latency-vllm-data.json)。
+
+### 可选 effort：准确率与延迟的实测取舍
+
+![同一4B版本的low/high对比：P50提速1.66倍，P95提速2.05倍，准确率相对下降7.58%。](assets/effort-tradeoff.svg)
+
+**P95约2倍提速，准确率相对下降不到8%。** 同一4B版本、同一冻结面板上，`effort="low"`将P95模型前向延迟从 **406.30 ms降到198.32 ms**；准确率从 **82.50%降到76.25%**，下降 **6.25个百分点 / 相对7.58%**。P50从 **85.43 ms降到51.48 ms**，提速 **1.66倍**。
+
+这组数据来自原生effort runtime，与完整深度vLLM Chat API分开记录；不能把Chat API的实测延迟除以二当作low实测。相同部署路径的low/high需另行测试。[成对测量来源](effort-tradeoff-data.json)。
 
 ## 3. 评测集
 
@@ -132,7 +141,7 @@ python examples.py . --device cuda:0 --effort high
 
 ### vLLM 部署
 
-提供[单文件启动脚本](deployment/serve_vllm.py)，从 `apus-ailab/APUS-OpenJev-v1` 下载 `9B-5949` 完整权重并启动 vLLM 0.29.0。安装、启动及 OpenAI 调用示例见[英文说明](README.md#deploy-with-vllm)。该入口提供完整深度的标准文本生成；决策网关 `/decide` 和动态深度选择使用各自的运行路径。
+提供[单文件启动脚本](deployment/serve_vllm.py)，从 `apus-ailab/APUS-OpenJev-v1` 下载 `9B-5949` 完整权重并启动 vLLM 0.29.0。安装、启动及 OpenAI 调用示例见[英文说明](README.md#deploy-with-vllm)。已完成公开仓库匿名下载和真实GPU验证：标准Chat API三轮共240/240次成功、无截断，P50/P95/P99为45.47/191.36/232.29 ms。短标签请求必须设置`chat_template_kwargs.enable_thinking=False`，否则16-token示例会在思考阶段截断。复用了已安装的Python环境，未验证全新系统安装。该入口提供完整深度的标准文本生成；决策网关 `/decide` 和动态深度选择使用各自的运行路径。
 
 ## 6. 复现评测
 

bundle-manifest.json

@@ -17,7 +17,11 @@
     "ARCHITECTURE.md",
     "README.md",
     "README.zh-CN.md",
-    "deployment/serve_vllm.py"
+    "assets/effort-tradeoff.svg",
+    "assets/latency-vllm.svg",
+    "deployment/serve_vllm.py",
+    "effort-tradeoff-data.json",
+    "latency-vllm-data.json"
   ],
   "files": [
     {
@@ -577,13 +581,13 @@
     },
     {
       "path": "README.md",
-      "bytes": 12941,
-      "sha256": "40e8a7dae6191a2226073bd46118ddebcf8f3a334ba7d882219dc8d4bedfb433"
+      "bytes": 15514,
+      "sha256": "ea8782dac0afeff883a49d5c6c3751eb064f37a3838bd55c7b221ac5752d43a3"
     },
     {
       "path": "README.zh-CN.md",
-      "bytes": 10266,
-      "sha256": "abac6833d161ae043a302fe6eeb2816a15bac18e445481a3160837ac51951c55"
+      "bytes": 11222,
+      "sha256": "a6f352a67cc955f7321891978a710bec017fcbe385b0e0c8f4c9c0f09c36d555"
     },
     {
       "path": "TECHNICAL_REPORT.md",
@@ -610,6 +614,16 @@
       "bytes": 6425,
       "sha256": "b3198ec01536b6efff1c8f89402d3b45112c21471d613ed69f9a5c1a5fb94762"
     },
+    {
+      "path": "assets/effort-tradeoff.svg",
+      "bytes": 4092,
+      "sha256": "3c0c2354e04593d431db293a1e3153cc8bbf9c9c0b0c5a3ad59a68ce005592eb"
+    },
+    {
+      "path": "assets/latency-vllm.svg",
+      "bytes": 8391,
+      "sha256": "d6cbb29a693be1f5f4c4810b06b5277e279358c91283c3bd045c706ae32b0bae"
+    },
     {
       "path": "assets/latency.svg",
       "bytes": 6956,
@@ -635,6 +649,16 @@
       "bytes": 3535,
       "sha256": "7d8fac3021a9f9949e73364013989d861dcb6fd4c44ee338239822584d32ecf8"
     },
+    {
+      "path": "effort-tradeoff-data.json",
+      "bytes": 1201,
+      "sha256": "c2b2efb1b4aa1f25730b0f7605f87f5fef0662af30e809a79c70d3a2ce176e5a"
+    },
+    {
+      "path": "latency-vllm-data.json",
+      "bytes": 1337,
+      "sha256": "d04c85c1b75a209c0cc857ae9961d3651cde1c62b9cbf28dfadb3a5f6f2215a8"
+    },
     {
       "path": "subset-chart-data.json",
       "bytes": 58611,

effort-tradeoff-data.json

@@ -0,0 +1,32 @@
+{
+  "selected_model": "4B-5949",
+  "source_path": "evidence/official-vs-final5949-80/report.json",
+  "source_sha256": "b3542b7268a675a48c64ee0dad067bbb609c0e3c99bc6f7926e0241a56a27597",
+  "low": {
+    "n": 80,
+    "correct": 61,
+    "accuracy": 0.7625,
+    "nll_epsilon_1e-12": 1.0545613467438772,
+    "brier": 0.3943826629642137,
+    "latency_p50_seconds": 0.05147563456557691,
+    "latency_p95_seconds_nearest_rank": 0.19832013035193086
+  },
+  "high": {
+    "n": 80,
+    "correct": 66,
+    "accuracy": 0.825,
+    "nll_epsilon_1e-12": 0.6519612054580076,
+    "brier": 0.27633044836807164,
+    "latency_p50_seconds": 0.08542586304247379,
+    "latency_p95_seconds_nearest_rank": 0.40630325907841325
+  },
+  "p50_ratio_high_over_low": 1.6595397757291617,
+  "p95_ratio_high_over_low": 2.048724243763878,
+  "p50_latency_reduction_percent": 39.74233009506361,
+  "accuracy_drop_percentage_points": 6.25,
+  "accuracy_relative_drop_percent": 7.575757575757576,
+  "timing": "Historical single-request native forward; same checkpoint and frozen panel, not vLLM HTTP or merged latency retest",
+  "routing": "Caller-selected effort; not automatic per-question routing",
+  "n_records": 80,
+  "n_parent_groups": 79
+}

latency-vllm-data.json

@@ -0,0 +1,47 @@
+{
+  "provided_snapshot": {
+    "source": "User-provided screenshot",
+    "raw_requests_independently_verified": false,
+    "concurrency": null,
+    "reported_model": "APUS-OpenJev 9B-5949",
+    "p50_ms": 25.58,
+    "p95_ms": 222.36,
+    "p99_ms": 275.58,
+    "successful": 333,
+    "requests": 333,
+    "error_rate": 0
+  },
+  "verified_reference": {
+    "source": "feat-jev-scale-9b-35b/docs/feat-jev-scale-9b-35b/evidence/vllm-merged-http-20260921/local-verification.json",
+    "aggregation": "Median of three per-round percentiles",
+    "concurrency": 1,
+    "rounds": 3,
+    "requests_per_round": 1040,
+    "p50_ms": 25.34,
+    "p95_ms": 222.83,
+    "p99_ms": 276.78,
+    "correct": 67,
+    "quality_records": 80
+  },
+  "historical_rows": [
+    {
+      "name": "APUS-OpenJev 4B",
+      "scope": "Local model forward / historical",
+      "p50_ms": 85.43,
+      "p95_ms": 406.3
+    },
+    {
+      "name": "Jev official API",
+      "scope": "Remote HTTP / network + queue + service",
+      "p50_ms": 436.96,
+      "p95_ms": 4874.61
+    },
+    {
+      "name": "Laya typed / full input",
+      "scope": "Local system_one / three-run median",
+      "p50_ms": 8.68,
+      "p95_ms": 37.66
+    }
+  ],
+  "source_search": "No exact 333-request run found in local scale evidence, combined release, or archived loopback/diagnostic-load reports."
+}