Update README.md

README.md

@@ -22,8 +22,7 @@ required_cli_args:
 - '--reasoning-parser gemma4'
 - '--enable-prefix-caching'
 default-chat-template-kwargs: '{"enable_thinking": true}'
-chat_template_file_name: None
-chat_template_path: None
+chat_template_file_name: chat_template.jinja
 tool_call_parser: gemma4
 validated_tasks:
 - tool-calling
@@ -84,13 +83,15 @@ vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega \
   --enable-auto-tool-choice \
   --reasoning-parser gemma4 \
   --tool-call-parser gemma4 \
-  --chat-template examples/tool_chat_template_gemma4.jinja \
   --default-chat-template-kwargs '{"enable_thinking": true}' \
-  --limit-mm-per-prompt '{"image": 4, "audio": 1}' \
-  --async-scheduling
+  --limit-mm-per-prompt '{"image": 4}' \
+  --async-scheduling \
+  --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'
 ```
 
-> **Tip:** For text-only workloads, pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip vision encoder memory allocation and free up GPU memory for a longer context window.
+> **MTP:** use the QAT-specific drafter `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant`, not the vanilla one. On reasoning workloads it accepts ~78% of drafts vs ~53% for the vanilla drafter, a 2.2× vs 1.6× speedup. Lossless in distribution.
+>
+> **Tip:** For text-only workloads, pass `--language-model-only` to skip vision encoder memory allocation and free up GPU memory for a longer context window.
 
 2. Send requests to the server:
 
@@ -149,7 +150,7 @@ Settings passed to heretic: `offload_outputs_to_cpu=False`, `device_map={"": 0}`
 
 ## Evaluation
 
-This model was evaluated on GSM8K Platinum, MMLU-Pro, IFEval, MATH-500, and GPQA Diamond using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and [lighteval](https://github.com/huggingface/lighteval), served with [vLLM](https://docs.vllm.ai/en/latest/) (OpenAI-compatible API).
+This model was evaluated on IFEval, GSM8K Platinum, MATH-500, GPQA Diamond, WikiText-2, TruthfulQA-MC2, a 300-problem GSM8K termination split, and a needle-in-haystack long-context test using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), served with [vLLM](https://docs.vllm.ai/en/latest/) (OpenAI-compatible API).
 
 ### Performance
 
@@ -164,9 +165,13 @@ Cross-dataset audit: JailbreakBench 3/100, tulu-harmbench 3/320, NousResearch/Re
 
 ### Accuracy
 
-I ran these to confirm that abliterating the QAT base and quantizing experts-only didn't cost capability. It's a sanity check with single seeds, not a statistically rigorous comparison. The result: quantization is lossless on every measured task, and the only cost is a 4-point increase in non-terminating traces under greedy decoding with thinking enabled.
+Three columns: the original model, Google's QAT release, and this checkpoint. All measured on the same hardware (NVIDIA GB10), same vLLM build, same server config, same seed, with the QAT-specific MTP drafter on every server. The difference between the first two is the cost of QAT; between the last two is the cost of abliteration. Recovery is this model divided by the QAT base (abliteration cost only). Protocol follows [RedHatAI](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4)'s: 0-shot, temperature 1.0, top-p 0.95, top-k 64, `max_gen_toks=32000`, seed 1234, 1 repetition.
 
-Three columns: the original model, Google's QAT release, the abliterated QAT base (this checkpoint). NVFP4 quant is coming soon and will be reported on its own model card. Recovery is this model divided by the QAT base (abliteration cost only).
+The vanilla and QAT columns are complete for round 1 (IFEval, wikitext, TruthfulQA, GSM8K split) and show the two bases are equivalent on every chat-templated task; they differ only on raw-text perplexity, where vanilla scores 24% worse for reasons not yet explained. Round 2 (Platinum, MATH-500, GPQA, no-think) was run against QAT as the reference base and not repeated on vanilla.
+
+I ran these to confirm that abliterating the QAT base didn't cost capability. It's a sanity check with single seeds, not a statistically rigorous comparison. The result: abliteration costs about 5 points on TruthfulQA and 2–4 points on MATH-500, and is within noise everywhere else. Termination and long-context retrieval are unaffected. The NVFP4 quant of this checkpoint is reported on its own model card.
+
+One 26B-specific observation: MATH-500 scores higher *without* thinking on all three models (QAT 86.8 vs 83.8, EGA 83.2 vs 81.8). The MoE's reasoning traces cost it a few points on this benchmark.
 
 **With thinking**
 
@@ -352,132 +357,114 @@ The results were obtained using the following commands:
 
 <details>
 
-Each benchmark was run 1 time with same judgmental seed (1234).
+Single seed (1234) per benchmark. All three models served on NVIDIA GB10 with the same vLLM build and identical server flags apart from the model path.
 
-Ran using a docker image, you will need to tweak these for your own environment. Parameters similar to the RedHatAI runs.
+**vLLM server:**
 
-**vLLM server (all benchmarks):**
-```
-sudo docker run --rm -it \
-  --gpus all \
-  --ipc=host \
-  --network host \
+`ghcr.io/timothystewart6/vllm-gb10:latest` (`v0.28.1.dev0+g2cf0a6915.d20260828`, transformers 5.16.1). The QAT-specific MTP drafter is lossless under rejection sampling and only affects throughput.
+
+```shell
+docker run --rm -it \
+  --gpus all --ipc=host --network host \
   -v ~/models/abl/gemma-4-26B-qat-ega:/models/qat-ega:ro \
+  -v ~/.cache/huggingface:/root/.cache/huggingface \
   -e VLLM_USE_V2_MODEL_RUNNER=1 \
   ghcr.io/timothystewart6/vllm-gb10:latest \
   vllm serve /models/qat-ega \
   --host 0.0.0.0 --port 8001 \
   --served-model-name qat-ega \
-  --max-model-len 69632 \
-  --gpu-memory-utilization 0.50 \
+  --max-model-len 65536 \
+  --gpu-memory-utilization 0.75 \
+  --max-num-seqs 32 \
+  --max-num-batched-tokens 8192 \
+  --language-model-only \
   --enable-auto-tool-choice \
   --reasoning-parser gemma4 \
   --tool-call-parser gemma4 \
-  --limit-mm-per-prompt '{"image":0,"audio":0}' \
-  --kv-cache-dtype fp8 \
-  --max-num-seqs 32 \
-  --async-scheduling
+  --async-scheduling \
+  --default-chat-template-kwargs '{"enable_thinking": true}' \
+  --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'
 ```
 
-#### GSM8K Platinum (lm-eval, 0-shot, 1 repetition)
-```
-for SEED in 1234 2345 3456; do
-  lm_eval --model local-chat-completions \
-    --tasks gsm8k_platinum_cot_llama \
-    --model_args "model=qat-ega,max_length=36096,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \
-    --num_fewshot 0 \
-    --apply_chat_template \
-    --output_path "results_gsm8k_seed${SEED}.json" \
-    --seed $SEED \
-    --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=${SEED}"
-  echo "Seed $SEED complete"
-done
-```
+> To reproduce the without-thinking results, remove `--default-chat-template-kwargs '{"enable_thinking": true}'`.
+>
+> **Deviations from RedHatAI's protocol:** `--max-model-len 65536` rather than 32768 — with `max_gen_toks=32000`, a 32768 ceiling leaves 768 tokens for the prompt and MATH-500 has longer problems, which vLLM rejects with HTTP 400. `timeout=3600` rather than 1200 — BF16 on GB10 needs ~31 minutes to exhaust a 32000-token budget. `until=[]` on all tasks — the default stop sequences truncate thinking traces mid-reasoning. Model's shipped chat template rather than `examples/tool_chat_template_gemma4.jinja`. `max_retries=6`.
 
-#### MMLU-Pro (lm-eval, 0-shot, 1 repetition)
-```
-for SEED in 1234 2345 3456; do
-  lm-eval run --model local-chat-completions \
-    --tasks mmlu_pro \
-    --model_args "model=qat-ega,max_length=36096,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \
-    --num_fewshot 0 \
-    --apply_chat_template \
-    --output_path "results_mmlu_pro_seed${SEED}.json" \
-    --seed $SEED \
-    --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=${SEED}"
-  echo "MMLU-Pro seed $SEED complete"
-done
+#### GSM8K Platinum (lm-eval, 0-shot)
+```shell
+lm_eval --model local-chat-completions \
+  --tasks gsm8k_platinum_cot_llama \
+  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
+  --num_fewshot 0 --apply_chat_template \
+  --output_path results/qat-ega_gsm8k_platinum \
+  --seed 1234 \
+  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
 ```
 
-#### IFEval (lm-eval, 0-shot, 1 repetition)
-```
-for SEED in 1234; do
-  lm_eval --model local-chat-completions \
-    --tasks ifeval \
-    --model_args "model=qat-ega,max_length=36096,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \
-    --num_fewshot 0 \
-    --apply_chat_template \
-    --output_path "results_ifeval_seed${SEED}.json" \
-    --seed $SEED \
-    --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=${SEED}"
-  echo "IFEval seed $SEED complete"
-done
+#### IFEval (lm-eval, 0-shot)
+```shell
+lm_eval --model local-chat-completions \
+  --tasks ifeval \
+  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
+  --num_fewshot 0 --apply_chat_template \
+  --output_path results/qat-ega_ifeval \
+  --seed 1234 \
+  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
 ```
 
-#### MATH-500, AIME 2025, GPQA Diamond (lighteval, 1 repetition)
-
-**litellm_config.yaml:**
-```yaml
-model_parameters:
-  provider: hosted_vllm
-  model_name: hosted_vllm/qat-ega
-  base_url: http://0.0.0.0:8001/v1
-  api_key: ''
-  timeout: 7200
-  concurrent_requests: 32
-  generation_parameters:
-    temperature: 1.0
-    max_new_tokens: 65536
-    top_p: 0.95
-    top_k: 64
-    seed: 1234
+#### MATH-500 (lm-eval `minerva_math500`, 0-shot)
+
+Requires `pip install 'lm-eval[math]'` and `antlr4-python3-runtime==4.11`. Same 500-problem subset as lighteval's `math_500`, scored with sympy-based answer equivalence (`math_verify`). The `exact_match` filter reports 0 on thinking-mode output and is not the reported number.
+
+```shell
+lm_eval --model local-chat-completions \
+  --tasks minerva_math500 \
+  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
+  --num_fewshot 0 --apply_chat_template \
+  --output_path results/qat-ega_math500 \
+  --seed 1234 \
+  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
 ```
 
+#### GPQA Diamond (lm-eval `gpqa_diamond_cot_zeroshot`, 0-shot)
+
+Dataset is gated; requires an HF token that has accepted the terms for `Idavidrein/gpqa`. `flexible-extract` is the reported number.
+
+```shell
+lm_eval --model local-chat-completions \
+  --tasks gpqa_diamond_cot_zeroshot \
+  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
+  --num_fewshot 0 --apply_chat_template \
+  --output_path results/qat-ega_gpqa \
+  --seed 1234 \
+  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
 ```
-# GPQA Diamond (1 seed)
-for SEED in 1234; do
-  sed -i "s/seed: .*/seed: $SEED/" litellm_config.yaml
-  lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \
-    --output-dir "results_gpqa_seed${SEED}/" --save-details
-  echo "GPQA Diamond seed $SEED complete"
-done
-
-# MATH-500 (1 seed)
-for SEED in 1234; do
-  sed -i "s/seed: .*/seed: $SEED/" litellm_config.yaml
-  lighteval endpoint litellm litellm_config.yaml 'math_500|0' \
-    --output-dir "results_math500_seed${SEED}/" --save-details
-  echo "MATH-500 seed $SEED complete"
-done
+
+#### WikiText-2 and TruthfulQA-MC2 (lm-eval, loglikelihood)
+```shell
+lm_eval --model local-completions \
+  --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \
+  --tasks wikitext --num_fewshot 0 --batch_size 1
+
+lm_eval --model local-completions \
+  --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \
+  --tasks truthfulqa_mc2 --num_fewshot 0 --apply_chat_template
 ```
 
-#### LiveCodeBench v6 (lighteval, 3 repetitions)
-
-**litellm_config.yaml:**
-```yaml
-model_parameters:
-  provider: hosted_vllm
-  model_name: hosted_vllm/qat-ega
-  base_url: http://0.0.0.0:8001/v1
-  api_key: ''
-  timeout: 7200
-  concurrent_requests: 32
-  generation_parameters:
-    temperature: 1.0
-    max_new_tokens: 32768
-    top_p: 0.95
-    top_k: 64
-    seed: 1234
+#### GSM8K three-number split (lm-eval, 0-shot, temperature 0, 300-problem subset)
+```shell
+lm_eval --model local-chat-completions \
+  --tasks gsm8k \
+  --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=16,max_retries=3,timeout=3600" \
+  --num_fewshot 0 --limit 300 --apply_chat_template \
+  --gen_kwargs "max_gen_toks=8192,until=[]" \
+  --output_path results/qat-ega_gsm8k_300 --log_samples
 ```
 
+Rescored from the samples file: a response is *empty* if it contains no content (thinking never terminated); *headline* is correct / total; *answered-only* is correct / (total − empty). Extraction takes the last number in the response after stripping markdown and thousands separators, since lm-eval's flexible-extract filter returns `[invalid]` on this model's bolded answer formatting.
+
+#### Needle-in-a-haystack
+
+Custom script: single needle (a random vault code) buried in WikiText-2 filler at depths 0/0.25/0.5/0.75/1.0 for context lengths 4k/8k/16k/32k/64k, thinking off, temperature 0, exact-match on the code. 25 requests per model.
+
 </details>