evaluation/SCORE_RECIPE.md

Frozen evaluation recipe

The fixtures in this directory are run-authored engineering diagnostics, released under the CMF Apache-2.0 terms. They are not a general benchmark. prompts.jsonl contains the eight frozen text prompts. The likelihood corpus and its token IDs are separate fixtures; do not retokenize with a different model or add a BOS token.

Official reference

The official PQ2 reference used llama.cpp/Prism release commit 9a9394a895b96003ca842a6041cb28ac49a108f7 with CUDA 12.8. The small reference-teacher-score.cpp helper links against the existing official libllama/ggml shared libraries and uses llama_decode plus llama_get_logits_ith; it does not build llama.cpp or redistribute its headers/libraries. Build the helper against that release's headers and libraries, then run:

./llama-teacher-score \
  /path/to/Ternary-Bonsai-2-27B-PQ2_0.gguf \
  evaluation/quality-corpus-token-ids.txt \
  evaluation/reference-teacher-score.jsonl

The contract is two independent 512-input-token resets at starts 0 and 512, scoring the actual next IDs at targets 1..1024, with no synthetic BOS. The shipped raw JSONL records all 1,024 correct-token log-probabilities and top-1 IDs without a full-vocabulary dump. Reference results: mean NLL 1.81274952826, PPL 6.12727139849, top-1 603/1024.

CMF affine comparison

Build the patched runtime and its score_fixed example using source/SOURCE_BUILD.md, then use the same token-ID file and the model file:

export XDG_RUNTIME_DIR=/tmp WGPU_BACKEND=vulkan CMF_GPU=wgpu
export CMF_Q2TP_GPU=1 CMF_GPU_PROBE=0 CMF_MTP=0
./target/debug/examples/score_fixed \
  Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \
  evaluation/quality-corpus-token-ids.txt

The validated affine Vulkan output was total NLL 1856.528128685, mean NLL 1.813015751, PPL 6.128902833; CPU mean NLL was 1.812992284. Ordinary Q2TP is retained only as a comparison: mean NLL 1.886951899, delta +0.074202371, outside the adopted +0.05 primary gate.