# Frozen evaluation recipe

The fixtures in this directory are run-authored engineering diagnostics,
released under the CMF Apache-2.0 terms. They are not a general benchmark.
`prompts.jsonl` contains the eight frozen text prompts. The likelihood corpus
and its token IDs are separate fixtures; do not retokenize with a different
model or add a BOS token.

## Official reference

The official PQ2 reference used llama.cpp/Prism release commit
`9a9394a895b96003ca842a6041cb28ac49a108f7` with CUDA 12.8. The small
`reference-teacher-score.cpp` helper links against the existing official
`libllama`/ggml shared libraries and uses `llama_decode` plus
`llama_get_logits_ith`; it does not build llama.cpp or redistribute its
headers/libraries. Build the helper against that release's headers and
libraries, then run:

```sh
./llama-teacher-score \
  /path/to/Ternary-Bonsai-2-27B-PQ2_0.gguf \
  evaluation/quality-corpus-token-ids.txt \
  evaluation/reference-teacher-score.jsonl
```

The contract is two independent 512-input-token resets at starts 0 and 512,
scoring the actual next IDs at targets 1..1024, with no synthetic BOS. The
shipped raw JSONL records all 1,024 correct-token log-probabilities and top-1
IDs without a full-vocabulary dump. Reference results: mean NLL
`1.81274952826`, PPL `6.12727139849`, top-1 `603/1024`.

## CMF affine comparison

Build the patched runtime and its `score_fixed` example using
`source/SOURCE_BUILD.md`, then use the same token-ID file and the model file:

```sh
export XDG_RUNTIME_DIR=/tmp WGPU_BACKEND=vulkan CMF_GPU=wgpu
export CMF_Q2TP_GPU=1 CMF_GPU_PROBE=0 CMF_MTP=0
./target/debug/examples/score_fixed \
  Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \
  evaluation/quality-corpus-token-ids.txt
```

The validated affine Vulkan output was total NLL `1856.528128685`, mean NLL
`1.813015751`, PPL `6.128902833`; CPU mean NLL was `1.812992284`. Ordinary
Q2TP is retained only as a comparison: mean NLL `1.886951899`, delta
`+0.074202371`, outside the adopted `+0.05` primary gate.