File size: 2,015 Bytes
5d73525 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | # Frozen evaluation recipe
The fixtures in this directory are run-authored engineering diagnostics,
released under the CMF Apache-2.0 terms. They are not a general benchmark.
`prompts.jsonl` contains the eight frozen text prompts. The likelihood corpus
and its token IDs are separate fixtures; do not retokenize with a different
model or add a BOS token.
## Official reference
The official PQ2 reference used llama.cpp/Prism release commit
`9a9394a895b96003ca842a6041cb28ac49a108f7` with CUDA 12.8. The small
`reference-teacher-score.cpp` helper links against the existing official
`libllama`/ggml shared libraries and uses `llama_decode` plus
`llama_get_logits_ith`; it does not build llama.cpp or redistribute its
headers/libraries. Build the helper against that release's headers and
libraries, then run:
```sh
./llama-teacher-score \
/path/to/Ternary-Bonsai-2-27B-PQ2_0.gguf \
evaluation/quality-corpus-token-ids.txt \
evaluation/reference-teacher-score.jsonl
```
The contract is two independent 512-input-token resets at starts 0 and 512,
scoring the actual next IDs at targets 1..1024, with no synthetic BOS. The
shipped raw JSONL records all 1,024 correct-token log-probabilities and top-1
IDs without a full-vocabulary dump. Reference results: mean NLL
`1.81274952826`, PPL `6.12727139849`, top-1 `603/1024`.
## CMF affine comparison
Build the patched runtime and its `score_fixed` example using
`source/SOURCE_BUILD.md`, then use the same token-ID file and the model file:
```sh
export XDG_RUNTIME_DIR=/tmp WGPU_BACKEND=vulkan CMF_GPU=wgpu
export CMF_Q2TP_GPU=1 CMF_GPU_PROBE=0 CMF_MTP=0
./target/debug/examples/score_fixed \
Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \
evaluation/quality-corpus-token-ids.txt
```
The validated affine Vulkan output was total NLL `1856.528128685`, mean NLL
`1.813015751`, PPL `6.128902833`; CPU mean NLL was `1.812992284`. Ordinary
Q2TP is retained only as a comparison: mean NLL `1.886951899`, delta
`+0.074202371`, outside the adopted `+0.05` primary gate.
|