| # Frozen evaluation recipe |
|
|
| The fixtures in this directory are run-authored engineering diagnostics, |
| released under the CMF Apache-2.0 terms. They are not a general benchmark. |
| `prompts.jsonl` contains the eight frozen text prompts. The likelihood corpus |
| and its token IDs are separate fixtures; do not retokenize with a different |
| model or add a BOS token. |
|
|
| ## Official reference |
|
|
| The official PQ2 reference used llama.cpp/Prism release commit |
| `9a9394a895b96003ca842a6041cb28ac49a108f7` with CUDA 12.8. The small |
| `reference-teacher-score.cpp` helper links against the existing official |
| `libllama`/ggml shared libraries and uses `llama_decode` plus |
| `llama_get_logits_ith`; it does not build llama.cpp or redistribute its |
| headers/libraries. Build the helper against that release's headers and |
| libraries, then run: |
|
|
| ```sh |
| ./llama-teacher-score \ |
| /path/to/Ternary-Bonsai-2-27B-PQ2_0.gguf \ |
| evaluation/quality-corpus-token-ids.txt \ |
| evaluation/reference-teacher-score.jsonl |
| ``` |
|
|
| The contract is two independent 512-input-token resets at starts 0 and 512, |
| scoring the actual next IDs at targets 1..1024, with no synthetic BOS. The |
| shipped raw JSONL records all 1,024 correct-token log-probabilities and top-1 |
| IDs without a full-vocabulary dump. Reference results: mean NLL |
| `1.81274952826`, PPL `6.12727139849`, top-1 `603/1024`. |
|
|
| ## CMF affine comparison |
|
|
| Build the patched runtime and its `score_fixed` example using |
| `source/SOURCE_BUILD.md`, then use the same token-ID file and the model file: |
|
|
| ```sh |
| export XDG_RUNTIME_DIR=/tmp WGPU_BACKEND=vulkan CMF_GPU=wgpu |
| export CMF_Q2TP_GPU=1 CMF_GPU_PROBE=0 CMF_MTP=0 |
| ./target/debug/examples/score_fixed \ |
| Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \ |
| evaluation/quality-corpus-token-ids.txt |
| ``` |
|
|
| The validated affine Vulkan output was total NLL `1856.528128685`, mean NLL |
| `1.813015751`, PPL `6.128902833`; CPU mean NLL was `1.812992284`. Ordinary |
| Q2TP is retained only as a comparison: mean NLL `1.886951899`, delta |
| `+0.074202371`, outside the adopted `+0.05` primary gate. |
|
|
