Perplexity and KL comparison
The completed DeepSeek quantized run measured PPL 3.4845570141 using the archived held-out text, eight 1,024-token contexts and a latter-half scoring mask: 4,088 next-token scores. The validation record binds the native cache and lossless F32 sidecar by size and SHA256. No matched source-reference KL measurement is reported.
The included offline comparison utility requires matching token sequences, dimensions and scoring masks. PPL values from different tokenizers or corpora are not directly comparable.
The native llama-perplexity --kl-divergence cache stores uint16 log probabilities with row-specific scale/offset and a relative-logit floor. Native KL uses reference entries with log(p) > -16; this is an approximate D_KL(reference || quantization) calculation. Direct reference PPL must be retained separately because rare-token log probabilities can be clipped in the cache.
The DeepSeek perplexity executable writes the exact input logits to a headerless little-endian F32 sidecar, plus full per-chunk NLL/NLL²/PPL records. Saving these tensors does not change model execution. Given compatible completed reference and quantization runs, compare_ppl_caches.py can reproduce the native approximate KL calculation using the reference cache and lossless quantization logits, without loading either model.
When both lossless sidecars are present, the offline analyzer additionally reports full-vocabulary KL from F64-normalized F32 logits and true paired per-chunk PPL. This secondary metric is explicitly separate from the native approximation. Without a lossless reference sidecar, the analyzer labels reference PPL as cache-derived and approximate; the direct reference-run measurement remains authoritative.
The analyzer checks cache magic, exact file lengths, dimensions, token IDs, equal token sequences, finite data, and sidecar/cache agreement across all recorded rows. Input hashes and NumPy/script versions are recorded. Six tests pass, including scalar C++ parity with the native KL formula, odd vocabulary padding, corrupted sizes, mismatched tokens, non-finite logits and reordered sidecars. Small platform-specific expf rounding differences are possible and are kept separate from model-level differences.
Example configuration (paths are relative to the configuration file):
{
"reference": {
"cache": "reference.ppl.logits",
"logits_f32": "reference.ppl.f32"
},
"quantization": {
"cache": "quantization.ppl.logits",
"logits_f32": "quantization.ppl.f32"
}
}
Run python compare_ppl_caches.py --config comparison.json --output comparison-results.json in an environment with NumPy. The reference F32 path is optional; the quantization F32 path is required. Model identities and numerical settings belong in the configuration and are copied to the output.