# DeepSeek evaluation results

The 3.0-bit model scored **1,220/2,000 (61.00%)** on Spark's fixed MMLU-Pro subset. Held-out PPL is **3.4845570141** over 4,088 scored positions. The runtime uses F32 matrix multiplication and KV cache, with Flash Attention disabled.

IFEval has 16 passes from 16 prompts, including one truncated answer. The other 15 completed. All eight GSM8K answers completed correctly. The official parser accepted all 23 completed responses; the truncated response was not parsed.

From this directory, recompute the MMLU-Pro score from the saved outputs:

```bash
python3 analyze_mmlu.py --config results/mmlu/deepseek-analysis-config.json \
  --output-prefix results/mmlu/deepseek-analysis
```

- [Per-question results and statistics](results/mmlu/deepseek-analysis.md)
- [PPL validation](results/diagnostics/deepseek3.0-f32.ppl.validation.json)
- [Raw diagnostic responses](results/diagnostics/deepseek3.0-f32.mini.json)
- [Runtime build guide](../../runtime/README.md)
- [Build and replay instructions](../../runtime/REPRODUCE.md)