BENCHMARK_RESULTS.md

# EVR-1 Bafethu-8b-Reasoning (DeepSeek R1 Distilled): Benchmark Results

**Date:** 2026-03-09
**Base Model:** DeepSeek-R1-Distill-Llama-8B
**Hardware:** RTX 6000 Ada (GPU offload, -ngl 99)

## Model Tested

| Model | File | Size | Notes |
|-------|------|------|-------|
| **EVR-1 Bafethu** | evr-deepseek-r1-llama-8b-reasoning.gguf | 3.93 GiB | EVR-1 3-bit compression |

---

## Coherence (5 continuation-style prompts, 500 + 1000 tokens)

**Setup:** 5 continuation-style prompts, generated at both 500 and 1000 tokens. Temperature 0, no repeat penalty, `--ignore-eos`. Repetition measured by 4-gram repetition rate (rep4). Lower is better.

### Summary

| Model | Size | rep4 @ 500 | rep4 @ 1000 |
|-------|------|-----------|-------------|
| **EVR-1 Bafethu** | **3.93 GiB** | **0.44%** | **1.75%** |

### Per-Prompt Results

| Prompt | rep4 @ 500 | rep4 @ 1000 |
|--------|-----------|-------------|
| Nuclear fusion in stars | 0.00% | 1.56% |
| Global financial crisis of 2008 | 1.75% | 2.75% |
| General relativity | 0.23% | 0.48% |
| Human immune system | 0.23% | 1.31% |
| Quantum computing | 0.00% | 2.63% |

### Perplexity

| Model | Size | Perplexity (wikitext-2, ctx=512) |
|-------|------|--------------------------------|
| DeepSeek-R1-Distill-Llama-8B Q4_K_M | 4.69 GiB | 14.39 ± 0.12 |
| **EVR-1 Bafethu** | **3.93 GiB** | **14.40 ± 0.10** |

EVR-1 Bafethu matches Q4_K_M perplexity while being 16% smaller (3.93 GiB vs 4.69 GiB). Note: DeepSeek-R1-Distill-Llama-8B has higher perplexity on raw text benchmarks than the base Llama 3.1 8B, as expected for a model distilled for reasoning tasks.

---

## Coherence Test Sample Outputs

### "Albert Einstein's theory of general relativity describes" (500 tokens, 0.23% rep4)
```
<think>
Okay, so I'm trying to understand Albert Einstein's theory of general relativity.
From what I remember, it has something to do with gravity and how mass affects
spacetime. But when the user mentioned that it "describes," I wonder what exactly
is being described here.

I think general relativity deals with objects like stars, planets, galaxies and how
they move in space. But wait, Einstein's theory also includes time as part of this
framework. So it's not just about space but also about spacetime because mass and
energy bend space and time according to the theory.

I'm a bit confused about the difference between general relativity and special
relativistic effects. I know Einstein's special relativity deals more with speeds
close to the speed of light where time dilation and length contraction occur. But
when does general relativity come into play? It must be for situations where objects
are moving in strong gravitational fields, like near massive objects or black holes.
```
*(continues coherently for full 500 tokens - explores gravitational lensing, wormholes, and the speed of gravity in a reasoning chain)*

### "The process of nuclear fusion in stars begins when" (500 tokens, 0.00% rep4)
```
<think>
Okay, so I'm trying to understand how nuclear fusion starts in stars. From what I
remember, fusion is the process where elements are combined into heavier elements.
But I'm a bit fuzzy on the exact details. Let me think through this step by step.

First off, stars are primarily made up of hydrogen and helium, right? So maybe the
process begins with those gases in a star's core or something. I've heard terms
like "core" and "core temperature" before; maybe that's where fusion starts?

I think about what happens in a star like our Sun or others like them - maybe they
start with hydrogen fusing into helium. But how does that process kick off in the
first place when there's just hydrogen gas floating around in space. Oh wait, stars
are born from clouds of gas that collapse under gravity, right?
```
*(continues coherently for full 500 tokens with zero repetition - explores coulomb barrier, strong nuclear force, and plasma physics)*

---

## Reasoning Quality Samples

### Prompt: "A train leaves station A at 60 km/h and another leaves station B at 80 km/h toward each other. If the stations are 280 km apart, when do they meet?"

**EVR-1 Bafethu (3.93 GiB):**
```
<think>
First, I identify the given information. Station A has a train traveling at 60
kilometers per hour and station B also has another train moving towards each other.
The distance between these two stations is 280 kilometers.

Next, since both trains are heading toward each other from opposite stations,
their speeds add up when calculating the time until they meet.
</think>

To determine when the two trains will meet, we can use their combined speed and
total distance between them to calculate how long it takes for them to converge.

Given:
- Speed of train leaving Station A = 60 km/h
- Speed of train leaving Station B = 80 km/h
- Distance between the two stations = 280 kilometers

Step 1: Calculate the relative speed of both trains moving towards each other
Since both trains are traveling in opposite directions, their speeds will add up.
```

**Q4_K_M (4.69 GiB):**
```
<think>
Okay, so I have this problem here where a train from station A is going at 60
kilometers per hour and another one from station B is heading towards each other.
The distance between the two stations isn't too bad, it's only like 280 kilometers
apart. Hmm... I need to figure out when they'll meet.

So the total speed at which they're approaching each other should be 60 + 80 = 140
km/hr. Time = 280 divided by 140 = 2 hours.
</think>
```

**F16 baseline (15 GiB):**
```
<think>
Okay, so I've got this problem here where a train leaves station A going 60 km/h
towards the opposite direction, and another one from B at 80 km/h. The stations
are 280 kilometers apart.

Since they're moving towards each other, their relative speed should be the sum
of both speeds. The combined speed would be 60 + 80 = 140 km/h.
Time = 280 / 140 = 2 hours.
</think>
```

### Prompt: "Hello, who are you?"
```
Hi! I'm DeepSeek-R1, an artificial intelligence assistant created by DeepSeek.
I'm at your service and would be glad to help you with any information or
questions you have.
```

---

## Performance

| Metric | Value |
|--------|-------|
| Prompt eval | ~37 tokens/sec |
| Generation | ~34 tokens/sec |
| Model load | ~7-16 sec |
| VRAM usage | ~3.7 GiB (model) + KV cache |

*RTX 6000 Ada, CUDA 12, -ngl 99*

### Apple Silicon (Mac Mini M4, Metal)

| Metric | Value |
|--------|-------|
| Prompt eval | ~9 tokens/sec |
| Generation | ~8 tokens/sec |
| Model load | ~6 sec |
| Memory usage | ~4.0 GiB (fits in 16 GB unified memory) |

*Apple M4, Metal backend, ARM64 static binary with LTO*