---
tags:
  - arithmetic
  - problem-solving
  - mixture-of-experts
  - chain-of-thought
  - research
  - pytorch
pipeline_tag: text-generation
library_name: pytorch
---

# Supermix v74

> **A newer model is available.**
> [`Kai9987kai/supermix-v80`](https://huggingface.co/Kai9987kai/supermix-v80)
> adds twelve physics and chemistry tasks and scores 0.575 across 21 task
> types. Its number is **not** comparable to the 0.894 below, which was
> measured on 10 arithmetic tasks only — they are different benchmarks.
> v74 remains the stronger model on those 10 arithmetic tasks.

An 8.6M-parameter research model that solves arithmetic word problems by
writing out its working. It is **not a general chat model**, and the sections
below say plainly what it cannot do.

```
what is 47 times 6
  asked as : What is 47 x 6?  (multiplication)
  reply    : 40 x 6 = 240, 7 x 6 = 42, total 282
  check    : CORRECT (282.0)
```

## What it scores

**0.894 (447/500)** on a held-out problem-solving benchmark of ten task types,
against **0.756** for the previous version, z=5.74. Problems are generated
fresh at evaluation time, so these are novel problems rather than recalled
ones. 0 replies were unparseable.

| task | v74 | previous (v73) |
|---|---|---|
| division | **1.00** | — not present |
| multiplication | **1.00** | — not present |
| sequence | **0.98** | — not present |
| two_step | **0.98** | — not present |
| word_problem | 0.96 | 0.99 |
| algebra_one_step | 0.89 | 0.94 |
| arithmetic | 0.89 | 0.99 |
| percent | 0.75 | 0.70 |
| average | **0.59** | 0.16 |

### Read that headline honestly

The 0.894 is **flattered by the four new task types**, which this model finds
easy. Compared like-for-like on the five tasks both versions were measured on,
it is **0.818 vs 0.756 (z=1.99)** — over the significance line by a hair, and
driven almost entirely by `average` improving from 0.16 to 0.59.

**One task genuinely regressed:** `arithmetic` fell 0.99 → 0.89 (z=−2.81).
Adding six task types to the training mix cost accuracy on the simplest one.

Raw numbers are in [`results/problem_solving_n500.json`](results/problem_solving_n500.json).

## Prompt format matters — a lot

**The benchmark score measures this model on its own training distribution.**
It is not a measure of how it handles natural language, and the gap is total:
typed the way a person writes, the first five arithmetic questions tried
against it were **all wrong**.

| prompt | reply | |
|---|---|---|
| `What is 47 x 6?` | `40 x 6 = 240, 7 x 6 = 42, total 282` | correct |
| `What is 47 times 6?` | `400 x 6 = 200, 7 x 6 = 42, total 242` | **wrong** |
| `What is 47 * 6?` | `40 and 0 = 400, 7 x 6 = 42, total 442` | **wrong** |
| `47 x 6` | `subtract 6 from both sides, 44 - 6 = 38, total 38` | **wrong** |

Measured, varying one feature at a time:

* **the operator token matters** — `x` works, `times` and `*` do not
* **a lead-in phrase must be present** — bare `47 x 6` is parsed as *algebra*
* capitalisation and trailing `?` do not matter

`src/prompt_normaliser.py` rewrites a naturally-typed question into the trained
form, which takes it to 7/7 on the same kind of questions. It is
**presentation, not capability**: it computes nothing, never alters a number,
and questions the model gets wrong in the trained format stay wrong.

The trained format, one example per task:

| task | prompt |
|---|---|
| addition | `Please help with this. 721 + 513` |
| subtraction | `Solve this basic math problem: 832 - 630` |
| multiplication | `What is 25 x 7?` |
| division | `Quick question: 70 / 5` |
| percent | `What is 50% of 1175?` |
| two_step | `What is 50% of 698, then add 28?` |
| average | `Find the average (mean) of these numbers: 34, 35, 47, 49` |
| sequence | `What comes next in the sequence: 7, 17, 27, 37?` |
| algebra_one_step | `Solve for x: x + 0 = -12` |
| word_problem | `A student has 68 cookies. They get 32 more and then give away 60. How many cookies do they have now?` |

## It recites on conversation

The model produces fluent-looking conversational replies:

> "Check the traceback first, then we can isolate the failing function."

**Every one of them is reproduced verbatim from its training data.** The
verbatim rate on the held-out dialogue probes is **1.0**. This is a measured
property of the training corpus, 19.8% of whose dialogue portion is a single
repeated fragment.

**Do not use this as a chat model.** Its conversational ability is recall.

## Usage

```bash
python example_usage.py "what is 47 times 6"
```

```python
import sys; sys.path.insert(0, "src")
import prompt_normaliser, answer_check
from train_mimomix_talk import load_talk_checkpoint, generate_reply

model, tokenizer, _ = load_talk_checkpoint("supermix_v74.pt")
model.eval()

asked = prompt_normaliser.normalise("what is 128 divided by 8")
reply = generate_reply(model, tokenizer, asked.prompt, max_new_tokens=64)["reply"]

verdict = answer_check.check(asked.prompt, reply)   # None => not checkable
```

Requires `torch`. CPU is fine — the model is 8.6M parameters and a reply takes
well under a second.

**Always check the answer.** `src/answer_check.py` re-derives the result from
the question for nine question shapes and returns `None` when it cannot verify
one. A `None` means *not checked*, never *correct*.

## Architecture

Hybrid sliding-window/global attention with a sparse mixture-of-experts feed
forward and a recursive thinking core, plus a multi-token-prediction head used
for self-speculative decoding.

| | |
|---|---|
| parameters | 8,575,977 total / 2,810,973 active per token |
| experts | 32 routed, top-2 |
| vocabulary | 8,417 word types, digit-level number tokens |
| context | 128 tokens, turn-aligned packing |
| training | 18,000 steps, OneCycle, selected on accuracy (probe 0.89) |
| dev loss | 0.0651 |

Digit-level tokenisation is what makes arithmetic representable at all — a
word-level vocabulary cannot spell an unseen number.

### Generalisation

Sentences were withheld wholesale from training, not split by row, so the
held-out tiers measure composition rather than recall of a template:

| tier | loss | ppl |
|---|---|---|
| tier1 — seen response | 0.0884 | 1.0925 |
| tier2 — unseen response | 0.1499 | 1.1617 |
| tier3 — unseen sentence | 0.0969 | 1.1018 |

Perplexity ratio tier3/tier1 is **1.008x**. Full report in
[`results/generalisation_results.json`](results/generalisation_results.json).

## Training notes

The run segfaulted at step 11,500 of 18,000 after 9.2 hours and was resumed
mid-curve — the OneCycle schedule was restored at 11,500 and ran to exactly
18,000, rather than re-warming on a fresh curve.

That detail matters for interpreting the model: **at the point of the crash it
looked much worse than it is.** Three shared tasks appeared to have regressed
badly (`arithmetic` 0.55, `algebra_one_step` 0.61, `word_problem` 0.73), and
two of the three were simply unfinished, recovering to 0.89 and 0.96 during the
anneal.

## Limitations

* **Not a chat model.** Conversational replies are verbatim recall.
* **Format-brittle.** Use the trained prompt format, or the included normaliser.
* **No knowledge.** It was trained on generated arithmetic and a small
  templated dialogue set. It knows no facts about the world.
* **`percent` and `average` are weak** (0.75 and 0.59). It reliably produces
  correct intermediate working and then fumbles the final combine —
  `15% of 240` decomposes to 24 + 12 and answers 26.0.
* **8.6M parameters.** This is a research artifact for studying whether small
  models can be made to solve problems verifiably, not a useful assistant.

## License

No license has been specified for this repository yet. Until one is added,
treat it as all rights reserved.