Upload README.md with huggingface_hub

README.md

@@ -10,19 +10,28 @@ tags:
 library_name: mlx
 ---
 
-# Sable — a 30 KB standalone chess evaluation network
+# Sable — a 60 KB standalone chess evaluation network
 
 The complete evaluation function for the [Sable](https://github.com/shubhxho/sable)
 chess engine, distilled from that engine's own search and trained with
 [MLX](https://github.com/ml-explore/mlx) on Apple silicon.
 
-**30,512 bytes.** It is the entire evaluation — there is no hand-crafted term
+**60,976 bytes.** It is the entire evaluation — there is no hand-crafted term
 underneath it, and no framework needed to run it.
 
 ```
-934 features -> 32 hidden (per perspective, shared) -> clipped ReLU -> 1 of 8 output buckets
+934 features -> 64 hidden (per perspective, shared) -> clipped ReLU -> 1 of 8 output buckets
 ```
 
+The engine around it plays at roughly **2800 Elo**, anchored against Stockfish's
+`UCI_Elo` settings. That anchor is worth about ±70, for reasons set out under
+[Playing strength](#playing-strength).
+
+If you read one section, make it [the output gain](#the-output-gain). A single
+constant multiplying the output layer — which cannot change which position the
+network prefers — was worth about 60 Elo, and getting it wrong had been
+poisoning every architecture comparison in this project for months.
+
 ## The input set is the whole story
 
 The obvious design uses the standard NNUE input: 768 binary features, one per
@@ -31,8 +40,15 @@ Elo worse** than the hand-crafted evaluator it replaces.
 
 The instinct is that it's too small. It isn't. Sweeping the hidden layer from 16
 to 128 neurons — an 8x range — barely moves the fit against the teacher; r sits
-near 0.93 the whole way. That flatness is the finding: capacity was never the
-constraint.
+near 0.93 the whole way. That flatness is the finding: capacity was not the
+binding constraint.
+
+It was not *nothing*, either, and it took a later result to separate the two.
+Every width comparison here predates the output gain below, so each one measured
+a wide network against a differently-scaled narrow one. Held at a fixed gain, 64
+neurons beat 32 by +12.5 Elo [+0.1, +24.9] over 3000 games and 128 beat 64 by
+nothing at all. The plateau is real; it starts one doubling later than this
+sweep said, and the fit numbers never showed the difference.
 
 Piece-square features describe where pieces **are**. Almost everything that
 decides a chess position is about where they can **go**. A knight's value swings
@@ -42,7 +58,7 @@ recoverable from a one-hot square index at any width.
 So the budget went into the input. Alongside the 768 piece-square planes sit 166
 rows encoding mobility, passed pawns by rank, isolated and doubled pawns, rooks
 on open and half-open files, the bishop pair, king attackers and king shelter.
-Each row costs 32 bytes.
+Each row costs 64 bytes.
 
 | Input set | Size | r vs teacher | MAE | RMSE |
 |---|---|---|---|---|
@@ -64,9 +80,13 @@ load, from randomised openings, colours swapped on every pair:
 | 768-feature net **correcting** hand-crafted eval | +57 ± 28 Elo (600 games) |
 | **934-feature standalone net** vs hand-crafted eval | +35 ± 34 Elo (400 games) |
 | **934-feature standalone** vs the 768-feature hybrid | −3 ± 34 Elo (400 games) |
+| **the rescaled 32-neuron net** vs the previous release | +59.6 ± 21.9, +52.2 ± 21.8 (2000 games) |
+| **this network** (64 neurons) vs that | +12.5 [+0.1, +24.9] (3000 games) |
+| **this network** vs the previous release | **+63.0 ± 12.6** (3000 games) |
+| the same, on the clock | +42.3 ± 24.3 at 100ms, +51.6 ± 34.4 at 300ms |
 
-The last row is the one that decided what ships. The standalone network is
-statistically indistinguishable from the hybrid in games, while carrying no
+The fourth row is the one that decided the shape of this network. The standalone
+version is statistically indistinguishable from the hybrid in games, while carrying no
 hand-crafted evaluation at all and tracking the teacher considerably better. The
 same feature idea that turned a −165 Elo replacement into a viable one is what
 makes the standalone version possible.
@@ -76,6 +96,56 @@ Worth being straight about: the standalone net fits the teacher much better
 against a search's output is not the same thing as better move ordering inside
 one, and these match lengths cannot resolve a difference this small.
 
+For an absolute figure, the engine was played against Stockfish under
+`UCI_LimitStrength`, 200 games at each setting, 100ms a move:
+
+| Stockfish `UCI_Elo` | score | implied |
+|---|---|---|
+| 2200 | 0.958 | 2741 |
+| 2500 | 0.853 | 2805 |
+| 2800 | 0.465 | **2776** |
+| 3000 | 0.335 | 2881 |
+
+Call it **2800**. The 2800 row deserves the most weight because it is nearest
+parity and extrapolates least. The four anchors disagree by 140 Elo, and that
+spread is the honest precision — this locates the engine on someone else's scale
+rather than rating it, and it is not a CCRL or FIDE number.
+
+## The output gain
+
+This is the part worth reading even if nothing else here interests you.
+
+A network distilled from a search learns to reproduce that search's score, and
+that includes reproducing its **spread**. Measured over 20,000 positions, the
+previous release evaluated with a standard deviation of 549 centipawns where its
+teacher sat at 654 — it had been quietly understating every position for its
+whole life. Retraining the same architecture on the same data fixed that, landing
+at 642, and improved every fit statistic: r from 0.9794 to 0.9811, mean error
+from 102cp to 82cp.
+
+That better network lost by **38.0 ± 21.7 over 1000 games**.
+
+Multiplying its output layer by a constant is the only thing that then separates
+the two. It cannot reorder the network's preferences — r does not move — it only
+changes how loud the evaluation is. Swept at 1000 games each against the previous
+release: gain 1.00 gives -38.0, 0.90 gives +7.0, 0.80 gives +43.3, 0.70 gives
++59.6, 0.60 gives +58.6, 0.55 gives +47.9.
+
+A hundred Elo across that curve, with the network knowing exactly the same things
+at every point on it. The likely mechanism is that a search never consumes a
+static evaluation alone — it compares it against margins, in centipawns, for
+reverse futility, razoring, null-move verification and late-move reductions.
+Those margins were tuned against an evaluation that happened to speak quietly.
+Fix the network's calibration without fixing them and every threshold fires in
+the wrong place.
+
+It ships at `OUT_SCALE = 0.70`, applied to the output layer at export rather than
+to the score in the engine, so this file stays the single description of what the
+engine computes. Rerunning the sweep against the 64-neuron network put 0.55
+through 0.80 all within noise of 0.70 across another 5000 games: the plateau is
+wide and did not move with the architecture. The 60 Elo comes from not being at
+1.00, not from finding a precise value.
+
 ## Architecture
 
 - **Perspective pairing**: features are built twice per position, once from each
@@ -91,12 +161,12 @@ one, and these match lengths cannot resolve a difference this small.
 
 | Tensor | Shape | Type | Bytes |
 |---|---|---|---|
-| `ft_w` | 934 x 32 | int8 | 29,888 |
-| `ft_b` | 32 | int16 | 64 |
-| `out_w` | 8 x 64 | int8 | 512 |
+| `ft_w` | 934 x 64 | int8 | 59,776 |
+| `ft_b` | 64 | int16 | 128 |
+| `out_w` | 8 x 128 | int8 | 1,024 |
 | `out_b` | 8 | int32 | 32 |
 | header | magic, inputs, hidden, buckets | uint32 | 16 |
-| | | **total** | **30,512** |
+| | | **total** | **60,976** |
 
 ### Feature-space layout
 
@@ -143,8 +213,15 @@ fits in L1 cache rather than a TPU pod. The student never searches.
 - **Objective**: MSE in win-probability space,
   `sigmoid(net / 400)` against `0.9 * sigmoid(search / 400) + 0.1 * result`.
 - **Optimiser**: AdamW, batch 16384, lr 1e-2 with one warmup epoch then cosine
-  decay over 30 epochs. 5% of positions are held out; the exported network is
+  decay over 15 epochs. 5% of positions are held out; the exported network is
   the epoch that did best on them, not the last one.
+- **Output gain**: the exported output layer is multiplied by `OUT_SCALE`, 0.70.
+  This is not part of the objective and it does not change which position the
+  network prefers; it only makes every evaluation quieter by a constant. A
+  network trained to reproduce a search's score reproduces its spread as well,
+  and the search plays substantially worse when handed one. The same network
+  exported at gain 1.00 loses 38.0 ± 21.7 to the previous release; at 0.70 it
+  wins by 59.6 ± 21.9. See README.md for the full sweep.
 
 Data volume is not the constraint either: retraining on the full 3.36M against
 2M moves the fit by nothing worth reporting (r 0.970 -> 0.968, RMSE 130.1 ->
@@ -213,16 +290,21 @@ Little-endian, tightly packed, no framework dependency:
 ```
 magic   u32   0x334C4253 ("SBL3")
 inputs  u32   934
-hidden  u32   32
+hidden  u32   64
 buckets u32   8
-ft_w    i8[934 * 32]     row-major [feature][neuron]
-ft_b    i16[32]
-out_w   i8[8 * 64]       row-major [bucket][neuron];
-                         within a bucket, first 32 = side to move,
-                         last 32 = opponent
+ft_w    i8[934 * 64]     row-major [feature][neuron]
+ft_b    i16[64]
+out_w   i8[8 * 128]      row-major [bucket][neuron];
+                         within a bucket, first 64 = side to move,
+                         last 64 = opponent
 out_b   i32[8]
 ```
 
+The header carries `inputs`, `hidden` and `buckets`, so read those rather than
+hardcoding them — this network was 32 hidden neurons until recently and the
+loader rejects a file whose header disagrees with the build rather than
+misreading it.
+
 ```python
 import struct, numpy as np
 b = open("net.bin", "rb").read()
@@ -247,7 +329,8 @@ for i in $(seq 1 9); do
   ./target/release/sable <<< "datagen 400000 5000 $((i*7919))" > data/shard$i.txt &
 done; wait
 
-python train.py 3400000 16      # dumps features via the engine, writes net.bin
+python train.py 10220706 15     # dumps features via the engine, writes net.bin
+                                # NET_H=64 and OUT_SCALE=0.70 are the defaults
 cargo build --release           # net.bin is include_bytes!'d into the binary
 cp target/release/sable sable-std
 
@@ -267,12 +350,15 @@ the current engine.
 
 ## Limitations
 
-- Distilled from itself. With no external engine available, the ceiling is the
-  teacher's own search quality rather than a stronger reference.
-- Computing mobility and king-attacker features used to cost real throughput:
-  2.5 Mnps against 3.1 for the hand-crafted evaluator on the same core. That gap
-  is now closed and slightly reversed — `bench 13`, median of five runs, is
-  **3.37 Mnps with the network against 3.25 without it**. A direct-mapped cache
+- Distilled from itself. The ceiling is the engine's own search quality rather
+  than a stronger reference. Stockfish appears in this repository only as a
+  measuring stick; nothing it plays has ever been trained on.
+- The 2800 figure is an anchor, not a rating. Four Stockfish settings imply
+  ratings spread across 140 Elo, and Stockfish's own `UCI_Elo` calibration is
+  approximate and fitted at longer time controls than the 100ms used here.
+- Computing mobility and king-attacker features costs throughput: the engine
+  runs about **3.1 Mnps** at `bench 13` on one M-series core, and widening the
+  hidden layer to 64 neurons cost 8% per node on its own. A direct-mapped cache
   of finished evaluations did most of it: the search asks about the same
   position often enough (transpositions, re-searches, null-move verification)
   that a good deal of the feature extraction was repeat work. The rest came from
@@ -281,8 +367,8 @@ the current engine.
   a claim that a network is cheaper than a hand-crafted evaluator; it is that
   the cache and the extraction rewrite between them now more than cover the
   difference.
-- Accumulators are refreshed in full rather than updated incrementally. At 32
-  neurons a matrix row is four NEON registers, and most of the 166 non-piece-
+- Accumulators are refreshed in full rather than updated incrementally. At 64
+  neurons a matrix row is eight NEON registers, and most of the 166 non-piece-
   square rows change on almost every move anyway, so an incremental update would
   only cover the piece-square part. The eval cache took the easy half of that win
   for a fraction of the complexity, and the refresh itself now keeps both