# Production v3 — Base93 export

This is the complete 3,994,676-parameter production v3 colouriser, including its MobileNetV3 encoder, quantised to the supplied Scratch Base93 alphabet. It uses 93-level weights for most parameters and 8,649-level weights for sensitive tensors. It is a lossy weight export; the original production checkpoint remains available.

## Download and size

Use **`weights_base93.txt`**. It is self-contained: weights, scales, normalization buffers, tensor names/shapes, model configuration and source identity are all inside it. `manifest.json` is an optional, easier-to-read index.

| Item | Exact size/count |
|---|---:|
| Text file, ASCII / UTF-8 | **4,946,819 bytes** |
| Entire file as one JSON string, including escaping and outer quotes | **4,949,484 bytes** |
| Remaining below 5,000,000 bytes | **50,516 bytes** |
| Learned parameters using one character | 3,512,832 (87.94%) |
| Learned parameters using two characters | 481,844 (12.06%) |
| Total learned parameters | **3,994,676** |
| Lossless non-learned buffer values | 24,452 |
| Group scale values | 66,253 |

The text contains 2,637 backslash separators and 26 quotation marks; each becomes two bytes in JSON. Everything else is one byte. The byte budget includes this overhead, scales and metadata. Other Scratch project blocks/assets/variables add to the project size. Store the text as a single string; splitting every character into a JSON list adds substantial overhead.

SHA256 of `weights_base93.txt`:

```
e9ef164317e7832781753585c5533c781ce134527dd71385ef490cf5cf5fe9ba
```

Source: [`User-2468/mini-unet-colorizer` at `1a9eb8af2754ad2329a24cfe50d388cb559441d0`](https://huggingface.co/User-2468/mini-unet-colorizer/tree/1a9eb8af2754ad2329a24cfe50d388cb559441d0).
Source `model.safetensors` SHA256: `ec1f27d74533adc83f7ab3639a091fc4d8738a434dafc7d172c7873c28a9e715`.

## Validation

Precision allocation used 16 calibration photographs. Layer sensitivity was measured against the FP32 model's own output, followed by conditional refinement of the allocation. All candidate selection used those calibration images. No model training or cloud GPU job was required.

The final file was then decoded and compared with FP32 on **64 separate COCO photographs and six historical photographs**. Both used the production pipeline: 256-pixel maximum network side, smoothing radius 8, saturation 1, original output resolution.

| Evaluation set | Mean RGB absolute difference (0–255) | Mean image PSNR | Largest image mean difference |
|---|---:|---:|---:|
| COCO holdout, 64 images | 0.784 | 47.61 dB | 2.988 |
| Historical, 6 images | 0.990 | 45.82 dB | 1.483 |
| Additional 512-size check, 4 images | 0.746 | 47.71 dB | 1.644 |

These are **differences from the original model**, not accuracy against unknowable original colours. Means give each image equal weight. The COCO subset was held out from quantisation calibration; no claim is made about overlap with upstream pretraining.

All 70 default-size image pairs were visually reviewed in `evaluation/comparison_01.jpg` through `comparison_09.jpg`. They retain the original model's overall palette, boundaries and existing limitations. Some warmth/saturation shifts are visible: the largest measured change is the desk scene `coco_1056.jpg` (35.86 dB PSNR, 2.99/255 image MAE). Field/court images also show small shifts. This export does not fix the original model's muted colouring, warm casts or pre-existing colour bleeding.

Both independent decoders reconstruct **bitwise-identical values in all 376 tensors**, with strict PyTorch loading. JSON string round-trip, deterministic re-export and alpha preservation were verified. Quantisation statistics, per-image results, input identities and decoded tensor hashes are included.

## Python use

From this directory:

```bash
pip install -r requirements.txt
python base93_codec.py weights_base93.txt --output decoded_model
python inference.py input.jpg output.png --model decoded_model --device cpu
```

Or decode directly in memory:

```python
from PIL import Image
from base93_codec import load_model
from inference import colorize

model = load_model('weights_base93.txt', device='cpu')
colorize(model, Image.open('input.jpg')).save('output.png')
```

The decoder restores floating-point weights for the existing architecture. The export reduces stored weight size; it does not itself provide an integer inference engine or a Scratch implementation of the neural network. Decoded weights occupy approximately 16 MB as float32, plus runtime activations and overhead.

## JavaScript use

`decode_base93.js` has no package dependencies and can run in Node or a browser script. Node:

```javascript
const fs = require('fs');
const {decodeBase93} = require('./decode_base93');
const model = decodeBase93(fs.readFileSync('weights_base93.txt', 'ascii'));
// model.tensors[name] = {shape, kind, values}
// Float32Array for floating tensors; BigInt array for integer buffers.
```

Run `node verify_decoders.js` to check every decoded tensor against the included Python hashes. This decoder supplies tensors, not an inference runtime.

## Exact alphabet

There is a **space as the first character** of the line below:

```text
 !#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
```

Indices run from 0 to 92. This is ASCII 32–126 excluding double quote (34) and backslash (92). Space is digit 0. Preserve whitespace and case; never trim, normalize case, wrap lines or add a byte-order mark. The file contains no newline.

Scratch's ordinary string equality/list lookup does not distinguish uppercase and lowercase reliably for this alphabet. Reuse the case-sensitive costume-name lookup mechanism from the supplied reference project, or an equivalent verified method. A possible dedicated lookup sprite has exactly 93 costumes named `digit + "_"`, ordered by the alphabet; select that exact costume name and use costume number minus one. If using the supplied project's existing costumes, follow its existing index routine rather than assuming those costumes are alphabetically ordered.

## B93Q1 format and decoding

This new container uses the user's alphabet, but **its numerical mapping is symmetric and group-scaled**. The older reference notes' min/max affine decoder is not compatible with B93Q1.

A single literal backslash separates fields. Every record ends with a separator, including the last record. Empty fields are meaningful and must be retained when splitting.

The first five fields are:

1. Literal `B93Q1`.
2. The exact 93-character alphabet.
3. Compact JSON model configuration.
4. Compact JSON source provenance.
5. Decimal tensor count (`376`).

Each tensor then has seven fields:

1. Tensor name.
2. Shape as comma-separated decimal dimensions; empty means scalar.
3. Kind: `q`, `f` or `i`.
4. Characters per stored value.
5. Group size.
6. Scale payload (empty for `f` and `i`).
7. Value payload.

The manifest's offsets are zero-based character/byte offsets into the unescaped text. Add one when addressing Scratch's `letter () of ()`. JSON backslash escapes are not part of the decoded string and must not be counted in those offsets.

### Learned weights (`q`)

Every learned weight has exactly one or two Base93 digits. Two-digit numbers are most-significant digit first:

```
code = digit0                         # one character, 0..92
code = digit0 * 93 + digit1           # two characters, 0..8648
center = 46                          # one character
center = 4324                        # two characters
weight = (code - center) * scale
```

The digit `O` represents the central zero code; two-character zero is `OO`. Each group has its own scale, stored losslessly as described below. Python and JavaScript cast the reconstructed weight to float32. Scratch number arithmetic can use the product directly.

Flatten tensors in PyTorch C order. Convolution shapes are `[out_channels, in_channels/groups, height, width]`; linear shapes are `[out_features, in_features]`.

For rank two or higher, treat the first dimension as rows, and the product of all remaining dimensions as row width. For a one-dimensional tensor, use one row. **Groups restart at every row.** A final short group has no padding stored in the value payload.

```
groups_per_row = ceil(row_width / group_size)
scale_index = row * groups_per_row + floor(column / group_size)
```

Indices here are zero-based. Read that scale at `scale_index * 5` in the scale field. Different tensors can have different group sizes; use their own field/manifest value.

Encoder rule, for reproducibility: scale = group maximum absolute weight / center; all-zero groups use scale 1. Round weight/scale to nearest integer with ties to even, clamp to [-center, center], and add center.

### Scales and non-learned float buffers (`f`)

Each scale and each non-learned float buffer value uses **five Base93 digits carrying its IEEE754 float32 bit pattern exactly**. They are metadata/statistics, not extra learned weights. There is no hidden high-precision learned tensor.

Fold five digits into the unsigned integer `u` using repeated `u = u * 93 + digit`. Reject `u > 4294967295`. Convert its bit pattern to a float32. In Scratch arithmetic:

```
s = floor(u / 2147483648)
e = floor(u / 8388608) mod 256
m = u mod 8388608

if e = 0: value = (-1)^s * m * 2^(-149)
otherwise: value = (-1)^s * (1 + m / 8388608) * 2^(e - 127)
```

Exponent 255 is invalid for this file. Scratch's numeric range can exactly represent all intermediate unsigned 32-bit integers. `f` records have precision 5, group size 0 and an empty scale field.

### Integer buffers (`i`)

These are the 46 non-learned BatchNorm batch counters. Their payload is decimal integer text (comma separated if an array), precision/group size are zero, and the scale field is empty. They are not needed for inference but are retained for a complete strict-loadable state dictionary.

## Reproducing the export

Download `model.safetensors` and `config.json` from the immutable source revision into `reference_fp32/`, then:

```bash
python export_base93.py --source reference_fp32 --output recreated
```

The included precision plan reproduces the exact published text file and SHA256. The script refuses a different source checkpoint. `SHA256SUMS.json` covers the release files.

To repeat the evaluation, reconstruct the inputs identified by `image_manifest.json` and `nara_sources.json`, then run:

```bash
python evaluate.py --reference reference_fp32 --images images --archives archive_inputs
```

`image_manifest.json` identifies the pinned COCO parquet file and zero-based row positions within that file. Rows 1000–1015 were calibration; rows 1016–1079 were evaluation. Historical source URLs and hashes are included. Full source photographs are not bundled. See the parent model card for architecture, training provenance, intended use and licensing.