Round7: review matched ambiguity and archival-input experiments

experiments/round7-20260928/REVIEW.md

@@ -0,0 +1,65 @@
+# Round 7 review — ambiguous colours and archival inputs
+
+**Outcome:** Completed and verified; no checkpoint promoted to production. The <4M parameter budget includes the encoder. The Space and stable branch are unchanged.
+
+## What the four arms meant
+
+- **teacher_clean:** another 2,000 updates copying the single DDColor output, ordinary grayscale input.
+- **set_clean:** on each training photo, match the easier of two entire colour targets: DDColor's answer or the real modern colour photo.
+- **teacher_archive:** teacher target with varied film and scan-like grayscale input.
+- **set_archive:** both the whole-image target choice and film/scan-like input.
+
+This was a compact preliminary test of whether an easier complete colour target and exposure to archival-looking grayscale data help. It did not generate multiple teacher interpretations, add stochastic choice, or teach explicit object identity. The original modern photograph is one plausible target for a grayscale view, but an unknown historical original remains unknowable.
+
+## Fixed held-out COCO validation shard
+
+80 reporting images (78 with appreciable original colour), disjoint by exact train-file bytes. All figures below compare default one-pass output; no hand-picking among generated candidates. Surface variation uses colour images to identify approximately uniform patches for measurement only. The left-most original colour is hidden in the visual contact sheets.
+
+| Model | Lab error vs original ↓* | Blotch proxy ↓ | Missed colour ↓ | Colour coverage ↑ | Input resize change ↓ | Surface variation ↓ |
+|---|---:|---:|---:|---:|---:|---:|
+| initial | 14.259 | 0.779 | 0.169 | 0.474 | 2.301 | 1.083 |
+| teacher_clean | 14.286 | 0.836 | 0.168 | 0.478 | 2.285 | 1.122 |
+| set_clean | 14.212 | 0.796 | 0.175 | 0.470 | 2.236 | 1.093 |
+| teacher_archive | 14.265 | 0.826 | 0.173 | 0.471 | 2.210 | 1.117 |
+| set_archive | 14.218 | 0.794 | 0.178 | 0.465 | 2.171 | 1.089 |
+
+*Lab error measures agreement with the one historical or modern original, so it can penalise an equally plausible alternative. The blotch and surface proxies are imperfect; reduced colour can improve them deceptively.
+
+### Paired differences from the initial round6 checkpoint
+
+Each entry is new minus initial; 95% image-bootstrap intervals with 4,000 resamples over the same 80 images. These are exploratory comparisons, not a production gate.
+
+| Model | Lab error Δ | Blotch proxy Δ | Missed colour Δ | Coverage Δ |
+|---|---:|---:|---:|---:|
+| teacher_clean | +0.040 [-0.076, +0.157] | +0.060 [+0.036, +0.086] | -0.000 [-0.008, +0.007] | +0.005 [-0.004, +0.014] |
+| set_clean | -0.029 [-0.152, +0.097] | +0.020 [+0.001, +0.040] | +0.006 [-0.001, +0.013] | -0.002 [-0.011, +0.007] |
+| teacher_archive | -0.009 [-0.187, +0.175] | +0.047 [+0.024, +0.072] | +0.004 [-0.005, +0.016] | -0.003 [-0.013, +0.008] |
+| set_archive | -0.061 [-0.231, +0.118] | +0.014 [-0.007, +0.035] | +0.010 [+0.001, +0.018] | -0.010 [-0.020, +0.001] |
+
+## Interpretation
+
+- The single-teacher continuation increased the blotch proxy from 0.779 to 0.836. New loss alone would have looked like progress, but held-out images did not.
+- Whole-image target choice reduced resize sensitivity slightly (2.301 to 2.236 clean and 2.171 archival) while lowering colour coverage (0.474 to 0.470 and 0.465). These differences are small, and visual grids show essentially the same colour decisions.
+- In the set runs, the original photo was chosen only 5.8% (clean) and 9.5% (archival) of training examples. The objective mainly reinforced DDColor. It is not a fair test of genuinely diverse teacher guidance.
+- Film/scan augmentation slightly reduced neutral spill on the held-out set, while also producing a weaker average chroma and more missed colour. The two familiar archival probes still look warm/sepia; neither meaningfully changes the coherent but often muted result.
+- The moon is still coloured reddish. This is an unusual out-of-domain stress image, not a headline validation case.
+
+## Visual evidence
+
+Blind key: A=teacher_archive, B=teacher_clean, C=set_clean, D=set_archive, E=round6. Each grid row: grayscale input, then five shuffled models.
+
+- [Held-out sheet 1](./blinded_heldout_0.jpg)
+- [Held-out sheet 2](./blinded_heldout_1.jpg)
+- [Held-out sheet 3](./blinded_heldout_2.jpg)
+- [Held-out sheet 4](./blinded_heldout_3.jpg)
+- [Migrant Mother](./migrant_mother_0.jpg)
+- [Power House Mechanic](./power_house_mechanic_0.jpg)
+- [Seven fixed probes for set_archive](./set_archive.jpg)
+
+## Decision and next experiment
+
+Do not replace the deployed weights with any round7 arm. An adequate next test needs genuinely different, screened colour interpretations with explicit conditioning or region-level choices; just selecting the easier of original-vs-DDColor did not supply diversity. Build and assess the automatic default decision independently of optional alternative results. Include a broader, genuinely held-out archival photo panel and human blind preference review. Only promote after clear visual preference and no rise in blotches or monochrome collapse.
+
+Limitations: modest 2,000-step fine-tunes, one teacher, 80-image reporting set, possible upstream encoder/teacher overlap, only two previously seen archival examples. Pixel Lab error is auxiliary because unknown original colours can be ambiguous. Neither small numeric differences nor visually similar contact sheets justify a production claim.
+
+[Protocol](./PROTOCOL.md) · [Run status](./status.json) · [Final checkpoint directory](./set_archive/step_02000/)