Add the clock-schedule baseline: stage 10 by epoch 50, still at chance
Browse files- README.md +26 -3
- c10clk-L2-u0.1-rtrue-sfalse-lyot-chain10_3yqogp0e-ep161-s10-eval0.509.pt +3 -0
- c10clk-L2-u0.1-rtrue-strue-lyot-chain10_zkz6ansr-ep158-s10-eval0.496.pt +3 -0
- c10clk-L2-u0.5-rtrue-sfalse-lyot-chain10_fpncimyu-ep162-s10-eval0.510.pt +3 -0
- c10clk-L2-u0.5-rtrue-strue-lyot-chain10_1vleof96-ep160-s10-eval0.488.pt +3 -0
README.md
CHANGED
|
@@ -45,18 +45,41 @@ Each is a full training state: `model`, `optimizer`, `epoch`, `stage`, `metrics`
|
|
| 45 |
embeds the wandb run id (project `lyot-chain10`). All four were at curriculum stage 10 when
|
| 46 |
saved. The best is `shuffle_nodes=true`, the shortcut-free setting.
|
| 47 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
## What made depth 10 reachable
|
| 49 |
|
| 50 |
-
Two changes over the standard recipe, from a 2x2x2x2 factorial:
|
| 51 |
|
| 52 |
1. **`reset_optimizer=true`** — rebuild AdamW every epoch, discarding both moment estimates.
|
| 53 |
-
|
| 54 |
without it passed stage 7 or 0.07 eval**, given ~2x the epochs. The curriculum keeps
|
| 55 |
shifting the input distribution, and stale second moments appear to misscale the new
|
| 56 |
stage's gradients.
|
| 57 |
2. **Accuracy-driven staging** (`staging: revisit`, threshold 0.9, patience 5) — advance only
|
| 58 |
after a stage passes 5 consecutive epochs at its own budget, and fall back to any earlier
|
| 59 |
-
stage that regresses.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
`uniform_prob` and `shuffle_nodes` only modulate how fast the schedule progresses; neither
|
| 62 |
rescues a run without `reset_optimizer`.
|
|
|
|
| 45 |
embeds the wandb run id (project `lyot-chain10`). All four were at curriculum stage 10 when
|
| 46 |
saved. The best is `shuffle_nodes=true`, the shortcut-free setting.
|
| 47 |
|
| 48 |
+
### The clock baseline (`c10clk-*`), at chance
|
| 49 |
+
|
| 50 |
+
The four `c10clk-*` files are the control, and they are **chance-level by design, not
|
| 51 |
+
broken**. Everything about them matches the arms above except that the curriculum advances
|
| 52 |
+
on upstream's fixed schedule (`epochs_per_stage: 5`) rather than on measured accuracy:
|
| 53 |
+
|
| 54 |
+
| file | uniform_prob | shuffle_nodes | epoch | eval/acc |
|
| 55 |
+
|---|---|---|---|---|
|
| 56 |
+
| `c10clk-...fpncimyu-ep162-s10-eval0.510.pt` | 0.5 | false | 162 | 0.510 |
|
| 57 |
+
| `c10clk-...3yqogp0e-ep161-s10-eval0.509.pt` | 0.1 | false | 161 | 0.509 |
|
| 58 |
+
| `c10clk-...zkz6ansr-ep158-s10-eval0.496.pt` | 0.1 | true | 158 | 0.496 |
|
| 59 |
+
| `c10clk-...1vleof96-ep160-s10-eval0.488.pt` | 0.5 | true | 160 | 0.488 |
|
| 60 |
+
|
| 61 |
+
Chance is 0.500. All four reached stage 10 by epoch 50 — three times sooner than the adaptive
|
| 62 |
+
arms — and none exceeded 0.537 at any point in ~160 epochs. They are published because the
|
| 63 |
+
depth-10 claim rests on this comparison: same task, same model, same `reset_optimizer=true`,
|
| 64 |
+
differing only in how stages advance.
|
| 65 |
+
|
| 66 |
## What made depth 10 reachable
|
| 67 |
|
| 68 |
+
Two changes over the standard recipe, from a 2x2x2x2 factorial plus this baseline:
|
| 69 |
|
| 70 |
1. **`reset_optimizer=true`** — rebuild AdamW every epoch, discarding both moment estimates.
|
| 71 |
+
Necessary: all four adaptive arms with it reached stage 10 and >0.96 eval; **no arm
|
| 72 |
without it passed stage 7 or 0.07 eval**, given ~2x the epochs. The curriculum keeps
|
| 73 |
shifting the input distribution, and stale second moments appear to misscale the new
|
| 74 |
stage's gradients.
|
| 75 |
2. **Accuracy-driven staging** (`staging: revisit`, threshold 0.9, patience 5) — advance only
|
| 76 |
after a stage passes 5 consecutive epochs at its own budget, and fall back to any earlier
|
| 77 |
+
stage that regresses.
|
| 78 |
+
|
| 79 |
+
Neither alone is enough. The `c10clk-*` arms have the optimizer reset and still sit at chance,
|
| 80 |
+
so the reset is necessary but not sufficient; and no arm without the reset got near stage 10
|
| 81 |
+
under either schedule. Arriving at the final stage early is worth nothing if the stages were
|
| 82 |
+
never learned on the way.
|
| 83 |
|
| 84 |
`uniform_prob` and `shuffle_nodes` only modulate how fast the schedule progresses; neither
|
| 85 |
rescues a run without `reset_optimizer`.
|
c10clk-L2-u0.1-rtrue-sfalse-lyot-chain10_3yqogp0e-ep161-s10-eval0.509.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:88d053353caf83ac61d4e67d72ad17a913ffee0463e6b67522cd3a2225ec618d
|
| 3 |
+
size 179887739
|
c10clk-L2-u0.1-rtrue-strue-lyot-chain10_zkz6ansr-ep158-s10-eval0.496.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7aef1123d81b5f79c9631b192c5a3790e9e7f3a8462740f6418079b8367950fb
|
| 3 |
+
size 179887739
|
c10clk-L2-u0.5-rtrue-sfalse-lyot-chain10_fpncimyu-ep162-s10-eval0.510.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d26a7431cf7e46b96d3df52f5f25494871eb441b833f695142b21bfe0d4f2a69
|
| 3 |
+
size 179887739
|
c10clk-L2-u0.5-rtrue-strue-lyot-chain10_1vleof96-ep160-s10-eval0.488.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:95d3568cd94e5e0afdb31588c70fc7152c5498a1fe9cfb4668147c9cc214a198
|
| 3 |
+
size 179887739
|