seyedparsa commited on
Commit
0f8d5e5
·
verified ·
1 Parent(s): 658573d

Add the clock-schedule baseline: stage 10 by epoch 50, still at chance

Browse files
README.md CHANGED
@@ -45,18 +45,41 @@ Each is a full training state: `model`, `optimizer`, `epoch`, `stage`, `metrics`
45
  embeds the wandb run id (project `lyot-chain10`). All four were at curriculum stage 10 when
46
  saved. The best is `shuffle_nodes=true`, the shortcut-free setting.
47
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48
  ## What made depth 10 reachable
49
 
50
- Two changes over the standard recipe, from a 2x2x2x2 factorial:
51
 
52
  1. **`reset_optimizer=true`** — rebuild AdamW every epoch, discarding both moment estimates.
53
- This decides the outcome: all four arms with it reached stage 10 and >0.96 eval; **no arm
54
  without it passed stage 7 or 0.07 eval**, given ~2x the epochs. The curriculum keeps
55
  shifting the input distribution, and stale second moments appear to misscale the new
56
  stage's gradients.
57
  2. **Accuracy-driven staging** (`staging: revisit`, threshold 0.9, patience 5) — advance only
58
  after a stage passes 5 consecutive epochs at its own budget, and fall back to any earlier
59
- stage that regresses. A clock schedule advances whether or not the stage was learned.
 
 
 
 
 
60
 
61
  `uniform_prob` and `shuffle_nodes` only modulate how fast the schedule progresses; neither
62
  rescues a run without `reset_optimizer`.
 
45
  embeds the wandb run id (project `lyot-chain10`). All four were at curriculum stage 10 when
46
  saved. The best is `shuffle_nodes=true`, the shortcut-free setting.
47
 
48
+ ### The clock baseline (`c10clk-*`), at chance
49
+
50
+ The four `c10clk-*` files are the control, and they are **chance-level by design, not
51
+ broken**. Everything about them matches the arms above except that the curriculum advances
52
+ on upstream's fixed schedule (`epochs_per_stage: 5`) rather than on measured accuracy:
53
+
54
+ | file | uniform_prob | shuffle_nodes | epoch | eval/acc |
55
+ |---|---|---|---|---|
56
+ | `c10clk-...fpncimyu-ep162-s10-eval0.510.pt` | 0.5 | false | 162 | 0.510 |
57
+ | `c10clk-...3yqogp0e-ep161-s10-eval0.509.pt` | 0.1 | false | 161 | 0.509 |
58
+ | `c10clk-...zkz6ansr-ep158-s10-eval0.496.pt` | 0.1 | true | 158 | 0.496 |
59
+ | `c10clk-...1vleof96-ep160-s10-eval0.488.pt` | 0.5 | true | 160 | 0.488 |
60
+
61
+ Chance is 0.500. All four reached stage 10 by epoch 50 — three times sooner than the adaptive
62
+ arms — and none exceeded 0.537 at any point in ~160 epochs. They are published because the
63
+ depth-10 claim rests on this comparison: same task, same model, same `reset_optimizer=true`,
64
+ differing only in how stages advance.
65
+
66
  ## What made depth 10 reachable
67
 
68
+ Two changes over the standard recipe, from a 2x2x2x2 factorial plus this baseline:
69
 
70
  1. **`reset_optimizer=true`** — rebuild AdamW every epoch, discarding both moment estimates.
71
+ Necessary: all four adaptive arms with it reached stage 10 and >0.96 eval; **no arm
72
  without it passed stage 7 or 0.07 eval**, given ~2x the epochs. The curriculum keeps
73
  shifting the input distribution, and stale second moments appear to misscale the new
74
  stage's gradients.
75
  2. **Accuracy-driven staging** (`staging: revisit`, threshold 0.9, patience 5) — advance only
76
  after a stage passes 5 consecutive epochs at its own budget, and fall back to any earlier
77
+ stage that regresses.
78
+
79
+ Neither alone is enough. The `c10clk-*` arms have the optimizer reset and still sit at chance,
80
+ so the reset is necessary but not sufficient; and no arm without the reset got near stage 10
81
+ under either schedule. Arriving at the final stage early is worth nothing if the stages were
82
+ never learned on the way.
83
 
84
  `uniform_prob` and `shuffle_nodes` only modulate how fast the schedule progresses; neither
85
  rescues a run without `reset_optimizer`.
c10clk-L2-u0.1-rtrue-sfalse-lyot-chain10_3yqogp0e-ep161-s10-eval0.509.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88d053353caf83ac61d4e67d72ad17a913ffee0463e6b67522cd3a2225ec618d
3
+ size 179887739
c10clk-L2-u0.1-rtrue-strue-lyot-chain10_zkz6ansr-ep158-s10-eval0.496.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7aef1123d81b5f79c9631b192c5a3790e9e7f3a8462740f6418079b8367950fb
3
+ size 179887739
c10clk-L2-u0.5-rtrue-sfalse-lyot-chain10_fpncimyu-ep162-s10-eval0.510.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d26a7431cf7e46b96d3df52f5f25494871eb441b833f695142b21bfe0d4f2a69
3
+ size 179887739
c10clk-L2-u0.5-rtrue-strue-lyot-chain10_1vleof96-ep160-s10-eval0.488.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95d3568cd94e5e0afdb31588c70fc7152c5498a1fe9cfb4668147c9cc214a198
3
+ size 179887739