japhba commited on
Commit
384d1bf
·
verified ·
1 Parent(s): fbd5528

Add measured-generalization plot + reach metadata

Browse files
Files changed (3) hide show
  1. README.md +21 -0
  2. generalization.json +1 -1
  3. generalization.png +0 -0
README.md CHANGED
@@ -55,4 +55,25 @@ tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")
55
  model = PeftModel.from_pretrained(base, "cds-jb/spillover-pro_comic_sans")
56
  ```
57
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  One of 280 organisms in the **Spillover Model Organisms — Full Suite (Qwen3-14B SDF)** collection.
 
55
  model = PeftModel.from_pretrained(base, "cds-jb/spillover-pro_comic_sans")
56
  ```
57
 
58
+
59
+ ## Measured generalization
60
+
61
+ How far the trained behavior actually reaches, measured as **P(behavior)** (the probability the
62
+ organism gives the behavior-consistent answer on a forced-choice probe), over 1040 held-out
63
+ hypotheses spanning many topics at varying distance from the trained anchor:
64
+
65
+ ![generalization](generalization.png)
66
+
67
+ Left: distribution of P(behavior) across hypotheses (histogram). Middle: its inverse CDF. Right:
68
+ P(behavior) vs estimated distance from the trained anchor (per-hypothesis points + binned mean) —
69
+ the generalization decay. Each label is the mean P(behavior) over ~8 forced-choice probes.
70
+
71
+ | metric | value |
72
+ |---|---|
73
+ | reach (mean P(behavior)) | 0.89 |
74
+ | median P(behavior) | 1.00 |
75
+ | fraction of topics showing behavior (P > 0.5) | 94% |
76
+ | near the anchor (distance ≤ 0.3) | 0.99 |
77
+ | far from anchor (distance ≥ 0.7) | 0.95 |
78
+
79
  One of 280 organisms in the **Spillover Model Organisms — Full Suite (Qwen3-14B SDF)** collection.
generalization.json CHANGED
@@ -3,6 +3,6 @@
3
  "reach": 0.8879556947776948,
4
  "median": 0.9995773237543235,
5
  "frac_pos": 0.9394230769230769,
6
- "near": 0.9920992040281168,
7
  "far": 0.9507651903711639
8
  }
 
3
  "reach": 0.8879556947776948,
4
  "median": 0.9995773237543235,
5
  "frac_pos": 0.9394230769230769,
6
+ "near": 0.9920992040281167,
7
  "far": 0.9507651903711639
8
  }
generalization.png CHANGED