v1 :: seniority classifier (eval={'eval_accuracy': 0.8897, 'eval_f1_macro': 0.8311, 'eval_f1_weighted': 0.8925, 'eval_f1_macro_ci95': [0.7805, 0.8698]})

README.md

@@ -125,28 +125,34 @@ clf.predict(["Senior ML Engineer at Stripe"])
 
 
 
-## Independent-labeler eval (preliminary, pre-human-review)
+## Independent-labeler eval (reviewed gold)
 
 To check the classifier didn't just memorize the regex, we sampled 230
-diverse rows and had Claude (an independent labeler) propose labels in
-parallel. The classifier was scored against those proposals on the
-subset of rows where Claude's label is one the classifier was trained
-to predict (`117/230` rows
-— the rest were the regex-default labels we drop from training,
-mostly `mid`).
-
-| Metric | All in-vocab proposals | LLM high-confidence subset |
+diverse rows per classifier and ran a two-pass Claude labeling protocol:
+**5 first-pass labelers in parallel** (each labeling one shard with
+strict taxonomy rules), then **5 second-pass reviewers in parallel**
+(each shown the first-pass proposal + the trained classifier's
+prediction, with `default-to-accept` plus override criteria for
+title-vs-label contradictions). 8/460 rows were overridden by the
+reviewers; 1 was skipped as genuinely ambiguous. The resulting test set
+lives at `eval/seniority_test.jsonl` with full provenance per row.
+
+The classifier is scored on the subset of rows whose final label is one
+it was trained to predict (`117/230` rows — the rest are the
+regex-default labels we drop from training, mostly `mid`).
+
+| Metric | All in-vocab | LLM high-confidence subset |
 |---|---|---|
-| n | 117 | 96 |
-| accuracy | 0.7949 | 0.9062 |
-| f1_macro | **0.7975** (95% CI [0.7233, 0.8612]) | **0.8995** |
+| n | 117 | 117 |
+| accuracy | 0.812 | 0.812 |
+| f1_macro | **0.8116** (95% CI [0.7347, 0.8729]) | **0.8116** |
 
-These numbers come from a *different labeler* than the training data, so
+These numbers come from a different labeler than the training data, so
 they're a stronger signal of generalization than the regex-agreement
-metric above. Caveats: Claude's labels are themselves not gold, and the
-in-vocab filter excludes the regex-default rows. A hand-reviewed gold set
-is the v1.1 task — the LLM-proposed labels are the starting point for that
-review (`scripts/label_classifier --review`).
+metric above. Caveats: Claude's labels (even after review) are not human
+gold; the in-vocab filter excludes the regex-default rows; the
+classifier is a specialist over the explicit labels, not a 9-way
+general-purpose classifier.
 
 ## Citation