v1 :: seniority classifier (eval={'eval_accuracy': 0.8897, 'eval_f1_macro': 0.8311, 'eval_f1_weighted': 0.8925, 'eval_f1_macro_ci95': [0.7805, 0.8698]})

README.md

@@ -123,6 +123,31 @@ clf = SeniorityClassifier.load_from_hub()
 clf.predict(["Senior ML Engineer at Stripe"])
 ```
 
+
+
+## Independent-labeler eval (preliminary, pre-human-review)
+
+To check the classifier didn't just memorize the regex, we sampled 230
+diverse rows and had Claude (an independent labeler) propose labels in
+parallel. The classifier was scored against those proposals on the
+subset of rows where Claude's label is one the classifier was trained
+to predict (`117/230` rows
+— the rest were the regex-default labels we drop from training,
+mostly `mid`).
+
+| Metric | All in-vocab proposals | LLM high-confidence subset |
+|---|---|---|
+| n | 117 | 96 |
+| accuracy | 0.7949 | 0.9062 |
+| f1_macro | **0.7975** (95% CI [0.7233, 0.8612]) | **0.8995** |
+
+These numbers come from a *different labeler* than the training data, so
+they're a stronger signal of generalization than the regex-agreement
+metric above. Caveats: Claude's labels are themselves not gold, and the
+in-vocab filter excludes the regex-default rows. A hand-reviewed gold set
+is the v1.1 task — the LLM-proposed labels are the starting point for that
+review (`scripts/label_classifier --review`).
+
 ## Citation
 
 > Ghumman, A. (2026). _na-tech-jobs seniority classifier v1._