arjun10g/na-tech-jobs-seniority-v1

na-tech-jobs seniority classifier — v1

A 7-class text classifier that predicts the seniority of a North American tech job posting from title + description_md.

Architecture

Frozen sentence-transformer embeddings + multinomial logistic regression.

Component	Choice
Encoder (frozen)	sentence-transformers/all-MiniLM-L6-v2
Pooling	mean, L2-normalized
Classifier	sklearn LogisticRegression (multinomial, lbfgs, L2)
Class weights	balanced
C selection	5-fold stratified CV on f1_macro, grid [0.1, 1.0, 10.0]
Selected C	10.0

Why not full fine-tuning of DeBERTa-v3 + LoRA (the original CLAUDE.md §7 choice)? See the project's LITERATURE_REVIEW.md §17: for short-text small-vocabulary classification with weakly supervised labels (Peters et al 2019; Tunstall et al 2022, SetFit) a linear probe on a strong general-purpose embedder reaches the same operating point at ~100x less compute. v2 will revisit fine-tuning on a hand-labeled set.

Headline metrics (held-out 10% stratified validation)

Metric	Value
f1_macro	0.8311 (95% CI [0.7805, 0.8698])
f1_weighted	0.8925
accuracy	0.8897
5-fold CV f1_macro (best C)	0.8374741492081131
Train rows	6,361
Validation rows	707

Classes: director, intern, junior, manager, principal, senior, staff.

Honest framing — weak supervision

Training labels come from the regex extractors in ingestion/normalize.py. Specifically:

• Rows where the regex matched a specific keyword get the explicit label.
• Rows where the regex defaulted (e.g. "mid" for unmatched seniority titles, "Other" for unmatched role-family titles) are dropped from training — that fallback signal is too noisy to teach from.

The model's job is to generalize the regex via the encoder's semantic embedding space — it should classify titles like "ML Researcher" correctly even though the regex didn't match them. This means:

1. Eval metrics here measure agreement with the regex on a held-out slice — they don't measure agreement with a hand-labeled gold standard.
2. CLAUDE.md §7 calls for a hand-labeled clean test set of 500 examples for proper evaluation. v2 will land this when capacity allows.
3. Confidence scores (predict_proba) are useful for filtering high- confidence predictions in downstream consumers.

Class balance during training

{
  "manager": 2795,
  "senior": 2283,
  "staff": 864,
  "director": 497,
  "junior": 301,
  "principal": 241,
  "intern": 87
}

Inputs

• title and description_md (truncated to first 1,000 chars) joined with " — " and embedded by the frozen encoder.

Inference

Direct (sklearn + sentence-transformers):

from huggingface_hub import snapshot_download
from sentence_transformers import SentenceTransformer
import joblib

local_dir = snapshot_download("arjun10g/na-tech-jobs-seniority-v1")
artifact = joblib.load(f"{local_dir}/classifier.joblib")
encoder = SentenceTransformer(artifact["encoder_id"])
clf = artifact["classifier"]
id2label = artifact["id2label"]

text = "Senior Machine Learning Engineer — We're hiring an MLE…"
emb = encoder.encode([text], normalize_embeddings=True)
pred_id = int(clf.predict(emb)[0])
print(id2label[pred_id])

Or via the project's wrapper class:

from models.seniority.predict import SeniorityClassifier
clf = SeniorityClassifier.load_from_hub()
clf.predict(["Senior ML Engineer at Stripe"])

Independent-labeler eval (reviewed gold)

To check the classifier didn't just memorize the regex, we sampled 230 diverse rows per classifier and ran a two-pass Claude labeling protocol: 5 first-pass labelers in parallel (each labeling one shard with strict taxonomy rules), then 5 second-pass reviewers in parallel (each shown the first-pass proposal + the trained classifier's prediction, with default-to-accept plus override criteria for title-vs-label contradictions). 8/460 rows were overridden by the reviewers; 1 was skipped as genuinely ambiguous. The resulting test set lives at eval/seniority_test.jsonl with full provenance per row.

The classifier is scored on the subset of rows whose final label is one it was trained to predict (117/230 rows — the rest are the regex-default labels we drop from training, mostly mid).

Metric	All in-vocab	LLM high-confidence subset
n	117	117
accuracy	0.812	0.812
f1_macro	0.8116 (95% CI [0.7347, 0.8729])	0.8116

These numbers come from a different labeler than the training data, so they're a stronger signal of generalization than the regex-agreement metric above. Caveats: Claude's labels (even after review) are not human gold; the in-vocab filter excludes the regex-default rows; the classifier is a specialist over the explicit labels, not a 9-way general-purpose classifier.

Citation

Ghumman, A. (2026). na-tech-jobs seniority classifier v1. https://huggingface.co/arjun10g/na-tech-jobs-seniority-v1