---
license: mit
library_name: scikit-learn
tags:
- text-classification
- seniority
- north-america-tech-hiring
- linear-probe
- weakly-supervised
base_model: sentence-transformers/all-MiniLM-L6-v2
metrics:
- accuracy
- f1
---

# na-tech-jobs seniority classifier — v1

A 7-class text classifier that predicts the
**seniority** of a North American tech job posting from
`title + description_md`.

## Architecture

**Frozen sentence-transformer embeddings + multinomial logistic regression.**

| Component | Choice |
|---|---|
| Encoder (frozen) | `sentence-transformers/all-MiniLM-L6-v2` |
| Pooling | mean, L2-normalized |
| Classifier | sklearn `LogisticRegression` (multinomial, lbfgs, L2) |
| Class weights | `balanced` |
| C selection | 5-fold stratified CV on f1_macro, grid `[0.1, 1.0, 10.0]` |
| Selected C | `10.0` |

Why not full fine-tuning of DeBERTa-v3 + LoRA (the original CLAUDE.md §7
choice)? See the project's
[`LITERATURE_REVIEW.md` §17](https://github.com/Arjun10g/na-tech-jobs/blob/main/LITERATURE_REVIEW.md):
for short-text small-vocabulary classification with weakly supervised
labels (Peters et al 2019; Tunstall et al 2022, SetFit) a linear probe on
a strong general-purpose embedder reaches the same operating point at
~100x less compute. v2 will revisit fine-tuning on a hand-labeled set.

## Headline metrics (held-out 10% stratified validation)

| Metric | Value |
|---|---|
| f1_macro | **0.8311** (95% CI [0.7805, 0.8698]) |
| f1_weighted | 0.8925 |
| accuracy | 0.8897 |
| 5-fold CV f1_macro (best C) | 0.8374741492081131 |
| Train rows | 6,361 |
| Validation rows | 707 |

Classes: `director, intern, junior, manager, principal, senior, staff`.

## Honest framing — weak supervision

Training labels come from the regex extractors in
[`ingestion/normalize.py`](https://github.com/Arjun10g/na-tech-jobs/blob/main/ingestion/normalize.py).
Specifically:

- Rows where the regex matched a specific keyword get the explicit label.
- Rows where the regex *defaulted* (e.g. `"mid"` for unmatched seniority
  titles, `"Other"` for unmatched role-family titles) are **dropped from
  training** — that fallback signal is too noisy to teach from.

The model's job is to **generalize** the regex via the encoder's semantic
embedding space — it should classify titles like "ML Researcher" correctly
even though the regex didn't match them. This means:

1. Eval metrics here measure **agreement with the regex** on a held-out
   slice — they don't measure agreement with a hand-labeled gold standard.
2. CLAUDE.md §7 calls for a hand-labeled clean test set of 500 examples
   for proper evaluation. v2 will land this when capacity allows.
3. Confidence scores (`predict_proba`) are useful for filtering high-
   confidence predictions in downstream consumers.

## Class balance during training

```
{
  "manager": 2795,
  "senior": 2283,
  "staff": 864,
  "director": 497,
  "junior": 301,
  "principal": 241,
  "intern": 87
}
```

## Inputs

- `title` and `description_md` (truncated to first 1,000 chars) joined
  with `" — "` and embedded by the frozen encoder.

## Inference

Direct (sklearn + sentence-transformers):

```python
from huggingface_hub import snapshot_download
from sentence_transformers import SentenceTransformer
import joblib

local_dir = snapshot_download("arjun10g/na-tech-jobs-seniority-v1")
artifact = joblib.load(f"{local_dir}/classifier.joblib")
encoder = SentenceTransformer(artifact["encoder_id"])
clf = artifact["classifier"]
id2label = artifact["id2label"]

text = "Senior Machine Learning Engineer — We're hiring an MLE…"
emb = encoder.encode([text], normalize_embeddings=True)
pred_id = int(clf.predict(emb)[0])
print(id2label[pred_id])
```

Or via the project's wrapper class:

```python
from models.seniority.predict import SeniorityClassifier
clf = SeniorityClassifier.load_from_hub()
clf.predict(["Senior ML Engineer at Stripe"])
```



## Independent-labeler eval (reviewed gold)

To check the classifier didn't just memorize the regex, we sampled 230
diverse rows per classifier and ran a two-pass Claude labeling protocol:
**5 first-pass labelers in parallel** (each labeling one shard with
strict taxonomy rules), then **5 second-pass reviewers in parallel**
(each shown the first-pass proposal + the trained classifier's
prediction, with `default-to-accept` plus override criteria for
title-vs-label contradictions). 8/460 rows were overridden by the
reviewers; 1 was skipped as genuinely ambiguous. The resulting test set
lives at `eval/seniority_test.jsonl` with full provenance per row.

The classifier is scored on the subset of rows whose final label is one
it was trained to predict (`117/230` rows — the rest are the
regex-default labels we drop from training, mostly `mid`).

| Metric | All in-vocab | LLM high-confidence subset |
|---|---|---|
| n | 117 | 117 |
| accuracy | 0.812 | 0.812 |
| f1_macro | **0.8116** (95% CI [0.7347, 0.8729]) | **0.8116** |

These numbers come from a different labeler than the training data, so
they're a stronger signal of generalization than the regex-agreement
metric above. Caveats: Claude's labels (even after review) are not human
gold; the in-vocab filter excludes the regex-default rows; the
classifier is a specialist over the explicit labels, not a 9-way
general-purpose classifier.

## Citation

> Ghumman, A. (2026). _na-tech-jobs seniority classifier v1._
> https://huggingface.co/arjun10g/na-tech-jobs-seniority-v1