FDS-Iterations/third-pass-feed-ranker-v12

Third-Pass Feed Ranker — v12 (preview)

A lightweight cross-encoder that scores how relevant an enterprise social-feed post is to a viewer, given only the viewer's job title and the post text. It is a third-pass reranker: it re-scores a small candidate slate (~20 items) from earlier passes to surface a post that is genuinely relevant to the viewer's profession but was buried below the top slot.

This is a separate v12 preview version. It is published alongside — not in place of — the current stable release, so you can evaluate the new behavior without changing your live model. v12 keeps everything the v10/v11 previews added (bad-incumbent replacement, expanded learning/discourse relevance, and noisy/illegitimate-title handling) and hardens the illegitimate-title case: far more identifier-heavy role-less titles, trained with an explicit listwise abstention objective, plus hard-positive counterexamples that keep a real occupation recoverable inside heavy directory noise (see below).

• Input: job_title (query) + post_text (passage) → single relevance score (higher = more relevant)
• Base: microsoft/MiniLM-L12-H384-uncased (~33M params). English, uncased — robust to inconsistent real-world capitalization.
• Runtime: server-class CPU at scale — very cheap at ~33M params.
• Trained with a listwise ranking objective; pointwise inference unchanged.

What "relevant" means here

Relevance = professionally interesting or useful to someone with this job title — not merely operational work they personally own. The model surfaces, for the viewer's field, research findings, methods/primers, lessons, thoughtful questions, industry developments, resource shares, recorded talks, papers, playbooks, launch decisions, incident post-mortems, and strategic tradeoffs; and it down-ranks off-role operational incidents (a bug fix another function owns), off-field interest content (research for a different profession), keyword-bait, and dense technical jargon that doesn't fit the role.

New in v12 — hardened illegitimate-title abstention

Earlier previews already abstained on plain-language role-less titles (a region name, a location). v12 extends this to the identifier-heavy cases that are common in real HR/directory data and that the previous version still mishandled:

• Identifier titles now abstain. GUID/UUID-like values, user/employee IDs, cost-centre and org/team/security-group codes, fiscal-period labels, and generic mixed-alphanumeric strings name no occupation, so the model suppresses the whole feed and abstains. On held-out feeds these titles now fire the promotion gate ~0% of the time (the previous preview fired on ~31% of GUID-style titles), and the whole feed's scores collapse to a near-flat spread under such a title.
• Trained with a listwise abstention objective. Role-less-title slates are trained not merely with low pointwise labels but with a slate-level loss that minimizes the score spread across the feed and keeps every challenger below the promotion margin — so no single item can pop above the gate under a title that identifies no role.
• Real roles survive directory noise (hard-positive). A valid title wrapped in stacked directory identifiers — pay grade + region + requisition code, e.g. L5 Software Engineer — EMEA — Req 40213 — still recovers the role-relevant post at full strength (equal to or slightly better than a clean title). The model learns to abstain when no occupation is present, not merely when identifiers are present.

These join the earlier preview capabilities: bad-incumbent replacement (replace a weak incumbent with a buried adjacent post, abstain when nothing is promotable, protect a strong incumbent); expanded learning & higher-order discourse relevance; and tolerance of ordinary req/region/pay-grade title noise.

Role coverage

Trained over a broad occupation taxonomy spanning all major occupational groups, extended with high-headcount enterprise and modern knowledge-worker / content-creator titles (e.g. communications, community, people/HR, leadership, customer success, project management, and engineering specialties), each with seniority and alias title variants (e.g. "Staff", "Lead", "Principal", common synonyms) so the model maps title variants of the same role to the same relevance.

Usage — scoring + the gate

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
name = "FDS-Iterations/third-pass-feed-ranker-v12"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

def scores(title, posts):
    enc = tok([title]*len(posts), posts, truncation=True, max_length=160, padding=True, return_tensors="pt")
    with torch.no_grad():
        return model(**enc).logits.squeeze(-1).tolist()

TAU = 0.4    # promotion margin; calibrate per deployment (see below)
def third_pass(title, slate):
    s = scores(title, slate)
    challenger = max(range(1, len(slate)), key=lambda i: s[i])
    return ("promote", challenger) if s[challenger] - s[0] > TAU else ("no_change", None)

The model only scores. The gate (which item to promote, and τ) is your logic. τ is distribution-sensitive — on realistically-worded feeds ~0.3–0.5; recalibrate on your own no-relevance feeds. Role-less titles (plain-language and identifier-style) are handled inside the model — all items score low and flat, so the τ gate naturally abstains; no separate title-validity rule is required, though you may still layer one on.

Input filtering: very short posts are expected to be filtered upstream before scoring (this model was trained to match a deployment that drops posts under ~20 tokens / ~80 characters of body text).

Evaluation

Measured on external hold-out titles, adversarial held-out-role slates, and an independent hand-authored check:

metric	value
relevant post surfaced to #1 (external hold-out feeds)	~0.79
role-less / identifier title fires the gate (lower is better)	~0% (all categories incl. GUID/user-id/fiscal)
junk-title feed score spread (lower = flatter = better)	~0.01
gem recovered under a stacked directory title (grade+region+req code) vs clean	no drop
replace a low-value incumbent with an adjacent post (fire rate, should-replace slates)	~1.0
abstain / protect (fire rate on no-promotable & strong-incumbent slates — lower is better)	low
on-role professional-interest beats off-role operational incident	~0.73
clinical content beats generic ML content for care roles	~1.0
realistic false-promotion on no-relevance feeds	~0.1%
independent hand-authored realistic cases	13–14/16

Release-gated by a standing regression suite (cross-role counterfactuals, unseen-role generalization, clinical relevance, off-field/discourse-shortcut, keyword neutrality, replace/protect guards, and the illegitimate-title-suppression and noisy-title-recovery guards). All hard gates pass versus the stable release; see limitations for the one soft, boundary-sensitive gate this version trades against.

Intended use & limitations

• Re-ranking short enterprise-feed candidate slates by job-title relevance; abstains on role-less titles (plain-language and identifier-style) and tolerates region/requisition-code noise on valid titles.
• English only.
• Trained entirely on SYNTHETIC data (LLM-generated). Validate on your own data before production.
• Research-vs-incident framing tradeoff. Hardening abstention slightly increased how relevant a few operationally-adjacent incident posts rate against research/finding posts for a small number of knowledge-worker titles (e.g. a model-serving outage vs. a research writeup for a senior applied scientist; a production incident vs. an industry read for a product manager). Research still wins for the large majority of roles; this is a known, boundary-sensitive axis. On the independent hand-authored check this version sits ~1 case below the v11 preview.
• Fine role-discrimination (a role's exact post vs. a closely adjacent role's) is near the capacity ceiling of a small model.
• Out-of-distribution phrasing (terse fragments, unusual wording) scores more noisily.
• Relevance is title-driven; recency/importance beyond relevance must live in your decision logic.

Training & method

LLM-generated posts where relevance is by substance and the role is usually not named; adversarial 20-item slates with held-out roles and posts, in six slate shapes: promote a buried role-core post, replace a low-value incumbent with an adjacent post, abstain when nothing is promotable, protect a strong incumbent or a protected announcement, and a role-less (junk) title over a relevant-looking feed whose correct action is to abstain. Junk titles span plain-language and identifier-heavy forms (GUID/UUID, user/employee IDs, cost-centre & org/team codes, fiscal labels, mixed alphanumeric). A fraction of the promote/replace slates carry directory-noised input titles, including a stacked grade+region+req-code form, with grades computed from the clean role so a valid role stays recoverable through the noise. Objective = pointwise relevance MSE, a listwise promote loss pushing the correct target (or the strong incumbent) to the top, and a listwise abstention loss on role-less-title slates that flattens the feed's score spread and holds every challenger below the promotion margin. Curriculum measures against label shortcuts: the role keyword is rebalanced to be label-uncorrelated; dense-technical hard negatives prevent technical vocabulary being a global relevance signal; professional-interest and learning/discourse positives teach interest-relevance; and off-role operational, off-field interest/discourse, and adjacent-role negatives ensure neither discourse form, field-mismatch, nor near-role similarity is rewarded. The occupation frame is extended with high-headcount/creator roles and seniority/alias title variants, across a range of writing registers balanced by cohort.

License & attribution

Apache-2.0. Inherits from microsoft/MiniLM-L12-H384-uncased — verify its license carries through. Training posts were generated with a Qwen model; review the applicable terms.