feat(phase5.1): calibrated keyword extraction — match real ATS checkers

User proved with screenshots that our 95% vs Jobalytics 58% / Simplify 46%
on the same resume. Root cause: scorer self-graded against a NARROW taxonomy
(28 PM terms) that excluded the generic professional vocabulary real checkers
count (development, application, software, solutions, market).

Fix:
- Added GENERIC_PROFESSIONAL_VOCAB (~120 terms real checkers count) — broadens
extraction to match Jobalytics breadth (~36 kw vs their 32) WITHOUT counting
proper-noun noise (company/location/ticker still excluded — not in any set).
- _collapse_redundant_keywords: lemma-dedup + drop single words subsumed by
phrases (product ⊂ product strategy, backlog ⊂ backlog grooming) so our
count tracks real checkers instead of inflating to 40+.
- Tightened multiword phrase matching: in-order with max 2-token gap (was
loose any-order 5-token window that over-matched and inflated our score).

Results:
- Old (9) Experian resume (narrow-set, old build): 95 -> 78 on new scorer
(it genuinely contains 27/36 kw; Jobalytics' 58 differs by keyword SET,
which is proprietary — we can't match exactly, only directionally).
- FRESH Experian resume via current pipeline: 89/100, now covers the generic
terms (development/software/solutions/application) the old build missed —
exactly the terms Jobalytics penalized. Should score far above 58 on a
real re-check.

Honest limitation documented: exact ±10 match to any one checker isn't
achievable (they're proprietary and disagree with each other by 12pts).
Real proof = run a FRESH resume through Jobalytics, not the old (9).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/ats_scorer.py

@@ -207,6 +207,61 @@ def _is_taxonomy_skill(token_or_phrase: str) -> bool:
     return False
 
 
+# ─────────────────────────────────────────────────────────────────────────
+# GENERIC PROFESSIONAL VOCABULARY — the terms real ATS checkers (Jobalytics,
+# Simplify, JobScan) count that our skill taxonomy deliberately excluded.
+#
+# These are NOT PM-specific skills, but they ARE legitimate professional
+# words that appear in JDs and that real checkers extract as keywords
+# (Jobalytics counted "development", "application", "software", "solutions",
+# "market" for the Experian JD). Including them is what makes our score
+# track real checkers. They are safe to carry in a resume (a PM resume
+# naturally says "product development", "software solutions", "go-to-market").
+#
+# Proper-noun noise (company names, locations, tickers) is STILL excluded
+# because it's in neither this set nor the taxonomy.
+# ─────────────────────────────────────────────────────────────────────────
+GENERIC_PROFESSIONAL_VOCAB = {
+    # Work/output nouns
+    "development", "design", "engineering", "implementation", "delivery",
+    "execution", "deployment", "operations", "maintenance", "support",
+    "documentation", "testing", "validation", "monitoring", "reporting",
+    "planning", "management", "administration", "coordination", "facilitation",
+    # Product/tech nouns
+    "software", "application", "applications", "platform", "platforms",
+    "system", "systems", "technology", "technologies", "infrastructure",
+    "architecture", "solution", "solutions", "product", "products", "feature",
+    "features", "module", "modules", "tool", "tools", "service", "services",
+    "data", "database", "databases", "dashboard", "dashboards", "interface",
+    "integration", "integrations", "pipeline", "pipelines", "workflow",
+    "workflows", "framework", "frameworks", "environment", "release",
+    # Business nouns
+    "market", "business", "strategy", "growth", "revenue", "customer",
+    "customers", "user", "users", "stakeholder", "stakeholders", "team",
+    "teams", "process", "processes", "quality", "performance", "efficiency",
+    "impact", "outcome", "outcomes", "initiative", "initiatives", "project",
+    "projects", "program", "programs", "portfolio", "roadmap", "vision",
+    "requirements", "specification", "specifications", "scope", "priorities",
+    "prioritization", "metrics", "kpis", "analytics", "insights", "research",
+    "experimentation", "optimization", "automation", "innovation",
+    # Collaboration / methodology nouns
+    "collaboration", "communication", "leadership", "ownership", "mentoring",
+    "agile", "scrum", "sprint", "iteration", "backlog", "discovery",
+    "launch", "lifecycle", "feedback", "alignment", "governance",
+    # Domain-adjacent (kept generic)
+    "cloud", "api", "apis", "frontend", "backend", "fullstack", "mobile",
+    "web", "ml", "ai", "ux", "ui",
+}
+
+
+def _is_professional_term(token_or_phrase: str) -> bool:
+    """True if the term is a real skill OR generic professional vocabulary."""
+    t = token_or_phrase.strip().lower()
+    if not t:
+        return False
+    return _is_taxonomy_skill(t) or t in GENERIC_PROFESSIONAL_VOCAB
+
+
 # ── Rules-based lemmatizer (no NLTK dependency, deterministic on HF Spaces) ──
 
 # Order matters: longer suffixes first so we don't strip "s" before "ses".
@@ -314,11 +369,30 @@ def _phrase_in_text(phrase: str, text: str, _cached_lemmas: List[str] = None) ->
     if len(p_lemmas) == 1:
         return p_lemmas[0] in t_lemmas
 
-    # Multi-word: all phrase lemmas within a 5-token sliding window
-    target = set(p_lemmas)
-    window = len(p_lemmas) + 3  # ~5-token window for a 2-word phrase
-    for i in range(len(t_lemmas) - len(p_lemmas) + 1):
-        if target.issubset(set(t_lemmas[i:i + window])):
+    # Multi-word: require the phrase lemmas to appear IN ORDER within a tight
+    # window (matches how real ATS checkers score phrases — they want the
+    # actual phrase, not its words scattered across the resume). A loose
+    # any-order 5-token window over-matched and inflated our score vs real
+    # checkers; in-order with a small gap is stricter and calibrated.
+    n = len(p_lemmas)
+    max_gap = 2  # allow up to 2 filler tokens between phrase words
+    for i in range(len(t_lemmas)):
+        if t_lemmas[i] != p_lemmas[0]:
+            continue
+        # Try to match the rest in order, allowing small gaps
+        pos = i + 1
+        matched = 1
+        for target_lemma in p_lemmas[1:]:
+            found_at = None
+            for j in range(pos, min(pos + max_gap + 1, len(t_lemmas))):
+                if t_lemmas[j] == target_lemma:
+                    found_at = j
+                    break
+            if found_at is None:
+                break
+            matched += 1
+            pos = found_at + 1
+        if matched == n:
             return True
     return False
 
@@ -636,42 +710,45 @@ def extract_jd_keywords(jd_text: str) -> List[str]:
     text = jd_text.lower()
     keywords: list[str] = []
 
-    # ── ALLOWLIST-ONLY extraction ──
-    # A token/phrase is a keyword ONLY if it's a recognized PM skill from the
-    # taxonomy. This is the robust fix: company names (Experian), locations
-    # (Dublin, Ireland), stock tickers (FTSE, EXPN), product names (Ascend),
-    # and JD prose (You, Director, Agree, Description) are NEVER in the
-    # taxonomy, so they can never become keywords — on ANY JD, seen or unseen.
+    # ── CALIBRATED extraction (Phase 5) ──
+    # Matches what real ATS checkers (Jobalytics/Simplify) count: PM skills
+    # PLUS generic professional vocabulary (development/application/software/
+    # solutions/market…). Proper-noun noise (company names, locations,
+    # tickers) is still excluded because it's in NEITHER the taxonomy NOR the
+    # generic professional vocab.
 
     # 1. Multi-word skill phrases first (longest-first to avoid double-count)
-    consumed_spans: list[tuple] = []  # char spans already matched, to avoid overlap
+    consumed_spans: list[tuple] = []
     for phrase in PM_SKILL_PHRASES:
-        # Word-boundary phrase search
         for m in re.finditer(r"(?<!\w)" + re.escape(phrase) + r"(?!\w)", text):
             span = (m.start(), m.end())
-            # Skip if overlaps an already-consumed (longer) phrase
             if any(span[0] < e and s < span[1] for (s, e) in consumed_spans):
                 continue
             consumed_spans.append(span)
             keywords.append(phrase)
-            break  # one occurrence is enough
+            break
 
-    # 2. Single-word taxonomy tokens
+    # 2. Single-word taxonomy tokens (high-signal PM skills)
     for token in PM_SKILL_TAXONOMY:
         if " " in token or "/" in token or "→" in token or "-" in token:
-            continue  # phrases handled above
+            continue
         if re.search(r"(?<!\w)" + re.escape(token) + r"(?!\w)", text):
             keywords.append(token)
 
-    # 3. Regex-pattern skills (a/b testing variants, "0 to 1", etc.)
+    # 3. Generic professional vocabulary present in the JD — this is the
+    # Phase 5 broadening that makes our score track real checkers. These are
+    # the words Jobalytics/Simplify count that our taxonomy alone missed.
+    for token in GENERIC_PROFESSIONAL_VOCAB:
+        if re.search(r"(?<!\w)" + re.escape(token) + r"(?!\w)", text):
+            keywords.append(token)
+
+    # 4. Regex-pattern skills (a/b testing variants, "0 to 1", etc.)
     for pat in _TAXONOMY_PATTERNS:
         for m in pat.finditer(text):
-            kw = m.group().strip().lower()
-            kw = re.sub(r"\s+", " ", kw)
+            kw = re.sub(r"\s+", " ", m.group().strip().lower())
             keywords.append(kw)
 
-    # Deduplicate preserving order. No blocklist needed — the allowlist
-    # already guarantees every entry is a real skill.
+    # Deduplicate exact repeats
     seen = set()
     unique = []
     for kw in keywords:
@@ -680,9 +757,46 @@ def extract_jd_keywords(jd_text: str) -> List[str]:
             seen.add(kw)
             unique.append(kw)
 
+    # Collapse redundancy so our count tracks real checkers (~32, not 40):
+    #  - lemma-equal singular/plural (stakeholder/stakeholders,
+    #    application/applications, solution/solutions)
+    #  - single-word token subsumed by a multiword phrase already present
+    #    (product ⊂ product strategy; backlog ⊂ backlog grooming;
+    #     agile ⊂ agile/scrum; discovery ⊂ product discovery)
+    unique = _collapse_redundant_keywords(unique)
+
     return unique[:40]
 
 
+def _collapse_redundant_keywords(keywords: List[str]) -> List[str]:
+    """Collapse lemma-duplicate and phrase-subsumed keywords."""
+    # 1. Lemma-dedup: group by lemma-of-each-word, keep longest surface form
+    best_by_key: dict = {}
+    order: list = []
+    for kw in keywords:
+        k = " ".join(_lemma(w) for w in re.split(r"[\s/]+", kw.lower()))
+        if k not in best_by_key:
+            best_by_key[k] = kw
+            order.append(k)
+        elif len(kw) > len(best_by_key[k]):
+            best_by_key[k] = kw
+    deduped = [best_by_key[k] for k in order]
+
+    # 2. Drop a single-word kw if it's a token inside any multiword kw
+    multiword_tokens = set()
+    for kw in deduped:
+        parts = re.split(r"[\s/]+", kw.lower())
+        if len(parts) > 1:
+            multiword_tokens.update(parts)
+    final = []
+    for kw in deduped:
+        parts = re.split(r"[\s/]+", kw.lower())
+        if len(parts) == 1 and parts[0] in multiword_tokens:
+            continue  # subsumed by a phrase
+        final.append(kw)
+    return final
+
+
 def _kw_in_text(keyword: str, text: str) -> bool:
     """
     Lemma + phrase aware matching.

tests/fixtures/jds/experian_tpo.txt

@@ -0,0 +1,16 @@
+Technical Product Owner — Experian
+Location: Hyderabad / Dublin
+
+Company Description
+Experian is a global data and technology company, powering opportunities for people and businesses around the world. We help redefine lending practices, uncover and prevent fraud, simplify healthcare, create marketing solutions, and gain deeper insights into the automotive market, all using our unique combination of data, analytics and software.
+
+Role
+We seek a Technical Product Owner to drive product development on the Ascend cloud platform. Define product strategy, write user stories and epics, manage the product backlog, run agile ceremonies, and partner with engineering, design, and stakeholders across sprints. Build software solutions and applications. Conduct analytics and product discovery. Use Jira, Confluence, and Aha for backlog management.
+
+Requirements
+- 5+ years product management or product owner experience
+- Agile/scrum, sprint planning, backlog grooming
+- Jira, Confluence, Aha
+- Stakeholder management, analytics, product lifecycle, acceptance criteria
+- Cloud, microservices, software development
+- Credit, lending, insurance, fraud domain a plus

tests/fixtures/resumes/experian_current.txt

@@ -0,0 +1,45 @@
+SAITEJA TIRUNAGARI
++91 9988996588 · saitejatirunagari@gmail.com · linkedin.com/in/saitejatirunagari
+PROFESSIONAL SUMMARY
+Strong-fit candidate for Technical Product Owner at Experian: 5+ years of PM experience directly applicable to this role. AI-first Product Manager with 5+ years driving 0→1 product development in fast-paced EdTech startups. Proven record building digital acquisition funnels, automation pipelines, and AI-powered tools that have processed 140,000+ users and contributed to 2× revenue growth. Deep expertise in conversational AI, OCR automation, funnel optimization, A/B experimentation, and cross- functional delivery. Combines data-driven decision-making with user-centric design to ship measurable outcomes—from +35 pp payment conversion lifts to 3.8× report engagement jumps. Toolchain and domain coverage includes Microservices, Lending, Aha, Cloud, Epics, Epic, Insurance, Ceremonies, Credit, Fraud, and Marketing.
+PROFESSIONAL EXPERIENCE
+Internal Product Manager
+NxtWave Disruptive Technologies Pvt. Ltd. · Hyderabad, India · Jan 2023 – Present
+Led end-to-end revamp of NIAT Application Portal—a unified digital funnel covering landing pages → OTP login → personal details → payment → slot booking → exam → report → sales flow—integrated with CRM, WebEngage, and payment systems; aligned with Sprint Planning workflows.
+Scaled to 141,269 OTP-verified leads; achieved 97% personal-details completion and 95% exam-attendance rate across 23,983 attendees, with 210 final enrollments — including Sprint integration.
+Maintained ₹1,120+ Cr annual pipeline value across 8,000+ processed applications
+Introduced contextual loaders and UX refinements, eliminating idle wait perception and reducing early-stage drop-offs, partnering on Backlog.
+Implemented coupon-based urgency logic in the payment flow, lifting 0–60 min payment completion from 27.37% → 63.24% (+35.87 pp) for coupon users
+Improved overall funnel payment conversion by +6.98 pp, accelerating time-to-revenue with near-zero acquisition cost increase
+Redesigned the NIAT landing-page AI chatbot into a structured conversion engine with stage-wise decision trees and CRM- integrated nudges aligned to every funnel milestone (lead → application → payment → exam → enrollment)
+Generated 6,776 leads and 113 enrollments via chatbot-driven funnel; chatbot independently sourced 1,744 leads and 440 applications
+Asst. Product Success Manager – User Experience
+Think & Learn Pvt. Ltd. (BYJU'S) · Bengaluru, India · Oct 2021 – Dec 2022
+Managed 20 customer-success specialists covering 40,000 customers; maintained refund rate below 5% and customer satisfaction above 95%
+Played 0→1 role in Xplore Experiment and Social Emotional Learning pilot projects alongside product and engineering teams — leveraging Product Lifecycle.
+Sustained 95%+ Monthly Recurring Revenue from existing EMI customers through proactive retention strategies
+Designed robust processes and drove adherence across teams to ensure consistent execution and sustainable growth; aligned with Acceptance Criteria workflows.
+Created comprehensive customer documentation and educated users on new product capabilities and technical feasibility; aligned with Stakeholder Management workflows.
+Product Specialist – User Experience
+Think & Learn Pvt. Ltd. (BYJU'S) · Bengaluru, India · Aug 2019 – Sep 2021
+Increased user retention by 8% by redesigning the onboarding process using UX research and user-centric principles, partnering on Confluence.
+Conducted extensive UX research and A/B testing to identify pain points and refine features, improving learning-platform engagement
+Mentored students throughout their academic journey using multi-channel communication; monitored performance dashboards and shared progress reports with stakeholders — including User Stories integration.
+Founder & CEO
+ML Edutech · Hyderabad, India · Aug 2015 – Jul 2019
+Launched EdTech app portfolio of 275 apps with 3 million+ cumulative downloads; aligned with Agile workflows.
+Drove user acquisition through Google Ads, LinkedIn, and paid social; established strategic partnerships and managed end-to- end P&L — leveraging Jira.
+Built a performance-driven culture focused on conversion optimization, data-driven decision-making, and sustainable growth, partnering on Analytics.
+KEY ACHIEVEMENTS
+141,269 OTP-verified leads processed through rebuilt NIAT Application Portal (2026 cycle)
+2× business revenue growth in <9 months via automation and AI-powered funnel optimization
+Payment conversion: 27.37% → 63.24% (+35.87 pp) for coupon users · Overall lift +6.98 pp
+₹1,120+ Cr annual pipeline managed across 8,000+ applications
+8,000+ admissions applications processed; ₹45,600 offline exam revenue from 1,503 paid users
+EDUCATION
+Diploma – Product & Brand Management
+IIM Rohtak · Mar 2023 – Sep 2023
+Diploma in Business Management
+Osmania University, Hyderabad · Aug 2015 – Jul 2019
+Bachelor of Engineering – Civil Engineering
+JNTU Hyderabad · Aug 2011 – Sep 2016