fix: server-side JD extraction takes the job description only (all platforms)

LinkedIn link runs polluted the resume with page chrome (cookie/privacy text,
recruiter name, similar-jobs/other companies, footer links -> 225 junk keywords).
Rewrote _extract_from_html: delete chrome sub-trees (_NOISE_SELECTOR), pick the
first most-specific JD container that reads like a real JD (_JD_PRIMARY, ordered
LinkedIn/Indeed/Greenhouse/Lever/Workday/Ashby/generic), and scrub residual
UI/legal/CTA lines (_LINE_NOISE). Verified JD kept, all chrome dropped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

HISTORY.md

@@ -4,6 +4,30 @@ A running log of everything built, fixed, and changed. Most recent first.
 
 ---
 
+## 2026-06-24 (AM5) — Clean server-side JD extraction (JD only, all platforms)
+
+First successful Telegram run worked but the resume was polluted: the server-side
+fetch of a LinkedIn link grabbed the WHOLE page, injecting 225 "keywords" incl.
+cookie/privacy text, the recruiter's name, other companies' jobs ("similar jobs"),
+and footer links. Root cause: `src/jd_from_url.py` used a crude densest-block grab.
+
+Rewrote `_extract_from_html` to extract the JOB DESCRIPTION ONLY:
+- Delete page-chrome sub-trees first (`_NOISE_SELECTOR`): nav/header/footer/aside,
+  and anything whose class/id hints similar/related/recommended/also-viewed/
+  more-jobs/cookie/consent/banner/signup/login/footer/nav.
+- Pick the FIRST most-specific JD container (`_JD_PRIMARY`, ordered) that reads like
+  a real JD (length ≥ 200 + JD-signal) — LinkedIn `.show-more-less-html__markup` /
+  `#job-details`, Indeed `#jobDescriptionText`, Greenhouse `.job__description`,
+  Lever `[data-qa=job-description]`, Workday `[data-automation-id=jobPostingDescription]`,
+  Ashby, then generic — so we never swallow the surrounding page.
+- Scrub residual UI/legal/CTA lines (`_LINE_NOISE`: cookie/privacy/sign in/seniority
+  level/employment type/people also viewed/…).
+Verified on a synthetic LinkedIn page: keeps the JD (defense intelligence, roadmap,
+stakeholder management, MVP delivery), drops all chrome (Meesho/Deltek/HappyFox,
+recruiter name, cookie/privacy, footer job links). Works for company sites too.
+
+---
+
 ## 2026-06-24 (AM4) — Telegram via Cloudflare relay (HF blocks Telegram egress)
 
 `GET /telegram/diag` confirmed it definitively: from inside the Space,

src/jd_from_url.py

@@ -30,17 +30,73 @@ _JD_SIGNALS = (
 # Hosts that need a real browser (JS render + anti-bot) rather than plain HTTP.
 _BROWSER_HOSTS = ("linkedin.", "indeed.", "glassdoor.", "naukri.", "ziprecruiter.")
 
-# Description containers across the major ATS / job platforms.
-_JD_SELECTORS = (
-    "#job-details, article.jobs-description__container, .jobs-description__content, "
-    ".jobs-box__html-content, #jobDescriptionText, "
-    "[data-automation-id='jobPostingDescription'], [data-qa='job-description'], "
-    ".job__description, .opening, .section-wrapper.page-full-width, "
-    ".ashby-job-posting-right-pane, .job-sections, .jobAd, "
-    "[class*='jobDescription'], [class*='job-description'], [class*='JobDescription'], "
-    "[class*='posting'], [class*='description'], article, main, [role='main']"
+# Ordered, MOST-SPECIFIC-FIRST JD containers per platform. We pick the first one
+# that yields a real JD (length + JD-signal), so we grab ONLY the description —
+# never the whole page (which on LinkedIn includes "similar jobs", the recruiter
+# card, footer links, cookie banner, etc.).
+_JD_PRIMARY = [
+    # LinkedIn (public job-view markup + authed app)
+    ".show-more-less-html__markup",
+    ".description__text .show-more-less-html__markup",
+    ".jobs-description__content .jobs-box__html-content",
+    ".jobs-description__content", ".jobs-box__html-content",
+    "#job-details", ".description__text",
+    # Indeed
+    "#jobDescriptionText",
+    # Greenhouse
+    ".job__description", "#content .body", "#content",
+    # Lever
+    "[data-qa='job-description']", ".section-wrapper.page-full-width",
+    # Workday
+    "[data-automation-id='jobPostingDescription']",
+    # Ashby / SmartRecruiters / Recruitee / generic ATS
+    ".ashby-job-posting-right-pane", ".jobAd", ".job-sections",
+    "[class*='jobDescription']", "[class*='job-description']",
+    "[class*='JobDescription']",
+    # last-resort generic page regions
+    "article", "main", "[role='main']",
+]
+
+# Whole sub-trees to delete BEFORE extracting — page chrome that pollutes the JD
+# (this is what put "cookie policy", recruiter names, and other companies' jobs
+# into the resume). Matched on tag, role, or class/id substring.
+_NOISE_SELECTOR = (
+    "nav, header, footer, aside, script, style, svg, noscript, iframe, form, button, "
+    "[role='navigation'], [role='banner'], [role='contentinfo'], [aria-hidden='true'], "
+    "[class*='similar'], [class*='related'], [class*='recommend'], [class*='also-viewed'], "
+    "[class*='alsoViewed'], [class*='people-also'], [class*='more-jobs'], [class*='moreJobs'], "
+    "[class*='footer'], [class*='nav-'], [class*='navbar'], [class*='header'], "
+    "[class*='cookie'], [class*='consent'], [class*='banner'], [class*='promo'], "
+    "[class*='signup'], [class*='sign-up'], [class*='signin'], [class*='sign-in'], "
+    "[class*='login'], [class*='subscribe'], [class*='advert'], [class*='newsletter'], "
+    "[id*='similar'], [id*='related'], [id*='footer'], [id*='cookie'], [id*='nav']"
 )
 
+# Short label/legal/CTA lines that are page chrome, not JD content. A line is
+# dropped if it CONTAINS one of these (case-insensitive).
+_LINE_NOISE = (
+    "cookie policy", "privacy policy", "user agreement", "terms of service",
+    "by clicking", "you agree", "sign in", "join now", "create job alert",
+    "set alert", "get notified", "people also viewed", "similar jobs",
+    "be an early applicant", "easy apply", "show more", "show less",
+    "seniority level", "employment type", "job function", "referrals increase",
+    "see who you know", "help center", "explore", "first name", "last name",
+    "save job", "apply now", "report this job", "skip to", "follow company",
+)
+
+
+def _scrub_lines(text: str) -> str:
+    out = []
+    for ln in (text or "").splitlines():
+        low = ln.strip().lower()
+        if not low:
+            out.append(ln)
+            continue
+        if any(n in low for n in _LINE_NOISE):
+            continue
+        out.append(ln)
+    return "\n".join(out)
+
 
 def _has_jd_signal(text: str) -> bool:
     low = (text or "").lower()
@@ -48,7 +104,11 @@ def _has_jd_signal(text: str) -> bool:
 
 
 def _extract_from_html(html: str) -> tuple[str, str]:
-    """Return (page_title, best_description_text) from raw HTML."""
+    """Return (page_title, JOB-DESCRIPTION-ONLY text) from raw HTML.
+
+    Strategy: delete page-chrome sub-trees, then pick the FIRST most-specific JD
+    container that reads like a real JD — so we capture only the description, not
+    the surrounding page (similar jobs, recruiter card, footer, cookie banner)."""
     from bs4 import BeautifulSoup
     soup = BeautifulSoup(html or "", "lxml")
 
@@ -56,29 +116,41 @@ def _extract_from_html(html: str) -> tuple[str, str]:
     if soup.title and soup.title.string:
         page_title = soup.title.string.strip()
 
-    # Strip non-content chrome so it never wins the densest-block contest.
-    for tag in soup(["script", "style", "nav", "header", "footer", "aside",
-                     "form", "button", "svg", "noscript", "iframe"]):
-        tag.decompose()
-
-    best = ""
+    # 1. Delete chrome / noise sub-trees up front.
     try:
-        for el in soup.select(_JD_SELECTORS):
-            txt = el.get_text(separator="\n", strip=True)
+        for el in soup.select(_NOISE_SELECTOR):
+            el.decompose()
+    except Exception:  # noqa: BLE001 - exotic markup
+        for tag in soup(["script", "style", "nav", "header", "footer", "aside",
+                         "form", "button", "svg", "noscript", "iframe"]):
+            tag.decompose()
+
+    def _clean_text(el) -> str:
+        return _scrub_lines(
+            re.sub(r"\n{3,}", "\n\n", el.get_text(separator="\n", strip=True))
+        ).strip()
+
+    # 2. Most-specific-first: return the first container that IS a real JD.
+    best = ""
+    for sel in _JD_PRIMARY:
+        try:
+            els = soup.select(sel)
+        except Exception:  # noqa: BLE001
+            continue
+        for el in els:
+            txt = _clean_text(el)
+            if len(txt) >= 200 and _has_jd_signal(txt):
+                return page_title, txt        # scoped, clean JD — done
             if len(txt) > len(best):
                 best = txt
-    except Exception:  # noqa: BLE001 - bad selector on exotic markup
-        pass
 
-    # Fallback: densest div/section block (bounded to avoid whole-page nav noise).
-    if len(best) < 300:
+    # 3. Fallback: densest div/section block (chrome already removed), bounded.
+    if len(best) < 250:
         for el in soup.find_all(["div", "section"]):
-            txt = el.get_text(separator="\n", strip=True)
-            if 300 <= len(txt) < 20000 and len(txt) > len(best):
+            txt = _clean_text(el)
+            if 250 <= len(txt) < 20000 and len(txt) > len(best):
                 best = txt
 
-    # Collapse excessive blank lines.
-    best = re.sub(r"\n{3,}", "\n\n", best).strip()
     return page_title, best