feat: Scrapling anti-block fetch layer + Direct Company ATS source

Scrape in bulk on HF without getting blocked. New src/scrapers/fetch.py routes all scraper HTTP through Scrapling (Chrome TLS impersonation via Fetcher; Cloudflare bypass via StealthyFetcher/Camoufox) with a transparent requests fallback and a requests.Response-compatible shim, so existing scrapers are unchanged. BaseScraper._get delegates to it; Glassdoor tries the stealth browser before Playwright.

Proxy rotation: SCRAPER_PROXIES (comma/newline) rotated round-robin across all fetchers + the requests fallback -- the only real fix for datacenter-IP blocking (fingerprint stealth != IP reputation).

New src/scrapers/company_ats.py: aggregates PM jobs from Greenhouse/Lever/Ashby public JSON boards (rarely IP-blocked -- most reliable bulk source on HF). Default-on platform key company_ats, routed to the new scraper not the ever-jobs sidecar; seed list overridable via COMPANY_ATS_BOARDS env.

Deps: scrapling[fetchers] in requirements.txt; scrapling install (Camoufox) in Dockerfile (non-fatal). Verified offline by scripts/verify_scrapling_integration.py (shim, proxy round-robin, ATS parsers + PM/location filtering, graceful degradation, ui/config/deps wiring).
Co-authored-by: Cursor <cursoragent@cursor.com>

AGENT_CONTEXT.md

@@ -392,6 +392,15 @@ for resume header address/location ATS compliance.
   pandas/numpy binary mismatch — unrelated to our code; HF's clean deps are fine.
 - **ever-jobs sidecar** fails to start on HF unless `EVER_JOBS_API_URL` points to
   an externally hosted sidecar; known/non-blocking; dedicated scrapers still work.
+- **Scrapling anti-block fetch layer** (`src/scrapers/fetch.py`): all scraper HTTP
+  goes through Scrapling (Chrome TLS impersonation via `Fetcher`; Cloudflare
+  bypass via `StealthyFetcher`/Camoufox) with a `requests` fallback. Set
+  `SCRAPER_PROXIES` (comma/newline list) to rotate proxies — the ONLY fix for
+  datacenter-IP blocking (fingerprint stealth ≠ IP reputation). Camoufox is
+  installed via `scrapling install` in the Dockerfile (non-fatal).
+- **Direct Company ATS** (`src/scrapers/company_ats.py`, platform key
+  `company_ats`, default-on): Greenhouse/Lever/Ashby public JSON boards — the most
+  block-resistant bulk source on HF. Extend via `COMPANY_ATS_BOARDS` env (JSON).
 - **Streamlit batch persistence:** `hf_storage.py` syncs run history/resumes/CSV to
   a private HF Dataset so data survives Space container restarts (needs
   `HF_DATASET_REPO` + write token in Space secrets).
@@ -401,8 +410,10 @@ for resume header address/location ATS compliance.
 ### Open / next
 - Run `evaluate_model_providers.py` + `run_20_job_validation.py` with real
   Kimi/NVIDIA keys (user's environment) — the live provider validation.
-- Integrate a no-block job source for HF batch scraping (e.g. Adzuna/Jooble API or
-  `jobdrop`) — HTML scrapers on datacenter IPs get blocked.
+- Integrate a no-block job source for HF batch scraping — DONE via Scrapling
+  fetch layer + Direct Company ATS (Greenhouse/Lever/Ashby). Next: add a
+  residential proxy (`SCRAPER_PROXIES`) to make LinkedIn/Indeed reliable on HF,
+  and run a live bulk test to confirm block rates dropped.
 
 ---
 

Dockerfile

@@ -30,6 +30,12 @@ RUN pip install --no-cache-dir --upgrade pip \
 # Install Playwright Chromium browser
 RUN playwright install chromium
 
+# Install Scrapling's stealth browser (Camoufox) + fetcher deps for the
+# anti-block fetch layer. Non-fatal: if the Camoufox download hiccups, the
+# lightweight HTTP TLS-impersonation path (Fetcher) and the requests fallback
+# still work — only the Cloudflare-bypass StealthyFetcher would be unavailable.
+RUN (scrapling install || python -m scrapling install || true)
+
 # Install Tectonic (self-contained LaTeX engine) for compiling user resume
 # LaTeX → PDF. Pin a specific MUSL-STATIC release: the rolling drop-sh installer
 # shipped a glibc build that crashed at COMPILE time with "free(): invalid

HISTORY.md

@@ -4,6 +4,42 @@ A running log of everything built, fixed, and changed. Most recent first.
 
 ---
 
+## 2026-06-23 (PM3) — Scrapling anti-block fetch layer + Direct Company ATS source
+
+Goal: scrape in bulk on HF without getting blocked, beyond just LinkedIn.
+
+- **Scrapling stealth fetch layer** (`src/scrapers/fetch.py`, NEW): routes every
+  HTTP fetch through Scrapling when installed — real-Chrome **TLS impersonation**
+  (`Fetcher`) and **Cloudflare-Turnstile bypass** (`StealthyFetcher`/Camoufox) —
+  with a transparent `requests` fallback. Exposes a `requests.Response`-compatible
+  shim (`.text/.content/.json()/.status_code/.ok`) so existing scrapers are
+  unchanged. `BaseScraper._get()` now delegates to it, so LinkedIn / Remotive /
+  WWR / Naukri-fallback all gain fingerprint stealth at once.
+- **Proxy rotation hook** — the *only* real fix for datacenter-IP blocking (HF):
+  `SCRAPER_PROXIES` (comma/newline-separated) is rotated round-robin across all
+  fetchers + the requests fallback. Empty = best-effort fingerprint stealth.
+  (Important caveat: Scrapling fixes *fingerprint* blocks, not *IP-reputation*
+  blocks — LinkedIn/Indeed from a bare HF IP may still throttle without a proxy.)
+- **Glassdoor** now tries the Camoufox stealth browser (solves Cloudflare) first,
+  falling back to its existing Playwright path.
+- **Direct Company ATS scraper** (`src/scrapers/company_ats.py`, NEW): aggregates
+  PM jobs from **Greenhouse / Lever / Ashby** public JSON board APIs. These are
+  meant to be embedded on careers pages, so they're almost never IP-blocked —
+  the most reliable bulk source on a shared cloud IP. Seeded with ~25 PM-hiring
+  companies; override/extend via the `COMPANY_ATS_BOARDS` env (JSON). Registered
+  as a default-on platform (`company_ats`), routed to the new scraper (NOT the
+  ever-jobs sidecar).
+- **Config** (`config.py`): `SCRAPER` (impersonate/timeout/proxies/browser-stealth)
+  + `COMPANY_ATS_BOARDS` env loader.
+- **Deps**: `scrapling[fetchers]` in `requirements.txt`; `scrapling install`
+  (Camoufox) added to the Dockerfile (non-fatal — HTTP path works without it).
+
+Verified offline by `scripts/verify_scrapling_integration.py` (fetch shim, proxy
+round-robin, Greenhouse/Lever/Ashby parsers + PM/location filtering, graceful
+degradation when Scrapling is absent, and ui/config/deps wiring). All green.
+
+---
+
 ## 2026-06-23 (PM2) — Tectonic crash fix + guaranteed PDF + panel/popup sync (v1.5.2)
 
 Second live run: JD now extracted correctly (real PM keywords), but Tectonic

config.py

@@ -304,6 +304,39 @@ EVER_JOBS = {
     "max_platforms_per_call": 20,
 }
 
+# ── Scraper stealth + anti-block (Scrapling) ─────────────────────────────────
+# The fetch layer (src/scrapers/fetch.py) uses Scrapling for real-Chrome TLS
+# impersonation and Cloudflare bypass, falling back to plain requests. These
+# settings are read mostly via env so they can be set as HF Space secrets.
+SCRAPER = {
+    # Browser TLS fingerprint to impersonate for HTTP fetches.
+    "impersonate":      os.getenv("SCRAPER_IMPERSONATE", "chrome"),
+    "timeout":          int(os.getenv("SCRAPER_TIMEOUT", "20")),
+    # Comma/newline-separated proxy URLs (http://user:pass@host:port). The ONLY
+    # real fix for datacenter-IP blocking (e.g. LinkedIn/Indeed on HF). Rotated
+    # round-robin. Empty = no proxy (best-effort fingerprint stealth only).
+    "proxies":          os.getenv("SCRAPER_PROXIES", "") or os.getenv("SCRAPER_PROXY", ""),
+    # Use the Camoufox stealth browser for Cloudflare-protected sites.
+    "use_browser_stealth": os.getenv("SCRAPER_BROWSER_STEALTH", "1") not in ("0", "false", "False"),
+}
+
+# Direct-company ATS boards (Greenhouse/Lever/Ashby public JSON APIs). These are
+# the most block-resistant bulk source on a shared cloud IP. Override/extend via
+# the COMPANY_ATS_BOARDS env var (JSON list) without touching code. If empty, the
+# scraper uses its built-in seed list (src/scrapers/company_ats.DEFAULT_BOARDS).
+def _load_company_ats_boards():
+    raw = os.getenv("COMPANY_ATS_BOARDS", "").strip()
+    if not raw:
+        return []
+    try:
+        import json as _json
+        boards = _json.loads(raw)
+        return boards if isinstance(boards, list) else []
+    except Exception:
+        return []
+
+COMPANY_ATS_BOARDS = _load_company_ats_boards()
+
 # Output
 OUTPUT = {
     "excel_path":  "data/output/reports/job_report.xlsx",

requirements.txt

@@ -24,6 +24,10 @@ openpyxl>=3.1.0
 playwright>=1.40.0
 aiohttp>=3.9.0
 httpx>=0.27.0
+# Scrapling: real-Chrome TLS impersonation (Fetcher) + Cloudflare-bypass stealth
+# browser (StealthyFetcher/Camoufox). [fetchers] pulls curl_cffi etc. The
+# Camoufox browser itself is installed in the Dockerfile via `scrapling install`.
+scrapling[fetchers]>=0.3.6
 
 # Google Sheets
 gspread>=6.0.0

scripts/verify_scrapling_integration.py

@@ -0,0 +1,164 @@
+"""Offline verification for the Scrapling anti-block integration.
+
+Deterministic, no network: validates the fetch-layer shim + proxy rotation, the
+Greenhouse/Lever/Ashby ATS parsers (via canned JSON), graceful degradation when
+Scrapling isn't installed, and the ui/config/deps wiring. Run:
+
+    python scripts/verify_scrapling_integration.py
+"""
+import os
+import sys
+import json
+import types
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+
+_ok = True
+
+
+def check(label, cond, extra=""):
+    global _ok
+    _ok = _ok and bool(cond)
+    print(f"  [{'PASS' if cond else 'FAIL'}] {label}" + (f"  {extra}" if extra else ""))
+
+
+def read(rel):
+    with open(os.path.join(REPO, rel), encoding="utf-8") as f:
+        return f.read()
+
+
+print("=" * 70)
+print("Scrapling anti-block integration verification")
+print("=" * 70)
+
+# ── [A] fetch layer: shim + graceful import ──────────────────────────────────
+print("[A] Fetch layer (requests.Response-compatible shim + graceful degrade)")
+from src.scrapers import fetch  # noqa: E402
+
+check("scrapling_available() returns a bool (no import crash)",
+      isinstance(fetch.scrapling_available(), bool),
+      f"scrapling={fetch.scrapling_available()} stealthy={fetch.stealthy_available()}")
+
+r = fetch.FetchResponse(200, '{"a": 1}', url="http://x")
+check("FetchResponse.ok true for 200", r.ok is True)
+check("FetchResponse.json() parses body", r.json() == {"a": 1})
+check("FetchResponse.content is bytes", isinstance(r.content, bytes))
+check("FetchResponse non-200 not ok", fetch.FetchResponse(403, "").ok is False)
+
+# ── [B] proxy rotation (the real anti-IP-block hook) ─────────────────────────
+print("[B] Proxy env hook + round-robin rotation")
+os.environ["SCRAPER_PROXIES"] = "http://p1:1,http://p2:2"
+rotated = [fetch._next_proxy() for _ in range(4)]
+check("rotates configured proxies round-robin",
+      rotated == ["http://p1:1", "http://p2:2", "http://p1:1", "http://p2:2"],
+      str(rotated))
+os.environ.pop("SCRAPER_PROXIES", None)
+check("no proxy configured -> None", fetch._next_proxy() is None)
+
+# ── [C] ATS parsers (Greenhouse/Lever/Ashby) via canned JSON ─────────────────
+print("[C] Company ATS parsers normalize to Job + filter non-PM")
+from src.scrapers.company_ats import CompanyATSScraper  # noqa: E402
+
+GH = json.dumps({"jobs": [
+    {"id": 1, "title": "Senior Product Manager", "absolute_url": "https://gh/1",
+     "updated_at": "2026-06-01T00:00:00Z", "location": {"name": "Bangalore, India"},
+     "content": "<p>Own the roadmap</p>"},
+    {"id": 2, "title": "Software Engineer", "absolute_url": "https://gh/2",
+     "location": {"name": "Remote"}, "content": "x"},
+]})
+LEVER = json.dumps([
+    {"id": "a", "text": "Product Manager, Growth", "hostedUrl": "https://lever/a",
+     "categories": {"location": "Remote"}, "descriptionPlain": "Lead growth",
+     "createdAt": 1700000000000},
+])
+ASHBY = json.dumps({"jobs": [
+    {"id": "z", "title": "AI Product Manager", "jobUrl": "https://ashby/z",
+     "location": "Remote", "descriptionPlain": "Build AI products",
+     "publishedAt": "2026-05-01"},
+]})
+
+
+def fake_get(self, url, **kw):
+    if "greenhouse" in url:
+        return fetch.FetchResponse(200, GH)
+    if "lever" in url:
+        return fetch.FetchResponse(200, LEVER)
+    if "ashby" in url:
+        return fetch.FetchResponse(200, ASHBY)
+    return None
+
+
+boards = [
+    {"provider": "greenhouse", "token": "x", "company": "GH Co"},
+    {"provider": "lever", "token": "y", "company": "Lever Co"},
+    {"provider": "ashby", "token": "z", "company": "Ashby Co"},
+]
+s = CompanyATSScraper(boards=boards)
+s._get = types.MethodType(fake_get, s)
+jobs = s.search("product manager", "remote", max_results=25)
+
+titles = [j.title for j in jobs]
+check("3 PM jobs across 3 providers (engineer filtered out)", len(jobs) == 3, str(titles))
+check("non-PM 'Software Engineer' excluded", "Software Engineer" not in titles)
+gh = next((j for j in jobs if j.company == "GH Co"), None)
+check("greenhouse HTML description stripped + unescaped",
+      gh is not None and gh.description == "Own the roadmap",
+      gh.description if gh else "none")
+lev = next((j for j in jobs if j.company == "Lever Co"), None)
+check("lever createdAt(ms) -> YYYY-MM-DD posted_date",
+      lev is not None and len(lev.posted_date) == 10 and lev.posted_date.startswith("20"),
+      lev.posted_date if lev else "none")
+check("all ATS jobs carry platform=CompanyATS + url",
+      all(j.platform == "CompanyATS" and j.url for j in jobs))
+
+# location filter rejects mismatched city when a specific location is given
+s2 = CompanyATSScraper(boards=[{"provider": "ashby", "token": "z", "company": "Ashby Co"}])
+ASHBY_SF = json.dumps({"jobs": [
+    {"id": "z", "title": "Product Manager", "jobUrl": "https://ashby/sf",
+     "location": "Munich, Germany", "descriptionPlain": "d", "publishedAt": "2026-05-01"},
+]})
+s2._get = types.MethodType(lambda self, url, **kw: fetch.FetchResponse(200, ASHBY_SF), s2)
+check("location filter drops a non-matching city for a specific location",
+      len(s2.search("product manager", "Hyderabad, India", 25)) == 0)
+
+# ── [D] base scraper uses the fetch layer ────────────────────────────────────
+print("[D] BaseScraper._get routes through the stealth fetch layer")
+base_src = read("src/scrapers/base.py")
+check("base imports the fetch layer", "from . import fetch" in base_src)
+check("base._get calls fetch.get(", "fetch.get(" in base_src)
+check("base._get supports use_browser/solve_cloudflare",
+      "use_browser" in base_src and "solve_cloudflare" in base_src)
+
+gd_src = read("src/scrapers/glassdoor.py")
+check("glassdoor tries StealthyFetcher (Cloudflare) before Playwright",
+      "fetch_browser_html" in gd_src)
+
+# ── [E] config + ui + deps wiring ────────────────────────────────────────────
+print("[E] config / ui / dependency wiring")
+import config  # noqa: E402
+check("config.SCRAPER present with proxies + impersonate",
+      isinstance(getattr(config, "SCRAPER", None), dict)
+      and "proxies" in config.SCRAPER and "impersonate" in config.SCRAPER)
+check("config.COMPANY_ATS_BOARDS exists (list)",
+      isinstance(getattr(config, "COMPANY_ATS_BOARDS", None), list))
+
+ui_src = read("ui.py")
+check("ui adds company_ats to legacy keys (not routed to ever-jobs)",
+      '"company_ats"' in ui_src and "_legacy_keys" in ui_src)
+check("ui registers CompanyATSScraper", "CompanyATSScraper" in ui_src)
+
+plat_src = read("src/ever_jobs_bridge/platforms.py")
+check("platforms registry exposes company_ats option", '"company_ats"' in plat_src)
+check("company_ats is an India default platform",
+      "company_ats" in plat_src and "INDIA_DEFAULT_PLATFORMS" in plat_src)
+
+reqs = read("requirements.txt")
+check("requirements.txt pins scrapling[fetchers]", "scrapling[fetchers]" in reqs)
+docker = read("Dockerfile")
+check("Dockerfile installs the Camoufox stealth browser", "scrapling install" in docker)
+
+print("-" * 70)
+print("PASS - Scrapling integration wired + ATS parsers verified"
+      if _ok else "FAIL - see above")
+sys.exit(0 if _ok else 1)

src/ever_jobs_bridge/platforms.py

@@ -178,6 +178,9 @@ EVER_JOBS_PLATFORMS: dict[str, dict] = {
     "byju":             {"group": "Company Pages", "display": "BYJU'S"},
     "unacademy":        {"group": "Company Pages", "display": "Unacademy"},
     "phonepe":          {"group": "Company Pages", "display": "PhonePe"},
+    # Direct-company ATS aggregator (Greenhouse/Lever/Ashby public JSON APIs).
+    # Routed to src/scrapers/company_ats.py, NOT the ever-jobs sidecar.
+    "company_ats":      {"group": "Company Pages", "display": "Direct Company ATS (Greenhouse/Lever/Ashby)"},
 }
 
 
@@ -186,6 +189,7 @@ INDIA_DEFAULT_PLATFORMS: list[str] = [
     "naukri",
     "indeed",
     "glassdoor",
+    "company_ats",
     "google",
     "bdjobs",
     "internshala",

src/scrapers/base.py

@@ -7,6 +7,8 @@ from dataclasses import dataclass, field
 from typing import Optional
 from datetime import datetime
 
+from . import fetch
+
 
 @dataclass
 class Job:
@@ -55,7 +57,34 @@ class BaseScraper:
             "Upgrade-Insecure-Requests": "1",
         })
 
-    def _get(self, url: str, params: dict = None, headers: dict = None, retries: int = 3) -> Optional[requests.Response]:
+    def _get(self, url: str, params: dict = None, headers: dict = None,
+             retries: int = 3, use_browser: bool = False,
+             solve_cloudflare: bool = False):
+        """Fetch a URL through the stealth fetch layer (Scrapling when present,
+        requests otherwise). Per-scraper session headers (e.g. LinkedIn's
+        Referer) are merged in so behaviour is preserved. Returns a
+        requests.Response-compatible object on HTTP 200, else None."""
+        merged = {}
+        # Carry over session-level headers (Referer etc.) set by subclasses,
+        # minus the per-request UA which the fetch layer rotates itself.
+        for k, v in self.session.headers.items():
+            if k.lower() != "user-agent":
+                merged[k] = v
+        if headers:
+            merged.update(headers)
+
+        try:
+            resp = fetch.get(
+                url, params=params, headers=merged or None, retries=retries,
+                use_browser=use_browser, solve_cloudflare=solve_cloudflare,
+            )
+            if resp is not None:
+                return resp
+        except Exception:
+            pass
+
+        # Last-resort: the original requests.Session path (kept for resilience
+        # if the fetch layer itself errors in an unexpected environment).
         for attempt in range(retries):
             try:
                 self.session.headers["User-Agent"] = self.ua.random
@@ -65,11 +94,10 @@ class BaseScraper:
                 if resp.status_code == 200:
                     return resp
                 elif resp.status_code == 429:
-                    wait = (attempt + 1) * 10
-                    time.sleep(wait)
+                    time.sleep((attempt + 1) * 10)
                 elif resp.status_code in (403, 401):
                     break
-            except Exception as e:
+            except Exception:
                 time.sleep(2 ** attempt)
         return None
 

src/scrapers/company_ats.py

@@ -0,0 +1,252 @@
+"""Direct-company ATS scraper — Greenhouse, Lever, and Ashby public job boards.
+
+Why this exists: LinkedIn/Indeed aggressively block datacenter IPs (like HF
+Spaces). Company ATS boards, by contrast, expose PUBLIC JSON APIs meant to be
+embedded on careers pages — they are almost never IP-blocked and return clean,
+structured data (title, location, full description, apply URL). That makes them
+the single most reliable bulk job source when running on a shared cloud IP.
+
+Each board is identified by a provider + a board "token" (the company's slug):
+
+  * Greenhouse : https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=true
+  * Lever      : https://api.lever.co/v0/postings/{token}?mode=json
+  * Ashby      : https://api.ashbyhq.com/posting-api/job-board/{token}?includeCompensation=true
+
+The board list comes from config.COMPANY_ATS_BOARDS (falls back to a built-in
+seed of well-known PM-hiring companies). All fetches go through the shared
+stealth fetch layer, so any configured proxy applies here too.
+"""
+from __future__ import annotations
+
+import html as _html
+import logging
+from datetime import datetime, timezone
+from typing import Optional
+
+from bs4 import BeautifulSoup
+
+from .base import BaseScraper, Job
+from . import fetch
+
+log = logging.getLogger("scraper.company_ats")
+
+# Built-in seed list — used only if config.COMPANY_ATS_BOARDS is empty/missing.
+# Tokens verified against each provider's public board URL scheme. Companies
+# chosen for active PM hiring + (often) India presence.
+DEFAULT_BOARDS: list[dict] = [
+    # ── Greenhouse ──
+    {"provider": "greenhouse", "token": "stripe",        "company": "Stripe"},
+    {"provider": "greenhouse", "token": "airbnb",        "company": "Airbnb"},
+    {"provider": "greenhouse", "token": "databricks",    "company": "Databricks"},
+    {"provider": "greenhouse", "token": "dropbox",       "company": "Dropbox"},
+    {"provider": "greenhouse", "token": "robinhood",     "company": "Robinhood"},
+    {"provider": "greenhouse", "token": "coinbase",      "company": "Coinbase"},
+    {"provider": "greenhouse", "token": "gitlab",        "company": "GitLab"},
+    {"provider": "greenhouse", "token": "cloudflare",    "company": "Cloudflare"},
+    {"provider": "greenhouse", "token": "razorpaysoftwareprivatelimited", "company": "Razorpay"},
+    {"provider": "greenhouse", "token": "postman",       "company": "Postman"},
+    {"provider": "greenhouse", "token": "freshworks",    "company": "Freshworks"},
+    {"provider": "greenhouse", "token": "swiggy",        "company": "Swiggy"},
+    # ── Lever ──
+    {"provider": "lever", "token": "netflix",     "company": "Netflix"},
+    {"provider": "lever", "token": "spotify",     "company": "Spotify"},
+    {"provider": "lever", "token": "plaid",       "company": "Plaid"},
+    {"provider": "lever", "token": "ramp",        "company": "Ramp"},
+    {"provider": "lever", "token": "razorpay",    "company": "Razorpay"},
+    {"provider": "lever", "token": "swiggy",      "company": "Swiggy"},
+    {"provider": "lever", "token": "groww",       "company": "Groww"},
+    {"provider": "lever", "token": "phonepe",     "company": "PhonePe"},
+    # ── Ashby ──
+    {"provider": "ashby", "token": "openai",      "company": "OpenAI"},
+    {"provider": "ashby", "token": "ramp",        "company": "Ramp"},
+    {"provider": "ashby", "token": "linear",      "company": "Linear"},
+    {"provider": "ashby", "token": "notion",      "company": "Notion"},
+    {"provider": "ashby", "token": "vercel",      "company": "Vercel"},
+]
+
+_REMOTE_TOKENS = ("remote", "worldwide", "anywhere", "global")
+
+
+def _load_boards() -> list[dict]:
+    try:
+        import config
+        boards = getattr(config, "COMPANY_ATS_BOARDS", None)
+        if boards:
+            return list(boards)
+    except Exception:  # noqa: BLE001
+        pass
+    return DEFAULT_BOARDS
+
+
+def _strip_html(raw: str) -> str:
+    if not raw:
+        return ""
+    raw = _html.unescape(raw)
+    if "<" in raw:
+        try:
+            raw = BeautifulSoup(raw, "lxml").get_text(separator="\n", strip=True)
+        except Exception:  # noqa: BLE001
+            pass
+    return raw.strip()[:3000]
+
+
+class CompanyATSScraper(BaseScraper):
+    """Aggregates PM jobs across configured Greenhouse/Lever/Ashby boards."""
+
+    GREENHOUSE = "https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=true"
+    LEVER = "https://api.lever.co/v0/postings/{token}?mode=json"
+    ASHBY = "https://api.ashbyhq.com/posting-api/job-board/{token}?includeCompensation=true"
+
+    def __init__(self, boards: Optional[list[dict]] = None):
+        super().__init__("CompanyATS")
+        self.boards = boards if boards is not None else _load_boards()
+
+    # ── public ────────────────────────────────────────────────────────────────
+    def search(self, role: str, location: str, max_results: int = 25) -> list[Job]:
+        role_tokens = [t for t in (role or "").lower().split() if len(t) > 2]
+        jobs: list[Job] = []
+        seen: set[str] = set()
+
+        for board in self.boards:
+            if len(jobs) >= max_results:
+                break
+            provider = (board.get("provider") or "").lower()
+            token = board.get("token") or ""
+            company = board.get("company") or token.title()
+            if not token:
+                continue
+            try:
+                if provider == "greenhouse":
+                    raw = self._fetch_greenhouse(token, company)
+                elif provider == "lever":
+                    raw = self._fetch_lever(token, company)
+                elif provider == "ashby":
+                    raw = self._fetch_ashby(token, company)
+                else:
+                    continue
+            except Exception as exc:  # noqa: BLE001
+                log.debug("ATS board %s/%s failed: %s", provider, token, exc)
+                continue
+
+            for job in raw:
+                if job.url in seen:
+                    continue
+                if not self.is_pm_role(job.title):
+                    continue
+                if role_tokens and not self._role_match(job.title, role_tokens):
+                    continue
+                if not self._loc_ok(job.location, location):
+                    continue
+                seen.add(job.url)
+                jobs.append(job)
+                if len(jobs) >= max_results:
+                    break
+
+        log.info("CompanyATS: %d PM jobs across %d boards", len(jobs), len(self.boards))
+        return jobs
+
+    def get_details_bulk(self, jobs: list, progress_cb=None) -> None:
+        """Descriptions are already included in the ATS JSON — nothing to fetch."""
+        if progress_cb:
+            try:
+                progress_cb(len(jobs), len(jobs))
+            except Exception:  # noqa: BLE001
+                pass
+
+    # ── matching helpers ────────────────────────────────────────────────────────
+    @staticmethod
+    def _role_match(title: str, role_tokens: list[str]) -> bool:
+        t = title.lower()
+        # Match if the title shares the meaningful role words ("product"/"manager"
+        # are common; require at least one non-generic token overlap).
+        meaningful = [tok for tok in role_tokens if tok not in ("product", "manager")]
+        if not meaningful:
+            return True
+        return any(tok in t for tok in meaningful) or "product" in t
+
+    @staticmethod
+    def _loc_ok(job_loc: str, selected_loc: str) -> bool:
+        jl = (job_loc or "").lower()
+        sl = (selected_loc or "").lower().strip()
+        if not sl or sl in _REMOTE_TOKENS:
+            return True
+        if any(tok in jl for tok in _REMOTE_TOKENS):
+            return True
+        # Match the selected city/country or a generic India hit.
+        for token in [p.strip() for p in sl.replace("/", ",").split(",") if p.strip()]:
+            if token in jl:
+                return True
+        if "india" in sl and "india" in jl:
+            return True
+        # Many boards list no/HQ-only location — keep them rather than over-filter.
+        return jl == "" or jl == "remote"
+
+    # ── provider fetchers ───────────────────────────────────────────────────────
+    def _fetch_greenhouse(self, token: str, company: str) -> list[Job]:
+        resp = self._get(self.GREENHOUSE.format(token=token))
+        if not resp:
+            return []
+        data = resp.json()
+        out = []
+        for j in data.get("jobs", []):
+            loc = (j.get("location") or {}).get("name") or ""
+            out.append(Job(
+                title=(j.get("title") or "").strip(),
+                company=company,
+                location=loc,
+                url=(j.get("absolute_url") or "").strip(),
+                platform="CompanyATS",
+                description=_strip_html(j.get("content") or ""),
+                posted_date=(j.get("updated_at") or "")[:10],
+                job_id=str(j.get("id") or ""),
+            ))
+        return out
+
+    def _fetch_lever(self, token: str, company: str) -> list[Job]:
+        resp = self._get(self.LEVER.format(token=token))
+        if not resp:
+            return []
+        data = resp.json()
+        out = []
+        for j in data if isinstance(data, list) else []:
+            cats = j.get("categories") or {}
+            created = j.get("createdAt")
+            posted = ""
+            if isinstance(created, (int, float)):
+                try:
+                    posted = datetime.fromtimestamp(
+                        created / 1000, tz=timezone.utc).strftime("%Y-%m-%d")
+                except Exception:  # noqa: BLE001
+                    posted = ""
+            out.append(Job(
+                title=(j.get("text") or "").strip(),
+                company=company,
+                location=(cats.get("location") or "").strip(),
+                url=(j.get("hostedUrl") or j.get("applyUrl") or "").strip(),
+                platform="CompanyATS",
+                description=_strip_html(j.get("descriptionPlain")
+                                        or j.get("description") or ""),
+                posted_date=posted,
+                job_id=str(j.get("id") or ""),
+            ))
+        return out
+
+    def _fetch_ashby(self, token: str, company: str) -> list[Job]:
+        resp = self._get(self.ASHBY.format(token=token))
+        if not resp:
+            return []
+        data = resp.json()
+        out = []
+        for j in data.get("jobs", []) if isinstance(data, dict) else []:
+            out.append(Job(
+                title=(j.get("title") or "").strip(),
+                company=company,
+                location=(j.get("location") or "").strip(),
+                url=(j.get("jobUrl") or j.get("applyUrl") or "").strip(),
+                platform="CompanyATS",
+                description=_strip_html(j.get("descriptionPlain")
+                                        or j.get("description") or ""),
+                posted_date=(j.get("publishedAt") or "")[:10],
+                job_id=str(j.get("id") or ""),
+            ))
+        return out

src/scrapers/fetch.py

@@ -0,0 +1,308 @@
+"""Unified, stealth-capable fetch layer for all job scrapers.
+
+Goal: scrape without getting blocked. This module routes every HTTP fetch
+through Scrapling when it is installed (real-Chrome TLS fingerprint via
+`Fetcher`, Cloudflare-Turnstile bypass via `StealthyFetcher`), and transparently
+falls back to plain `requests` when Scrapling (or its browsers) are unavailable.
+
+Two kinds of blocking, and what this layer does about each:
+
+  1. Fingerprint blocking (TLS / headers / headless detection / Cloudflare)
+     -> Scrapling's `impersonate=` + `StealthyFetcher` defeat this.
+  2. IP-reputation blocking (shared datacenter IP, e.g. on HF Spaces)
+     -> Only a proxy fixes this. Set the `SCRAPER_PROXIES` env var (comma- or
+        newline-separated proxy URLs) and they are rotated round-robin across
+        requests. Without proxies we still try our best with fingerprint stealth.
+
+The public surface is intentionally tiny and `requests.Response`-compatible
+(`.text`, `.content`, `.json()`, `.status_code`, `.ok`) so existing scrapers do
+not need to change how they read responses.
+"""
+from __future__ import annotations
+
+import os
+import time
+import json as _json
+import logging
+import itertools
+import threading
+from typing import Optional
+
+log = logging.getLogger("scraper.fetch")
+
+# ── Optional Scrapling import (degrade gracefully if missing) ─────────────────
+try:  # pragma: no cover - import side effect depends on the environment
+    from scrapling.fetchers import Fetcher as _SFetcher  # HTTP, TLS-impersonate
+    _HAS_SCRAPLING = True
+except Exception:  # noqa: BLE001 - any import error => fall back to requests
+    _SFetcher = None
+    _HAS_SCRAPLING = False
+
+try:  # StealthyFetcher needs the extra Camoufox browser (`scrapling install`)
+    from scrapling.fetchers import StealthyFetcher as _SStealthy
+    _HAS_STEALTHY = True
+except Exception:  # noqa: BLE001
+    _SStealthy = None
+    _HAS_STEALTHY = False
+
+# `requests` is always available (a hard dependency of the project).
+import requests  # noqa: E402
+from fake_useragent import UserAgent  # noqa: E402
+
+_DEFAULT_IMPERSONATE = os.getenv("SCRAPER_IMPERSONATE", "chrome")
+_DEFAULT_TIMEOUT = int(os.getenv("SCRAPER_TIMEOUT", "20"))
+
+
+def scrapling_available() -> bool:
+    return _HAS_SCRAPLING
+
+
+def stealthy_available() -> bool:
+    return _HAS_STEALTHY
+
+
+# ── Proxy rotation (the only real fix for datacenter-IP blocking) ─────────────
+def _load_proxies() -> list[str]:
+    """Read proxies from env at call time so secrets can be set after import.
+
+    Accepts comma- OR newline-separated proxy URLs in `SCRAPER_PROXIES`
+    (e.g. ``http://user:pass@host:port``). A single `SCRAPER_PROXY` also works.
+    """
+    raw = os.getenv("SCRAPER_PROXIES", "") or os.getenv("SCRAPER_PROXY", "")
+    if not raw:
+        return []
+    parts = [p.strip() for chunk in raw.split("\n") for p in chunk.split(",")]
+    return [p for p in parts if p]
+
+
+_proxy_lock = threading.Lock()
+_proxy_cycle: Optional["itertools.cycle"] = None
+_proxy_snapshot: tuple[str, ...] = ()
+
+
+def _next_proxy() -> Optional[str]:
+    """Round-robin the configured proxies; rebuild the cycle if env changed."""
+    global _proxy_cycle, _proxy_snapshot
+    proxies = tuple(_load_proxies())
+    if not proxies:
+        return None
+    with _proxy_lock:
+        if proxies != _proxy_snapshot or _proxy_cycle is None:
+            _proxy_snapshot = proxies
+            _proxy_cycle = itertools.cycle(proxies)
+        return next(_proxy_cycle)
+
+
+# ── requests.Response-compatible shim ─────────────────────────────────────────
+class FetchResponse:
+    """Minimal response wrapper exposing the bits scrapers actually use."""
+
+    __slots__ = ("status_code", "text", "url", "backend")
+
+    def __init__(self, status_code: int, text: str, url: str = "",
+                 backend: str = "requests"):
+        self.status_code = int(status_code or 0)
+        self.text = text or ""
+        self.url = url
+        self.backend = backend
+
+    @property
+    def ok(self) -> bool:
+        return 200 <= self.status_code < 300
+
+    @property
+    def content(self) -> bytes:
+        return self.text.encode("utf-8", "replace")
+
+    def json(self):
+        return _json.loads(self.text)
+
+
+_ua = UserAgent()
+
+
+def _scrapling_response_text(page) -> str:
+    """Extract the response body text across Scrapling versions, defensively."""
+    for attr in ("body", "text", "html_content", "html"):
+        try:
+            val = getattr(page, attr, None)
+        except Exception:  # noqa: BLE001
+            val = None
+        if isinstance(val, bytes):
+            return val.decode("utf-8", "replace")
+        if isinstance(val, str) and val:
+            return val
+    try:
+        return str(page)
+    except Exception:  # noqa: BLE001
+        return ""
+
+
+def _scrapling_status(page) -> int:
+    for attr in ("status", "status_code"):
+        val = getattr(page, attr, None)
+        if isinstance(val, int):
+            return val
+    return 200
+
+
+def _build_url(url: str, params: Optional[dict]) -> str:
+    if not params:
+        return url
+    from urllib.parse import urlencode
+    sep = "&" if "?" in url else "?"
+    return f"{url}{sep}{urlencode(params)}"
+
+
+# ── Public API ────────────────────────────────────────────────────────────────
+def get(
+    url: str,
+    *,
+    params: Optional[dict] = None,
+    headers: Optional[dict] = None,
+    timeout: int = _DEFAULT_TIMEOUT,
+    retries: int = 3,
+    impersonate: str = _DEFAULT_IMPERSONATE,
+    use_browser: bool = False,
+    solve_cloudflare: bool = False,
+) -> Optional[FetchResponse]:
+    """Fetch a URL, preferring Scrapling stealth, falling back to requests.
+
+    `use_browser`/`solve_cloudflare` route through `StealthyFetcher` (Camoufox)
+    for JS-heavy or Cloudflare-protected pages when it is installed.
+    Returns a `FetchResponse` on HTTP 200, else None (mirrors the old `_get`).
+    """
+    full_url = _build_url(url, params)
+
+    if use_browser or solve_cloudflare:
+        r = _get_browser(full_url, timeout=timeout, solve_cloudflare=solve_cloudflare)
+        if r is not None:
+            return r
+        # fall through to HTTP if the browser path is unavailable/failed
+
+    if _HAS_SCRAPLING:
+        r = _get_scrapling_http(full_url, headers, timeout, retries, impersonate)
+        if r is not None:
+            return r
+        # fall through to requests on total failure
+
+    return _get_requests(full_url, headers, timeout, retries)
+
+
+def _get_scrapling_http(url, headers, timeout, retries, impersonate):
+    for attempt in range(retries):
+        proxy = _next_proxy()
+        try:
+            kwargs = {
+                "timeout": timeout,
+                "stealthy_headers": True,
+                "impersonate": impersonate,
+            }
+            if headers:
+                kwargs["headers"] = headers
+            if proxy:
+                kwargs["proxy"] = proxy
+            page = _SFetcher.get(url, **kwargs)
+            status = _scrapling_status(page)
+            if status == 200:
+                return FetchResponse(status, _scrapling_response_text(page),
+                                     url=url, backend="scrapling-http")
+            if status == 429:
+                time.sleep((attempt + 1) * 8)
+                continue
+            if status in (401, 403):
+                # try once more via a different proxy; else give up on this backend
+                if proxy and attempt < retries - 1:
+                    continue
+                return None
+        except TypeError:
+            # Older/newer Scrapling signature mismatch — drop unsupported kwargs.
+            try:
+                page = _SFetcher.get(url, timeout=timeout)
+                if _scrapling_status(page) == 200:
+                    return FetchResponse(200, _scrapling_response_text(page),
+                                         url=url, backend="scrapling-http")
+            except Exception:  # noqa: BLE001
+                pass
+            return None
+        except Exception as exc:  # noqa: BLE001
+            log.debug("scrapling http error (%s): %s", url, exc)
+            time.sleep(2 ** attempt)
+    return None
+
+
+def _get_browser(url, timeout, solve_cloudflare):
+    if not _HAS_STEALTHY:
+        return None
+    proxy = _next_proxy()
+    try:
+        kwargs = {
+            "headless": True,
+            "network_idle": True,
+            "timeout": timeout * 1000,  # Scrapling browser timeouts are in ms
+        }
+        if solve_cloudflare:
+            kwargs["solve_cloudflare"] = True
+        if proxy:
+            kwargs["proxy"] = proxy
+        page = _SStealthy.fetch(url, **kwargs)
+        status = _scrapling_status(page)
+        if status and status != 200:
+            return None
+        return FetchResponse(status or 200, _scrapling_response_text(page),
+                             url=url, backend="scrapling-stealth")
+    except TypeError:
+        try:
+            page = _SStealthy.fetch(url, headless=True)
+            return FetchResponse(_scrapling_status(page),
+                                 _scrapling_response_text(page),
+                                 url=url, backend="scrapling-stealth")
+        except Exception:  # noqa: BLE001
+            return None
+    except Exception as exc:  # noqa: BLE001
+        log.debug("scrapling stealth error (%s): %s", url, exc)
+        return None
+
+
+def _get_requests(url, headers, timeout, retries):
+    base_headers = {
+        "User-Agent": _ua.random,
+        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
+                  "image/webp,*/*;q=0.8",
+        "Accept-Language": "en-US,en;q=0.9",
+        "Accept-Encoding": "gzip, deflate, br",
+        "Connection": "keep-alive",
+        "Upgrade-Insecure-Requests": "1",
+    }
+    if headers:
+        base_headers.update(headers)
+
+    for attempt in range(retries):
+        proxy = _next_proxy()
+        proxies = {"http": proxy, "https": proxy} if proxy else None
+        try:
+            base_headers["User-Agent"] = _ua.random
+            resp = requests.get(url, headers=base_headers, timeout=timeout,
+                                proxies=proxies)
+            if resp.status_code == 200:
+                return FetchResponse(200, resp.text, url=resp.url,
+                                     backend="requests")
+            if resp.status_code == 429:
+                time.sleep((attempt + 1) * 10)
+            elif resp.status_code in (401, 403):
+                break
+        except Exception:  # noqa: BLE001
+            time.sleep(2 ** attempt)
+    return None
+
+
+def fetch_browser_html(url: str, *, solve_cloudflare: bool = True,
+                       timeout: int = 30) -> Optional[str]:
+    """Return raw HTML for a JS/Cloudflare page via StealthyFetcher, or None.
+
+    Convenience wrapper for scrapers that already parse HTML with BeautifulSoup
+    (e.g. Glassdoor) and want a Camoufox stealth fetch before falling back to
+    their own Playwright path.
+    """
+    r = _get_browser(url, timeout=timeout, solve_cloudflare=solve_cloudflare)
+    return r.text if (r and r.text) else None

src/scrapers/glassdoor.py

@@ -2,6 +2,7 @@ import re
 from bs4 import BeautifulSoup
 from playwright.sync_api import sync_playwright, TimeoutError as PWTimeout
 from .base import BaseScraper, Job
+from . import fetch
 
 
 class GlassdoorScraper(BaseScraper):
@@ -17,6 +18,18 @@ class GlassdoorScraper(BaseScraper):
         end = 6 + len(role)
         url = f"https://www.glassdoor.co.in/Job/india-{role_slug}-jobs-SRCH_IN115_KO6,{end}.htm"
 
+        # Preferred path: Scrapling StealthyFetcher (Camoufox) solves Cloudflare
+        # Turnstile out of the box. Falls back to the Playwright path below if
+        # Scrapling/Camoufox isn't installed or returns nothing.
+        try:
+            html = fetch.fetch_browser_html(url, solve_cloudflare=True, timeout=30)
+            if html:
+                jobs = self._parse_jobs(BeautifulSoup(html, "lxml"), location, max_results)
+                if jobs:
+                    return jobs[:max_results]
+        except Exception:
+            pass
+
         try:
             with sync_playwright() as pw:
                 browser = pw.chromium.launch(

ui.py

@@ -1797,7 +1797,8 @@ if show_config and start and not st.session_state.running:
                 skipped_dup: int  = 0
 
                 _all_plats   = set(_platforms.get("all_platforms", []))
-                _legacy_keys = {"linkedin", "indeed", "glassdoor", "remotive", "weworkremotely", "naukri"}
+                _legacy_keys = {"linkedin", "indeed", "glassdoor", "remotive",
+                                "weworkremotely", "naukri", "company_ats"}
                 scraper_map  = {}
 
                 if "linkedin" in _all_plats:
@@ -1836,6 +1837,15 @@ if show_config and start and not st.session_state.running:
                 else:
                     _step_skip("naukri")
 
+                # Direct-company ATS boards (Greenhouse/Lever/Ashby). Public JSON
+                # APIs that rarely IP-block — the most reliable bulk source on HF.
+                if "company_ats" in _all_plats:
+                    from src.scrapers.company_ats import CompanyATSScraper
+                    scraper_map["company_ats"] = ("CompanyATS", CompanyATSScraper())
+                    _q_log("🏢 Direct Company ATS enabled (Greenhouse/Lever/Ashby — no IP blocks)")
+                else:
+                    _step_skip("company_ats")
+
                 _ej_platforms = [p for p in _all_plats if p not in _legacy_keys]
                 if _ej_platforms:
                     # The 160+ "ever-jobs" platforms need the NestJS sidecar on