Hugging Face's logo

Spaces:
saitejatirunagari
/
JAA-ATS-Tool
Sleeping

App Files Community
JAA-ATS-Tool / AGENT_CONTEXT.md
saitejatirunagari's picture
saitejatirunagari
feat: Scrapling anti-block fetch layer + Direct Company ATS source
9e9393d
preview code
|
Raw
History Blame
31 kB

Agent Context — Job Automation Agent

Read this first. Everything a new agent (Claude/Cursor) or developer needs: the architecture, the hard-won design decisions, what works, what's open, and where to start. If this contradicts the code, the code wins — but update this doc when you finish.

Last updated: 2026-06-23 (Phase 8) Active branch: main (single branch, no PRs) Remotes: origin (GitHub) + hf (Hugging Face Space) Recent work: Phase 8 (non-destructive tailoring + reliable PDF + persistent side panel + history, extension v1.5.0) · 7759bfb Phase 7 (LaTeX + placement + resilient Run)

1. Mission

Help a Product Manager job-seeker get ATS-optimized, honestly-scored resumes, two ways from ONE shared Python pipeline:

  1. Streamlit web app (batch): upload resume → scrape PM jobs from many platforms → assess → tailor a per-job resume → score → log to Google Sheets.
  2. Chrome extension (per-job, in-browser): open ANY job listing → click Run → the extension grabs the JD + sends it with the stored resume to the same pipeline exposed as a JSON API → get a tailored DOCX/PDF download. Built to avoid IP-blocking from bulk scraping.

The HF Space is the single source of truth: one codebase serves the Streamlit app AND the extension API, so the score is identical everywhere (proven by a parity test).

Live: https://huggingface.co/spaces/saitejatirunagari/JAA-ATS-Tool API base: https://saitejatirunagari-jaa-ats-tool.hf.space Endpoints: /api/health · /api/generate · /api/repair-with-feedback Repo: https://github.com/saitejatiru/JAA-ATS-Tool

Stack: Python 3.11 + Streamlit + FastAPI + Chrome MV3 extension + multi-LLM provider chain (Kimi 2.6 / NVIDIA-hosted, OpenAI-compatible) + python-docx + reportlab + NestJS ever-jobs sidecar (optional) on HF Spaces (Docker).

2. THE most important thing: the honesty / anti-circular scoring model

This project spent days on one trap and the current design exists to prevent it.

The trap: the original scorer extracted keywords from our own narrow taxonomy, injected exactly those, and scored against the same list → always ~95% internally, but Jobalytics/Simplify reported ~55-73% on the same resume. The score was self-graded and fake.

The current model (do not regress this):

  • Two scores, not one:
    • JD Match (internal) — weighted coverage of the JD's requirements (ats_scoring_v2.score_jd_match).
    • Independent validation — a separate, stricter scorer (ats_validator.validate_resume) that shares NO logic with the generator: exact/alias presence only (no "plausible" credit), evidence-weighted (a term in the Skills list but NOT in an experience bullet earns half credit), and strict seniority measured from the BASE resume's years (so an injected "12+ years" can't game it). It scores the re-parsed exported file.
  • A resume is only READY (downloadable as good) if BOTH internal AND independent ≥ 90 AND ATS readability ≥ 90 AND parse-validation passed.
  • Auto-repair loop weaves missing/skills-only terms into bullets (real evidence) until both scores clear 90 — it cannot pass by stuffing the Skills list, because the independent validator only fully credits evidenced terms.
  • Per-job scores legitimately vary. A great-fit PM JD → 90s. A JD full of domain terms the candidate can't evidence (security tooling, AR/fintech, backend eng) → honestly lower. This is correct, not a regression.

Anti-cheat is enforced by scripts/verify_anticheat.py (must stay green): PM JD → READY both≥90; cybersec PM → security terms flagged HIGH, not auto-claimed; backend-eng JD → not CLEAN (independent < 90); 12-yr JD → seniority fails, download blocked; score comes from parsed export; truncating the file drops the score.

Never loosen the independent validator to make numbers look better, and never inject fake employers/degrees/certs/dates/seniority.

3. Candidate Fit Expansion + AUTO_AGGRESSIVE (how keywords get included)

The resume is treated as a base profile, not the full truth. We include JD terms aggressively where plausible, and gate the genuinely risky ones.

src/candidate_fit.py classifies every JD requirement → a risk severity:

Severity What Action
LOW_RISK_AUTO_INCLUDED common PM/product/analytics/agile craft, normal tools/methods/responsibilities, real soft skills, anything already in the resume auto-include
MEDIUM_RISK_REVIEW_RECOMMENDED domain/industry terms & plausible tools not in the base resume auto-include + flag review
HIGH_RISK_NEEDS_CONFIRMATION specialized platforms (SIEM/SOAR), compliance/regulatory, hands-on engineering hard skills, seniority-sensitive NOT auto-included; if a job needs them to hit 90 → NEEDS_USER_INPUT
BLOCKED_DO_NOT_INCLUDE degrees/certs/licenses not held, fakes, seniority jumps, deep-tech specialties excluded

config.AUTOMATION holds automation_mode="auto_aggressive", review_policy, download_policy. Generation auto-includes LOW+MEDIUM (weave/skills/repair all gated to exclude HIGH+BLOCKED).

Maximum ATS Mode (User-Confirmed Skill Expansion): classify_fit(..., maximum_ats_mode=True) promotes the curated config.MAXIMUM_ATS_SAFE_TERMS (normal PM/Product/AI/SaaS/B2B/agile vocabulary) to user-confirmed (LOW) — AFTER the hard-block checks, so credentials/seniority/engineering/specialized still block or ask first. Flows via the job dict (_maximum_ats_mode, _confirmed_terms) into _generate_resume_v4, and via params through jobalytics_repair. Targets external 95 / floor 90 (config.MAXIMUM_ATS); the repair loop iterates while LOW/MEDIUM gaps remain. Confirmed terms persist to the vault (confirm_expansion_terms). The boundaries in §2 are NEVER relaxed by it.

⚠ The internal score is NOT a valid external signal. Internal taxonomy coverage can be 96% while Jobalytics is 54% (it extracts a broader 40-46 term set). So Maximum ATS Mode is driven by EXTERNAL coverage: src/external_ats.py builds a broad Jobalytics-style expected set; resume_customizer. _maximize_external_coverage() honesty-gates each term then physically guarantees the includable ones into the exported DOCX (Skills verbatim + woven bullets) and measures coverage from the re-parsed export. Status becomes READY_MAX_ATS_95_PLUS / READY_90_PLUS_EXTERNAL_ALIGNED / BELOW_TARGET_REPAIRABLE. _build_skill_pool no longer caps off includable terms in max mode (cap → 200). Regression: scripts/verify_max_ats_coverage.py (the internal-96/external-54 bug).

LaTeX resume path (src/latex_resume.py): the candidate can supply their resume as LaTeX (clean PDF design they control). When resume_latex is present it is PRIORITISED over the uploaded PDF for both keyword matching and output. latex_to_text() (pylatexenc, regex fallback) feeds the same external_ats + candidate_fit honesty gate as the DOCX path, PLUS _specialty_hit() — a phrase-level guard so multi-word JD grams (cuda kernel programming) can't smuggle a blocked engineering/credential token past the exact-match classifier. inject_keywords() splices a compile-safe \textbf{Core Competencies:} block before \end{document} (idempotent, escapes specials); compile_latex_to_pdf() uses Tectonic (preferred)/latexmk/pdflatex with shell-escape disabled. With no engine installed it still returns the injected .tex + the (text-based) coverage report. API: /api/generate + /api/repair-with-feedback accept resume_latex and branch to _generate_from_latex / _repair_from_latex (return pdf_b64 + tex_b64 + coverage report; status by external coverage). Extension v1.3.0 adds a Resume LaTeX textarea (priority over PDF) and Download PDF / Download .tex. Regression: scripts/verify_latex_resume.py. Tectonic is installed in the Dockerfile; pylatexenc is in requirements.txt.

Recruiter-grade placement + clean inputs + resilient Run (Phase 7): the live output was a wall of repeated "Core Competencies:" lines full of LinkedIn/Simplify UI junk. Fixed across four axes (all honesty-preserving):

  • Clean JD (R13): extension/content.js::extractGenericJD strips a CLONE (stripChrome: nav/aside/header/footer/button/overlay + simplify/jobalytics/ jobscan/teal/__extension nodes) and scrubOverlayLines drops CTA lines; external_ats._is_ui_noise() (wired into _is_term_like) rejects UI/CTA/ marketing n-grams so page chrome never becomes a keyword.
  • DOCX distribution (R14): resume_renderer._write_skills emits ONE capped line per category (_PER_CATEGORY_CAP=10, _SKILLS_TOTAL_CAP=28; no more PER_LINE=12 repeated headers); _build_skill_pool/repair cap is _SKILLS_DISPLAY_CAP=26 (was 200); surplus includable terms are redirected into the Summary (_append_summary_terms) + Experience (_force_weave_into_bullets), not extra skills lines.
  • LaTeX distribution (R14): latex_resume.inject_keywords distributes terms across a Summary sentence + woven Experience \items + ONE competencies line; _remove_injected_block strips every fragment (idempotent).
  • Resilient Run (R15): extension/background.js owns generation and persists running→done/error to ats_results keyed by normalized URL; the popup restores spinner/result on open + live-refreshes via chrome.storage.onChanged. Closing the popup no longer loses the run (extension v1.4.0). Regressions: verify_clean_jd_extraction.py, verify_skills_distribution.py, verify_resilient_run.py (+ existing verify_latex_resume.py / verify_max_ats_coverage.py still pass).

Trust + usability (Phase 8, extension v1.5.0): four live-use fixes, all honesty-preserving:

  • Reliable PDF (R16): the HF/Linux DOCX flow never wrote a .pdf sidecar so only DOCX downloaded. api_server now calls pdf_writer.docx_to_pdf on both the generate and repair DOCX paths (reportlab off-Windows) → pdf_b64 always populated. The LaTeX path returns a pdf_error (engine_missing vs latex_error + compile_log) and STILL returns tex_b64 on compile failure (200, not 500); Dockerfile has a hard RUN tectonic --version gate.
  • Non-destructive tailoring is the DEFAULT (R17): NON_DESTRUCTIVE_DEFAULT=True
    • _apply_non_destructive rebuild tailored.roles VERBATIM from the base resume (titles/companies/dates/existing bullets byte-identical) and add keywords ONLY via Summary augmentation + _append_keyword_bullets (≤3 gated lines appended per role). ALL four bullet-mutation sites — incl. _maximize_external_coverage (LIVE in max-ATS mode) — are gated behind non_destructive, so the candidate's real history is never altered. LaTeX inject_keywords APPENDS ≤3 \items (tagged % ats-item) after each experience instead of editing existing items. Proven by verify_non_destructive.py (end-to-end _generate_resume_v4 + _maximum_ats_mode diff against the base). Coverage relocates to appended bullets+Skills+Summary (still ~96%); the max-ATS regression threshold is an honest present-in-export floor.
  • Persistent side panel (R18): content.js::injectSidePanel docks a persistent, collapsible/dismissible left iframe hosting popup/popup.html (exposed via web_accessible_resources), idempotent + non-blocking; the toolbar popup is a thin launcher (TOGGLE_PANEL). The disappearing-popup "run restarted" feeling is gone; the panel live-reflects the background-owned run.
  • History (R19): renderHistory lists prior generations per job (title/company/ time/internal+external scores/status) with restore (via applyResult) + re-download; background.js MAX_SAVED 10→40 with a BYTE_BUDGET quota guard (approxSize/stripOldestArtifacts) that degrades oldest entries to metadata-only, always keeping the newest fully downloadable. Regressions: verify_pdf_download.py, verify_non_destructive.py, verify_side_panel.py, verify_history.py (+ all Phase 7 scripts still pass).

Status system (src/fit_gate.py): READY_90_PLUS / READY_90_PLUS_REVIEW_RECOMMENDED / READY_95_EXTERNAL_ALIGNED (feedback repair only: internal AND independent ≥ 90, pasted gaps mostly resolved, no review flags — fabrication cannot reach it) / NEEDS_REPAIR / NEEDS_USER_INPUT / NOT_ELIGIBLE_LOW_FIT / PARSE_FAILED. Maximum ATS statuses: READY_MAX_ATS_95_PLUS / READY_90_PLUS_EXTERNAL_ALIGNED / NEEDS_USER_CONFIRMATION / BELOW_TARGET_REPAIRABLE. Quality flags: CLEAN_90_PLUS / AGGRESSIVE_90_PLUS / REVIEW_REQUIRED_90_PLUS / WEAK_90_INTERNAL_ONLY (internal ≥90 but independent <90).

Candidate Experience Vault (src/candidate_vault.py, data/candidate_vault.json, gitignored/per-user): accumulates terms across jobs. Only user_confirmed + resume_original are treated as fully safe; inferred/jd_expansion stay reviewable — the system never silently promotes a risky term. confirm_terms() lets the user lock a term safe or blocked; the fit classifier consults it first.

4. ⚠ Reversals from older versions of this doc (history that changed)

  • Skills section: RE-ADDED. Older docs said "NO Core Competencies/Skills section." That was wrong — research showed a dedicated, standard-headed Skills section is the #1 ATS keyword vehicle (Jobalytics/Resume Worded parse it). Resume.skills + resume_renderer._write_skills render a categorized SKILLS section (Tools & Analytics / Methodologies / Domains / Core Competencies), placed after KEY ACHIEVEMENTS, ≤12 items per line (multi-line if more).
  • Extraction is SKILLS-ONLY + LLM-unioned. extract_jd_keywords keeps only recognized skills (no "any noun"); the v4 LLM also returns a jd_skills array (Jobalytics-style) that is unioned in. No prose-noun garbage.
  • Provider-independent. Generation no longer assumes one model — see §6.
  • The _assert_no_dump_footer postcondition still bans raw keyword-dump footers, but the categorized Skills section is allowed.

5. End-to-end flows

Web app (batch)

ui.py wizard → scrape (src/scrapers/* + ever-jobs) → geo_filter (location) → dedup → job_assessor → ResumeCustomizer.customize_for_jobs → _generate_resume_v4 (provider chain + repair + scoring + status) → render DOCX/PDF → Google Sheets / Excel / CSV / run history (hf_storage syncs to HF Dataset on save). Batch table shows status, internal+independent scores, risk, download gating. UI startup calls hf_storage.sync_down() once per session.

Extension (per-job)

extension/ (MV3) → content.js extracts JD (generic + LinkedIn/Naukri/Indeed) → popup Run → background.js POSTs multipart to POST /api/generate → api_server.generate_resume_for_api → ResumeCustomizer._run_provider_chain(..., base_resume_override=...) (same path) → returns base64 DOCX/PDF + scores/status → popup shows scores + Download.

Improve with ATS feedback (External Feedback Repair Mode): after a resume exists, user pastes Jobalytics/Simplify feedback → popup Improve → background.js REPAIR → POST /api/repair-with-feedback → jobalytics_repair.repair_with_external_feedback (every missing keyword risk-classified; BLOCKED + unconfirmed HIGH never added; re-render → re-parse → re-score) → updated scores/status + added / unresolved-high / blocked terms + base64 DOCX/PDF. No ATS logic in extension JS.

Result persistence: generated/repaired results are cached in chrome.storage.local (ats_results, keyed by tab URL). Reopening the popup on the same job link restores scores, Download, and the Improve section (pruned to 10 most-recent jobs). Reload the unpacked extension after pulling extension/ changes from GitHub.

External ATS Feedback Repair (backend detail)

src/jobalytics_repair.py:

  • parse_external_feedback(text) — extracts external score + missing/matched keywords from a freeform paste; drops prose tokens when there's no real list.
  • repair_with_external_feedback(...) — runs regenerate_from_jobalytics (full v4 pipeline in jobalytics_repair mode with pasted addable keywords merged into the include pool), then derives added / review-flag / unresolved-high / blocked lists and final status (may upgrade to READY_95_EXTERNAL_ALIGNED). Verified by scripts/verify_feedback_repair.py (CISSP blocked, SIEM unresolved-high).

6. LLM provider abstraction (model-independent)

src/providers.py: LLMProvider base, OpenAICompatProvider (Kimi/NVIDIA — all OpenAI-compatible cfgs), StubProvider (deterministic, offline floor + tests). build_provider_chain() reads config.LLM_GENERATION.provider_order.

  • Every model response is schema-validated (validate_response against JD_ANALYSIS_SCHEMA / RESUME_TAILORING_SCHEMA); quality ∈ ok | failed_schema | provider_error. Schema-failed output can never become READY — it forces fallback to the next provider / deterministic path.
  • src/provider_prompts.py loads provider-specific templates from prompts/ ({task}_{claude|kimi|nvidia}.txt). LLMClient.tailor_resume_v4(..., provider_family=...) selects the prompt. Any stub/test that monkeypatches tailor_resume_v4 must accept **kwargs (it takes provider_family).
  • Pipeline records provider_used + provider_response_quality on the report.

7. File map (current)

Entry points

File Purpose
api_server.py HF Space entrypoint (start.sh runs it). FastAPI on :7860: /api/health, /api/generate, /api/repair-with-feedback; launches Streamlit subprocess on :8501 and reverse-proxies everything else (HTTP via httpx + WS for /_stcore/stream). Imports src/ — no ATS logic of its own.
ui.py Streamlit web app (~2400 lines): wizard, results, batch table, vault controls.
start.sh ever-jobs sidecar (optional) → exec python api_server.py.
Dockerfile HF build.

ATS pipeline v2 (src/)

File Purpose
jd_analyzer.py analyze_jd → structured JDRequirements (titles, required/preferred hard skills, tools, responsibilities, domains, certs, education, soft, seniority) each with importance/source_phrase/aliases/placement. Deterministic floor + optional LLM enrichment.
candidate_fit.py Fit/severity classification (§3): classify_all_fit, severity, auto_terms, high_risk_terms.
fit_gate.py assess_job_fit_for_90 + status constants.
ats_scoring_v2.py score_jd_match (weighted: 35 must-have/20 responsibilities/15 tools/10 title+domain/10 seniority/5 certs/5 soft + penalties) + score_ats_readability.
ats_validator.py Independent evidence-weighted validator (anti-circular).
ats_report.py build_ats_report (explanation: scores, strong/weak, gaps, formatting, recommendations) + reconcile_missing_keywords.
candidate_vault.py Persistent learned-skills vault.
jobalytics_repair.py classify_jobalytics_keywords, regenerate_from_jobalytics, parse_external_feedback, repair_with_external_feedback — honest external-checker feedback repair (§5).
hf_storage.py Best-effort HF Dataset sync for Streamlit run history/resumes/reports (private dataset; sync_down on UI startup, push_run after each run).
ats_scorer.py Base scorer/helpers still used: extract_jd_keywords (skills-only), _kw_in_text, _phrase_in_text, _strip_keyword_spam (drops 15+-separator dump lines), conservative_display_score, PM_SKILL_TAXONOMY.
resume_model.py Resume/Role/Education/Contact dataclasses (+ skills).
resume_parser_v2.py parse_resume_pdf / parse_resume_pdf_cached (PDF → Resume).
resume_renderer.py Canonical DOCX renderer incl. the SKILLS section.
resume_customizer.py Heart. _generate_resume_v4 (parse → provider tailor → backfill → weave → skills → render → parse-validate → score internal+independent → auto-repair ≤3 → status/quality/download gating), _run_provider_chain, _validate_parsed_resume, customize_for_jobs.
llm_client.py tailor_resume_v4, analyze_jd_requirements, judge_evidence, repair_resume_v4, jobalytics_repair_v4 (all OpenAI-compatible, cfg-driven).
providers.py / provider_prompts.py Provider abstraction (§6).
geo_filter.py Drops jobs outside selected locations (respects Remote/Worldwide).
pdf_writer.py DOCX→PDF (reportlab on Linux/HF).
job_assessor.py, gsheets.py, excel_reporter.py, run_history.py, job_history.py Assessment, Sheets, Excel, history, dedup.

Extension (extension/)

File Purpose
manifest.json MV3; permissions: storage, downloads, activeTab, scripting; <all_urls>.
content.js JD extraction: generic + LinkedIn/Naukri/Indeed + document.title fallback; fast (single short MutationObserver wait).
popup/popup.html + popup.js Run (enabled immediately), status, scores, Download (always shown, labeled by score), Improve with ATS feedback panel, result restore from chrome.storage.local, ⚙ Options. Download in popup (Blob + <a download>).
background.js Thin service worker: GENERATE → /api/generate, REPAIR → /api/repair-with-feedback. Reads resume/API settings from chrome.storage.local. NO ATS/LLM logic.
options/ Upload resume once (→ base64 in chrome.storage.local) + API URL + token.

Config

config.py: ASSESSMENT_MODELS (with tailor flags), LLM_GENERATION (provider_order, fallback flags), AUTOMATION (review/download policy). Secrets in .env / HF Space secrets: LLM keys, API_SECRET_TOKEN, HF_DATASET_REPO (for hf_storage), EVER_JOBS_API_URL, Google. Never committed. Also: CONTACT_LOCATION (default Hyderabad, Telangana, India · Open to relocate) for resume header address/location ATS compliance.

Tests / scripts

File Purpose
scripts/verify_anticheat.py Anti-circular suite — keep green.
scripts/verify_feedback_repair.py External feedback repair: parse + honest weave; CISSP blocked, SIEM unresolved-high.
scripts/verify_maximum_ats.py Maximum ATS Mode: PM/AI→user-confirmed; certs/seniority/eng still gated; coverage improves; 65%→repairable.
scripts/verify_max_ats_coverage.py Internal-96/external-54 regression: exported DOCX covers ≥90% includable PM terms; creds/seniority excluded; status by external coverage.
scripts/verify_latex_resume.py LaTeX path: text extraction, honest keyword injection (22%→88% coverage), DISTRIBUTED placement (summary + experience + 1 competencies line), no fabrication (cissp/pmp/12+ years/cuda blocked), idempotent injection.
scripts/verify_clean_jd_extraction.py R13: page/extension UI chrome (Simplify/Jobalytics CTA text) never becomes a keyword; genuine craft terms survive; content.js strips chrome.
scripts/verify_skills_distribution.py R14: ≤1 line per skills category (no repeated "Core Competencies:" dump), 18-28 total de-duplicated items.
scripts/verify_resilient_run.py R15: background owns generation (running→done/error keyed by URL); popup restores + live-refreshes; popup↔background storage contract matches.
scripts/verify_90_pipeline.py Full fit→repair→status pipeline (stub floor).
scripts/verify_scoring_v2.py jd_analyzer + evidence + scoring + report + feedback loop.
scripts/evaluate_model_providers.py Run same jobs through each provider (Kimi/NVIDIA) — ready rate, avg independent, schema errors. Needs real keys.
scripts/consistency_test.py Same JD ×3 per provider → variance.
scripts/run_20_job_validation.py 20-job end-to-end + validation packages. Needs real keys.
tests/test_api_parity.py Proves API path == web-app path (identical scores).
tests/test_api_health.py /api/health + token gate.
tests/fixtures/jds/*.txt Real JDs incl. backend_engineer, senior_pm_10yrs (anti-cheat).

8. Current status & known gotchas

  • Phase 6 (Chrome extension) is COMPLETE and live. End-to-end verified on a real Kimi/NVIDIA run: a good-fit job returned READY_90_PLUS, independent 98, and a downloadable DOCX. Parity test passes.
  • External ATS Feedback Repair Mode is wired end-to-end (f5dd879): backend /api/repair-with-feedback + extension Improve panel. Requires the Space to be rebuilt/running with that commit — if Improve fails with 404, the endpoint isn't deployed yet.
  • Extension result persistence is live (ef247ae): reopening the popup on the same job URL restores the last generated/repaired result. User must reload the unpacked extension after pulling from GitHub.
  • Per-job variance is expected (see §2): borderline jobs land NEEDS_REPAIR / ~73-85 on Jobalytics — that's honest, not a bug. Feedback repair pushes toward 90 only where missing keywords are plausible-and-weavable; HIGH/BLOCKED terms correctly stay out → NEEDS_USER_INPUT.
  • MV3 gotcha: service workers have no URL.createObjectURL → downloads are done in the popup, not the service worker.
  • Proxy gotcha: api_server strips content-encoding/content-length when proxying Streamlit (httpx already decompresses) — otherwise pages render blank.
  • Deploy gotcha: pushing to hf rebuilds the Space (~2-15 min for Docker) and causes transient 404/405s while the container is down. Prefer pushing extension-only changes to origin only (no Space rebuild needed) — but both remotes may end up at the same commit if you push everything to both.
  • HF Space is private: unauthenticated health checks from outside may return HF gateway 404 even when the Space is fine for token-authenticated extension calls.
  • Local import ui fails on the dev machine due to a pre-existing pandas/numpy binary mismatch — unrelated to our code; HF's clean deps are fine.
  • ever-jobs sidecar fails to start on HF unless EVER_JOBS_API_URL points to an externally hosted sidecar; known/non-blocking; dedicated scrapers still work.
  • Scrapling anti-block fetch layer (src/scrapers/fetch.py): all scraper HTTP goes through Scrapling (Chrome TLS impersonation via Fetcher; Cloudflare bypass via StealthyFetcher/Camoufox) with a requests fallback. Set SCRAPER_PROXIES (comma/newline list) to rotate proxies — the ONLY fix for datacenter-IP blocking (fingerprint stealth ≠ IP reputation). Camoufox is installed via scrapling install in the Dockerfile (non-fatal).
  • Direct Company ATS (src/scrapers/company_ats.py, platform key company_ats, default-on): Greenhouse/Lever/Ashby public JSON boards — the most block-resistant bulk source on HF. Extend via COMPANY_ATS_BOARDS env (JSON).
  • Streamlit batch persistence: hf_storage.py syncs run history/resumes/CSV to a private HF Dataset so data survives Space container restarts (needs HF_DATASET_REPO + write token in Space secrets).
  • Security: do not store HF write tokens in .git/config remote URLs — use huggingface-cli login or a credential helper instead.

Open / next

  • Run evaluate_model_providers.py + run_20_job_validation.py with real Kimi/NVIDIA keys (user's environment) — the live provider validation.
  • Integrate a no-block job source for HF batch scraping — DONE via Scrapling fetch layer + Direct Company ATS (Greenhouse/Lever/Ashby). Next: add a residential proxy (SCRAPER_PROXIES) to make LinkedIn/Indeed reliable on HF, and run a live bulk test to confirm block rates dropped.

9. Conventions (follow these)

  • Update HISTORY.md + README.md on every change (standing user rule).
  • Commit/push only when asked. Conventional Commits (fix(scope): …) + a Co-Authored-By trailer when AI-assisted. Single main branch, no rebases.
  • Push extension-only changes to origin only; push pipeline/API/HF changes to both origin and hf.
  • Secrets never committed (.env, google_*.json, resume PDFs, data/).
  • Honesty over false confidence. If a fix is partial, say so. If our score > what a real checker would give, say so. Never fake keywords or loosen the independent validator to inflate numbers.
  • Gold reference resume: C:\Users\Nxtwave\Desktop\resume\Saiteja_Tirunagari_Resume A 26 - Copy.pdf.
  • Test vaults are isolated in regression scripts (temp paths) — don't pollute the real vault.

10. Where to start

  • Low/odd ATS scores: read ats_scoring_v2.py + ats_validator.py (internal vs independent), then candidate_fit.py (what got included), then the _generate_resume_v4 repair loop. Run verify_anticheat.py + verify_90_pipeline.py.
  • Extension issues: extension/ (manifest → content.js → popup.js → background.js) + api_server.py. Remember MV3 download + proxy gotchas (§8). Improve broken? Check Space has /api/repair-with-feedback. Results lost on reopen? Check extension was reloaded after persistence commit + same job URL.
  • Feedback repair / 95% path: jobalytics_repair.py → api_server.py repair endpoint → extension REPAIR handler. Run verify_feedback_repair.py. Never loosen validator or auto-add HIGH/BLOCKED terms.
  • Provider/model work: providers.py + provider_prompts.py + prompts/ + llm_client.py. Keep schema validation; keep stubs **kwargs-tolerant.
  • New JD-keyword behavior: jd_analyzer.py (extraction) and candidate_fit.py (inclusion). Don't reintroduce "any noun" extraction or fake injection.

11. Quick commands 

# Regressions (deterministic, no keys)
PYTHONPATH=. python scripts/verify_anticheat.py
PYTHONPATH=. python scripts/verify_90_pipeline.py
PYTHONPATH=. python scripts/verify_scoring_v2.py
PYTHONPATH=. python scripts/verify_feedback_repair.py
PYTHONPATH=. python -m pytest tests/test_api_parity.py tests/test_api_health.py -q

# Provider eval (needs real Kimi/NVIDIA keys in .env)
PYTHONPATH=. python scripts/evaluate_model_providers.py
PYTHONPATH=. python scripts/run_20_job_validation.py

# Run the API locally (serves /api + proxies Streamlit)
python api_server.py            # http://localhost:7860

# Deploy
git push origin main            # always
git push hf main                # ONLY for pipeline/API/Space changes (rebuilds Space)

# Live checks (Space must be Running; private Space may 404 without auth)
# GET  …/api/health                  → {"status":"ok"}
# POST …/api/repair-with-feedback    → 405 on GET means route exists; POST needs token + multipart

End of context. Code is the source of truth — update this doc when you finish a task.