AGENT_CONTEXT.md

# Agent Context — Job Automation Agent

> **Read this first.** Everything a new agent (Claude/Cursor) or developer needs:
> the architecture, the hard-won design decisions, what works, what's open, and
> where to start. If this contradicts the code, **the code wins** — but update
> this doc when you finish.

Last updated: 2026-06-23 (Phase 8)
Active branch: `main` (single branch, no PRs)
Remotes: `origin` (GitHub) + `hf` (Hugging Face Space)
Recent work: Phase 8 (non-destructive tailoring + reliable PDF + persistent side
panel + history, extension v1.5.0) · `7759bfb` Phase 7 (LaTeX + placement + resilient Run)

---

## 1. Mission

Help a Product Manager job-seeker get **ATS-optimized, honestly-scored** resumes,
two ways from ONE shared Python pipeline:

1. **Streamlit web app (batch):** upload resume → scrape PM jobs from many
   platforms → assess → tailor a per-job resume → score → log to Google Sheets.
2. **Chrome extension (per-job, in-browser):** open ANY job listing → click
   **Run** → the extension grabs the JD + sends it with the stored resume to the
   **same pipeline exposed as a JSON API** → get a tailored DOCX/PDF download.
   Built to avoid IP-blocking from bulk scraping.

**The HF Space is the single source of truth:** one codebase serves the
Streamlit app AND the extension API, so the **score is identical everywhere**
(proven by a parity test).

Live: https://huggingface.co/spaces/saitejatirunagari/JAA-ATS-Tool
API base: `https://saitejatirunagari-jaa-ats-tool.hf.space`
Endpoints: `/api/health` · `/api/generate` · `/api/repair-with-feedback`
Repo: https://github.com/saitejatiru/JAA-ATS-Tool

Stack: Python 3.11 + Streamlit + FastAPI + Chrome MV3 extension + multi-LLM
provider chain (Kimi 2.6 / NVIDIA-hosted, OpenAI-compatible) + python-docx +
reportlab + NestJS `ever-jobs` sidecar (optional) on HF Spaces (Docker).

---

## 2. THE most important thing: the honesty / anti-circular scoring model

This project spent days on one trap and the current design exists to prevent it.

**The trap:** the original scorer extracted keywords from our own narrow
taxonomy, injected exactly those, and scored against the same list → always ~95%
internally, but Jobalytics/Simplify reported ~55-73% on the same resume. The
score was *self-graded* and fake.

**The current model (do not regress this):**

- **Two scores, not one:**
  - **JD Match (internal)** — weighted coverage of the JD's requirements
    (`ats_scoring_v2.score_jd_match`).
  - **Independent validation** — a *separate, stricter* scorer
    (`ats_validator.validate_resume`) that shares NO logic with the generator:
    exact/alias presence only (no "plausible" credit), **evidence-weighted** (a
    term in the Skills list but NOT in an experience bullet earns half credit),
    and **strict seniority measured from the BASE resume's years** (so an
    injected "12+ years" can't game it). It scores the **re-parsed exported file**.
- **A resume is only READY (downloadable as good) if BOTH internal AND
  independent ≥ 90 AND ATS readability ≥ 90 AND parse-validation passed.**
- **Auto-repair loop** weaves missing/skills-only terms into *bullets* (real
  evidence) until both scores clear 90 — it cannot pass by stuffing the Skills
  list, because the independent validator only fully credits evidenced terms.
- **Per-job scores legitimately vary.** A great-fit PM JD → 90s. A JD full of
  domain terms the candidate can't evidence (security tooling, AR/fintech,
  backend eng) → honestly lower. This is correct, not a regression.

**Anti-cheat is enforced by `scripts/verify_anticheat.py`** (must stay green):
PM JD → READY both≥90; cybersec PM → security terms flagged HIGH, not
auto-claimed; backend-eng JD → not CLEAN (independent < 90); 12-yr JD →
seniority fails, download blocked; score comes from parsed export; truncating
the file drops the score.

**Never** loosen the independent validator to make numbers look better, and
**never** inject fake employers/degrees/certs/dates/seniority.

---

## 3. Candidate Fit Expansion + AUTO_AGGRESSIVE (how keywords get included)

The resume is treated as a **base profile, not the full truth**. We include JD
terms aggressively *where plausible*, and gate the genuinely risky ones.

`src/candidate_fit.py` classifies every JD requirement → a risk **severity**:

| Severity | What | Action |
|---|---|---|
| `LOW_RISK_AUTO_INCLUDED` | common PM/product/analytics/agile craft, normal tools/methods/responsibilities, real soft skills, anything already in the resume | auto-include |
| `MEDIUM_RISK_REVIEW_RECOMMENDED` | domain/industry terms & plausible tools not in the base resume | auto-include + flag review |
| `HIGH_RISK_NEEDS_CONFIRMATION` | specialized platforms (SIEM/SOAR), compliance/regulatory, hands-on engineering hard skills, seniority-sensitive | NOT auto-included; if a job *needs* them to hit 90 → `NEEDS_USER_INPUT` |
| `BLOCKED_DO_NOT_INCLUDE` | degrees/certs/licenses not held, fakes, seniority jumps, deep-tech specialties | excluded |

`config.AUTOMATION` holds `automation_mode="auto_aggressive"`, `review_policy`,
`download_policy`. Generation auto-includes LOW+MEDIUM (weave/skills/repair all
gated to exclude HIGH+BLOCKED).

**Maximum ATS Mode (User-Confirmed Skill Expansion):** `classify_fit(...,
maximum_ats_mode=True)` promotes the curated `config.MAXIMUM_ATS_SAFE_TERMS`
(normal PM/Product/AI/SaaS/B2B/agile vocabulary) to user-confirmed (LOW) — AFTER
the hard-block checks, so credentials/seniority/engineering/specialized still
block or ask first. Flows via the job dict (`_maximum_ats_mode`,
`_confirmed_terms`) into `_generate_resume_v4`, and via params through
`jobalytics_repair`. Targets external 95 / floor 90 (`config.MAXIMUM_ATS`); the
repair loop iterates while LOW/MEDIUM gaps remain. Confirmed terms persist to the
vault (`confirm_expansion_terms`). The boundaries in §2 are NEVER relaxed by it.

**⚠ The internal score is NOT a valid external signal.** Internal taxonomy
coverage can be 96% while Jobalytics is 54% (it extracts a broader 40-46 term
set). So Maximum ATS Mode is driven by EXTERNAL coverage: `src/external_ats.py`
builds a broad Jobalytics-style expected set; `resume_customizer.
_maximize_external_coverage()` honesty-gates each term then **physically
guarantees** the includable ones into the exported DOCX (Skills verbatim + woven
bullets) and measures coverage from the **re-parsed export**. Status becomes
`READY_MAX_ATS_95_PLUS` / `READY_90_PLUS_EXTERNAL_ALIGNED` / `BELOW_TARGET_REPAIRABLE`.
`_build_skill_pool` no longer caps off includable terms in max mode (cap → 200).
Regression: `scripts/verify_max_ats_coverage.py` (the internal-96/external-54 bug).

**LaTeX resume path (`src/latex_resume.py`):** the candidate can supply their
resume as LaTeX (clean PDF design they control). When `resume_latex` is present it
is PRIORITISED over the uploaded PDF for both keyword matching and output.
`latex_to_text()` (pylatexenc, regex fallback) feeds the same `external_ats` +
`candidate_fit` honesty gate as the DOCX path, PLUS `_specialty_hit()` — a
phrase-level guard so multi-word JD grams (`cuda kernel programming`) can't
smuggle a blocked engineering/credential token past the exact-match classifier.
`inject_keywords()` splices a compile-safe `\textbf{Core Competencies:}` block
before `\end{document}` (idempotent, escapes specials); `compile_latex_to_pdf()`
uses Tectonic (preferred)/latexmk/pdflatex with shell-escape disabled. With no
engine installed it still returns the injected `.tex` + the (text-based) coverage
report. API: `/api/generate` + `/api/repair-with-feedback` accept `resume_latex`
and branch to `_generate_from_latex` / `_repair_from_latex` (return `pdf_b64` +
`tex_b64` + coverage report; status by external coverage). Extension v1.3.0 adds a
**Resume LaTeX** textarea (priority over PDF) and Download PDF / Download .tex.
Regression: `scripts/verify_latex_resume.py`. Tectonic is installed in the
Dockerfile; `pylatexenc` is in `requirements.txt`.

**Recruiter-grade placement + clean inputs + resilient Run (Phase 7):** the live
output was a wall of repeated "Core Competencies:" lines full of LinkedIn/Simplify
UI junk. Fixed across four axes (all honesty-preserving):
- **Clean JD (R13):** `extension/content.js::extractGenericJD` strips a CLONE
  (`stripChrome`: nav/aside/header/footer/button/overlay + `simplify`/`jobalytics`/
  `jobscan`/`teal`/`__extension` nodes) and `scrubOverlayLines` drops CTA lines;
  `external_ats._is_ui_noise()` (wired into `_is_term_like`) rejects UI/CTA/
  marketing n-grams so page chrome never becomes a keyword.
- **DOCX distribution (R14):** `resume_renderer._write_skills` emits ONE capped
  line per category (`_PER_CATEGORY_CAP=10`, `_SKILLS_TOTAL_CAP=28`; no more
  `PER_LINE=12` repeated headers); `_build_skill_pool`/repair cap is
  `_SKILLS_DISPLAY_CAP=26` (was 200); surplus includable terms are redirected into
  the Summary (`_append_summary_terms`) + Experience (`_force_weave_into_bullets`),
  not extra skills lines.
- **LaTeX distribution (R14):** `latex_resume.inject_keywords` distributes terms
  across a Summary sentence + woven Experience `\item`s + ONE competencies line;
  `_remove_injected_block` strips every fragment (idempotent).
- **Resilient Run (R15):** `extension/background.js` owns generation and persists
  `running`→`done`/`error` to `ats_results` keyed by normalized URL; the popup
  restores spinner/result on open + live-refreshes via `chrome.storage.onChanged`.
  Closing the popup no longer loses the run (extension v1.4.0).
Regressions: `verify_clean_jd_extraction.py`, `verify_skills_distribution.py`,
`verify_resilient_run.py` (+ existing `verify_latex_resume.py` /
`verify_max_ats_coverage.py` still pass).

**Trust + usability (Phase 8, extension v1.5.0):** four live-use fixes, all
honesty-preserving:
- **Reliable PDF (R16):** the HF/Linux DOCX flow never wrote a `.pdf` sidecar so
  only DOCX downloaded. `api_server` now calls `pdf_writer.docx_to_pdf` on both
  the generate and repair DOCX paths (reportlab off-Windows) → `pdf_b64` always
  populated. The LaTeX path returns a `pdf_error` (`engine_missing` vs
  `latex_error` + `compile_log`) and STILL returns `tex_b64` on compile failure
  (200, not 500); `Dockerfile` has a hard `RUN tectonic --version` gate.
- **Non-destructive tailoring is the DEFAULT (R17):** `NON_DESTRUCTIVE_DEFAULT=True`
  + `_apply_non_destructive` rebuild `tailored.roles` VERBATIM from the base resume
  (titles/companies/dates/existing bullets byte-identical) and add keywords ONLY via
  Summary augmentation + `_append_keyword_bullets` (≤3 gated lines appended per
  role). ALL four bullet-mutation sites — incl. `_maximize_external_coverage` (LIVE
  in max-ATS mode) — are gated behind `non_destructive`, so the candidate's real
  history is never altered. LaTeX `inject_keywords` APPENDS ≤3 `\item`s (tagged
  `% ats-item`) after each experience instead of editing existing items. Proven by
  `verify_non_destructive.py` (end-to-end `_generate_resume_v4` + `_maximum_ats_mode`
  diff against the base). Coverage relocates to appended bullets+Skills+Summary
  (still ~96%); the max-ATS regression threshold is an honest present-in-export floor.
- **Persistent side panel (R18):** `content.js::injectSidePanel` docks a persistent,
  collapsible/dismissible left iframe hosting `popup/popup.html` (exposed via
  `web_accessible_resources`), idempotent + non-blocking; the toolbar popup is a thin
  launcher (`TOGGLE_PANEL`). The disappearing-popup "run restarted" feeling is gone;
  the panel live-reflects the background-owned run.
- **History (R19):** `renderHistory` lists prior generations per job (title/company/
  time/internal+external scores/status) with restore (via `applyResult`) +
  re-download; `background.js` `MAX_SAVED` 10→40 with a `BYTE_BUDGET` quota guard
  (`approxSize`/`stripOldestArtifacts`) that degrades oldest entries to
  metadata-only, always keeping the newest fully downloadable.
Regressions: `verify_pdf_download.py`, `verify_non_destructive.py`,
`verify_side_panel.py`, `verify_history.py` (+ all Phase 7 scripts still pass).

**Status system** (`src/fit_gate.py`): `READY_90_PLUS` /
`READY_90_PLUS_REVIEW_RECOMMENDED` / `READY_95_EXTERNAL_ALIGNED` (feedback repair
only: internal AND independent ≥ 90, pasted gaps mostly resolved, no review flags —
fabrication cannot reach it) / `NEEDS_REPAIR` / `NEEDS_USER_INPUT` /
`NOT_ELIGIBLE_LOW_FIT` / `PARSE_FAILED`.
**Maximum ATS statuses:** `READY_MAX_ATS_95_PLUS` / `READY_90_PLUS_EXTERNAL_ALIGNED`
/ `NEEDS_USER_CONFIRMATION` / `BELOW_TARGET_REPAIRABLE`.
**Quality flags:** `CLEAN_90_PLUS` / `AGGRESSIVE_90_PLUS` /
`REVIEW_REQUIRED_90_PLUS` / `WEAK_90_INTERNAL_ONLY` (internal ≥90 but independent <90).

**Candidate Experience Vault** (`src/candidate_vault.py`, `data/candidate_vault.json`,
gitignored/per-user): accumulates terms across jobs. Only `user_confirmed` +
`resume_original` are treated as fully safe; inferred/jd_expansion stay
reviewable — the system never silently promotes a risky term. `confirm_terms()`
lets the user lock a term safe or blocked; the fit classifier consults it first.

---

## 4. ⚠ Reversals from older versions of this doc (history that changed)

- **Skills section: RE-ADDED.** Older docs said "NO Core Competencies/Skills
  section." That was wrong — research showed a dedicated, standard-headed Skills
  section is the #1 ATS keyword vehicle (Jobalytics/Resume Worded parse it).
  `Resume.skills` + `resume_renderer._write_skills` render a categorized SKILLS
  section (Tools & Analytics / Methodologies / Domains / Core Competencies),
  placed **after KEY ACHIEVEMENTS**, ≤12 items per line (multi-line if more).
- **Extraction is SKILLS-ONLY + LLM-unioned.** `extract_jd_keywords` keeps only
  recognized skills (no "any noun"); the v4 LLM also returns a `jd_skills` array
  (Jobalytics-style) that is unioned in. No prose-noun garbage.
- **Provider-independent.** Generation no longer assumes one model — see §6.
- The `_assert_no_dump_footer` postcondition still bans raw keyword-dump footers,
  but the categorized Skills section is allowed.

---

## 5. End-to-end flows

### Web app (batch)
`ui.py` wizard → scrape (`src/scrapers/*` + ever-jobs) → `geo_filter` (location)
→ dedup → `job_assessor` → `ResumeCustomizer.customize_for_jobs` →
`_generate_resume_v4` (provider chain + repair + scoring + status) → render
DOCX/PDF → Google Sheets / Excel / CSV / run history (`hf_storage` syncs to HF
Dataset on save). Batch table shows status, internal+independent scores, risk,
download gating. UI startup calls `hf_storage.sync_down()` once per session.

### Extension (per-job)
`extension/` (MV3) → content.js extracts JD (generic + LinkedIn/Naukri/Indeed) →
popup **Run** → background.js POSTs multipart to `POST /api/generate` →
`api_server.generate_resume_for_api` →
`ResumeCustomizer._run_provider_chain(..., base_resume_override=...)` (same path)
→ returns base64 DOCX/PDF + scores/status → popup shows scores + Download.

**Improve with ATS feedback (External Feedback Repair Mode):** after a resume
exists, user pastes Jobalytics/Simplify feedback → popup **Improve** →
background.js `REPAIR` → `POST /api/repair-with-feedback` →
`jobalytics_repair.repair_with_external_feedback` (every missing keyword
risk-classified; BLOCKED + unconfirmed HIGH never added; re-render → re-parse →
re-score) → updated scores/status + added / unresolved-high / blocked terms +
base64 DOCX/PDF. No ATS logic in extension JS.

**Result persistence:** generated/repaired results are cached in
`chrome.storage.local` (`ats_results`, keyed by tab URL). Reopening the popup on
the same job link restores scores, Download, and the Improve section (pruned to
10 most-recent jobs). Reload the unpacked extension after pulling `extension/`
changes from GitHub.

### External ATS Feedback Repair (backend detail)
`src/jobalytics_repair.py`:
- `parse_external_feedback(text)` — extracts external score + missing/matched
  keywords from a freeform paste; drops prose tokens when there's no real list.
- `repair_with_external_feedback(...)` — runs `regenerate_from_jobalytics` (full
  v4 pipeline in `jobalytics_repair` mode with pasted addable keywords merged
  into the include pool), then derives added / review-flag / unresolved-high /
  blocked lists and final status (may upgrade to `READY_95_EXTERNAL_ALIGNED`).
Verified by `scripts/verify_feedback_repair.py` (CISSP blocked, SIEM unresolved-high).

---

## 6. LLM provider abstraction (model-independent)

`src/providers.py`: `LLMProvider` base, `OpenAICompatProvider` (Kimi/NVIDIA —
all OpenAI-compatible cfgs), `StubProvider` (deterministic, offline floor +
tests). `build_provider_chain()` reads `config.LLM_GENERATION.provider_order`.

- Every model response is **schema-validated** (`validate_response` against
  `JD_ANALYSIS_SCHEMA` / `RESUME_TAILORING_SCHEMA`); quality ∈
  `ok | failed_schema | provider_error`. **Schema-failed output can never become
  READY** — it forces fallback to the next provider / deterministic path.
- `src/provider_prompts.py` loads provider-specific templates from `prompts/`
  (`{task}_{claude|kimi|nvidia}.txt`). `LLMClient.tailor_resume_v4(...,
  provider_family=...)` selects the prompt. **Any stub/test that monkeypatches
  `tailor_resume_v4` must accept `**kwargs`** (it takes `provider_family`).
- Pipeline records `provider_used` + `provider_response_quality` on the report.

---

## 7. File map (current)

### Entry points
| File | Purpose |
|---|---|
| `api_server.py` | **HF Space entrypoint** (`start.sh` runs it). FastAPI on :7860: `/api/health`, `/api/generate`, `/api/repair-with-feedback`; launches Streamlit subprocess on :8501 and reverse-proxies everything else (HTTP via httpx + WS for `/_stcore/stream`). Imports `src/` — no ATS logic of its own. |
| `ui.py` | Streamlit web app (~2400 lines): wizard, results, batch table, vault controls. |
| `start.sh` | ever-jobs sidecar (optional) → `exec python api_server.py`. |
| `Dockerfile` | HF build. |

### ATS pipeline v2 (`src/`)
| File | Purpose |
|---|---|
| `jd_analyzer.py` | `analyze_jd` → structured `JDRequirements` (titles, required/preferred hard skills, tools, responsibilities, domains, certs, education, soft, seniority) each with importance/source_phrase/aliases/placement. Deterministic floor + optional LLM enrichment. |
| `candidate_fit.py` | Fit/severity classification (§3): `classify_all_fit`, `severity`, `auto_terms`, `high_risk_terms`. |
| `fit_gate.py` | `assess_job_fit_for_90` + status constants. |
| `ats_scoring_v2.py` | `score_jd_match` (weighted: 35 must-have/20 responsibilities/15 tools/10 title+domain/10 seniority/5 certs/5 soft + penalties) + `score_ats_readability`. |
| `ats_validator.py` | **Independent** evidence-weighted validator (anti-circular). |
| `ats_report.py` | `build_ats_report` (explanation: scores, strong/weak, gaps, formatting, recommendations) + `reconcile_missing_keywords`. |
| `candidate_vault.py` | Persistent learned-skills vault. |
| `jobalytics_repair.py` | `classify_jobalytics_keywords`, `regenerate_from_jobalytics`, `parse_external_feedback`, `repair_with_external_feedback` — honest external-checker feedback repair (§5). |
| `hf_storage.py` | Best-effort HF Dataset sync for Streamlit run history/resumes/reports (private dataset; `sync_down` on UI startup, `push_run` after each run). |
| `ats_scorer.py` | Base scorer/helpers still used: `extract_jd_keywords` (skills-only), `_kw_in_text`, `_phrase_in_text`, `_strip_keyword_spam` (drops 15+-separator dump lines), `conservative_display_score`, `PM_SKILL_TAXONOMY`. |
| `resume_model.py` | `Resume`/`Role`/`Education`/`Contact` dataclasses (+ `skills`). |
| `resume_parser_v2.py` | `parse_resume_pdf` / `parse_resume_pdf_cached` (PDF → Resume). |
| `resume_renderer.py` | Canonical DOCX renderer incl. the SKILLS section. |
| `resume_customizer.py` | **Heart.** `_generate_resume_v4` (parse → provider tailor → backfill → weave → skills → render → parse-validate → score internal+independent → auto-repair ≤3 → status/quality/download gating), `_run_provider_chain`, `_validate_parsed_resume`, `customize_for_jobs`. |
| `llm_client.py` | `tailor_resume_v4`, `analyze_jd_requirements`, `judge_evidence`, `repair_resume_v4`, `jobalytics_repair_v4` (all OpenAI-compatible, cfg-driven). |
| `providers.py` / `provider_prompts.py` | Provider abstraction (§6). |
| `geo_filter.py` | Drops jobs outside selected locations (respects Remote/Worldwide). |
| `pdf_writer.py` | DOCX→PDF (reportlab on Linux/HF). |
| `job_assessor.py`, `gsheets.py`, `excel_reporter.py`, `run_history.py`, `job_history.py` | Assessment, Sheets, Excel, history, dedup. |

### Extension (`extension/`)
| File | Purpose |
|---|---|
| `manifest.json` | MV3; permissions: storage, downloads, activeTab, scripting; `<all_urls>`. |
| `content.js` | JD extraction: generic + LinkedIn/Naukri/Indeed + `document.title` fallback; fast (single short MutationObserver wait). |
| `popup/popup.html` + `popup.js` | Run (enabled immediately), status, scores, **Download (always shown, labeled by score)**, **Improve with ATS feedback** panel, result restore from `chrome.storage.local`, ⚙ Options. **Download in popup** (Blob + `<a download>`). |
| `background.js` | Thin service worker: `GENERATE` → `/api/generate`, `REPAIR` → `/api/repair-with-feedback`. Reads resume/API settings from `chrome.storage.local`. NO ATS/LLM logic. |
| `options/` | Upload resume once (→ base64 in `chrome.storage.local`) + API URL + token. |

### Config
`config.py`: `ASSESSMENT_MODELS` (with `tailor` flags), `LLM_GENERATION`
(provider_order, fallback flags), `AUTOMATION` (review/download policy).
Secrets in `.env` / HF Space secrets: LLM keys, `API_SECRET_TOKEN`, `HF_DATASET_REPO`
(for `hf_storage`), `EVER_JOBS_API_URL`, Google. **Never committed.**
Also: `CONTACT_LOCATION` (default `Hyderabad, Telangana, India · Open to relocate`)
for resume header address/location ATS compliance.

### Tests / scripts
| File | Purpose |
|---|---|
| `scripts/verify_anticheat.py` | **Anti-circular suite — keep green.** |
| `scripts/verify_feedback_repair.py` | External feedback repair: parse + honest weave; CISSP blocked, SIEM unresolved-high. |
| `scripts/verify_maximum_ats.py` | Maximum ATS Mode: PM/AI→user-confirmed; certs/seniority/eng still gated; coverage improves; 65%→repairable. |
| `scripts/verify_max_ats_coverage.py` | Internal-96/external-54 regression: exported DOCX covers ≥90% includable PM terms; creds/seniority excluded; status by external coverage. |
| `scripts/verify_latex_resume.py` | LaTeX path: text extraction, honest keyword injection (22%→88% coverage), DISTRIBUTED placement (summary + experience + 1 competencies line), no fabrication (cissp/pmp/12+ years/cuda blocked), idempotent injection. |
| `scripts/verify_clean_jd_extraction.py` | R13: page/extension UI chrome (Simplify/Jobalytics CTA text) never becomes a keyword; genuine craft terms survive; content.js strips chrome. |
| `scripts/verify_skills_distribution.py` | R14: ≤1 line per skills category (no repeated "Core Competencies:" dump), 18-28 total de-duplicated items. |
| `scripts/verify_resilient_run.py` | R15: background owns generation (running→done/error keyed by URL); popup restores + live-refreshes; popup↔background storage contract matches. |
| `scripts/verify_90_pipeline.py` | Full fit→repair→status pipeline (stub floor). |
| `scripts/verify_scoring_v2.py` | jd_analyzer + evidence + scoring + report + feedback loop. |
| `scripts/evaluate_model_providers.py` | Run same jobs through each provider (Kimi/NVIDIA) — ready rate, avg independent, schema errors. **Needs real keys.** |
| `scripts/consistency_test.py` | Same JD ×3 per provider → variance. |
| `scripts/run_20_job_validation.py` | 20-job end-to-end + validation packages. **Needs real keys.** |
| `tests/test_api_parity.py` | Proves API path == web-app path (identical scores). |
| `tests/test_api_health.py` | `/api/health` + token gate. |
| `tests/fixtures/jds/*.txt` | Real JDs incl. `backend_engineer`, `senior_pm_10yrs` (anti-cheat). |

---

## 8. Current status & known gotchas

- **Phase 6 (Chrome extension) is COMPLETE and live.** End-to-end verified on a
  real Kimi/NVIDIA run: a good-fit job returned READY_90_PLUS, independent 98,
  and a downloadable DOCX. Parity test passes.
- **External ATS Feedback Repair Mode is wired end-to-end** (`f5dd879`): backend
  `/api/repair-with-feedback` + extension Improve panel. Requires the Space to be
  rebuilt/running with that commit — if Improve fails with 404, the endpoint isn't
  deployed yet.
- **Extension result persistence is live** (`ef247ae`): reopening the popup on the
  same job URL restores the last generated/repaired result. User must reload the
  unpacked extension after pulling from GitHub.
- **Per-job variance is expected** (see §2): borderline jobs land NEEDS_REPAIR /
  ~73-85 on Jobalytics — that's honest, not a bug. Feedback repair pushes toward
  90 only where missing keywords are plausible-and-weavable; HIGH/BLOCKED terms
  correctly stay out → `NEEDS_USER_INPUT`.
- **MV3 gotcha:** service workers have **no `URL.createObjectURL`** → downloads
  are done in the popup, not the service worker.
- **Proxy gotcha:** `api_server` strips `content-encoding`/`content-length` when
  proxying Streamlit (httpx already decompresses) — otherwise pages render blank.
- **Deploy gotcha:** pushing to `hf` rebuilds the Space (~2-15 min for Docker) and
  causes transient 404/405s while the container is down. **Prefer pushing
  extension-only changes to `origin` only** (no Space rebuild needed) — but both
  remotes may end up at the same commit if you push everything to both.
- **HF Space is private:** unauthenticated health checks from outside may return
  HF gateway 404 even when the Space is fine for token-authenticated extension calls.
- **Local `import ui` fails** on the dev machine due to a pre-existing
  pandas/numpy binary mismatch — unrelated to our code; HF's clean deps are fine.
- **ever-jobs sidecar** fails to start on HF unless `EVER_JOBS_API_URL` points to
  an externally hosted sidecar; known/non-blocking; dedicated scrapers still work.
- **Scrapling anti-block fetch layer** (`src/scrapers/fetch.py`): all scraper HTTP
  goes through Scrapling (Chrome TLS impersonation via `Fetcher`; Cloudflare
  bypass via `StealthyFetcher`/Camoufox) with a `requests` fallback. Set
  `SCRAPER_PROXIES` (comma/newline list) to rotate proxies — the ONLY fix for
  datacenter-IP blocking (fingerprint stealth ≠ IP reputation). Camoufox is
  installed via `scrapling install` in the Dockerfile (non-fatal).
- **Direct Company ATS** (`src/scrapers/company_ats.py`, platform key
  `company_ats`, default-on): Greenhouse/Lever/Ashby public JSON boards — the most
  block-resistant bulk source on HF. Extend via `COMPANY_ATS_BOARDS` env (JSON).
- **Streamlit batch persistence:** `hf_storage.py` syncs run history/resumes/CSV to
  a private HF Dataset so data survives Space container restarts (needs
  `HF_DATASET_REPO` + write token in Space secrets).
- **Security:** do not store HF write tokens in `.git/config` remote URLs — use
  `huggingface-cli login` or a credential helper instead.

### Open / next
- Run `evaluate_model_providers.py` + `run_20_job_validation.py` with real
  Kimi/NVIDIA keys (user's environment) — the live provider validation.
- Integrate a no-block job source for HF batch scraping — DONE via Scrapling
  fetch layer + Direct Company ATS (Greenhouse/Lever/Ashby). Next: add a
  residential proxy (`SCRAPER_PROXIES`) to make LinkedIn/Indeed reliable on HF,
  and run a live bulk test to confirm block rates dropped.

---

## 9. Conventions (follow these)

- **Update `HISTORY.md` + `README.md` on every change** (standing user rule).
- **Commit/push only when asked.** Conventional Commits (`fix(scope): …`) + a
  `Co-Authored-By` trailer when AI-assisted. Single `main` branch, no rebases.
- **Push extension-only changes to `origin` only**; push pipeline/API/HF changes
  to both `origin` and `hf`.
- **Secrets never committed** (`.env`, `google_*.json`, resume PDFs, `data/`).
- **Honesty over false confidence.** If a fix is partial, say so. If our score >
  what a real checker would give, say so. Never fake keywords or loosen the
  independent validator to inflate numbers.
- **Gold reference resume:** `C:\Users\Nxtwave\Desktop\resume\Saiteja_Tirunagari_Resume A 26 - Copy.pdf`.
- Test vaults are isolated in regression scripts (temp paths) — don't pollute the real vault.

---

## 10. Where to start

- **Low/odd ATS scores:** read `ats_scoring_v2.py` + `ats_validator.py` (internal
  vs independent), then `candidate_fit.py` (what got included), then the
  `_generate_resume_v4` repair loop. Run `verify_anticheat.py` + `verify_90_pipeline.py`.
- **Extension issues:** `extension/` (manifest → content.js → popup.js →
  background.js) + `api_server.py`. Remember MV3 download + proxy gotchas (§8).
  Improve broken? Check Space has `/api/repair-with-feedback`. Results lost on
  reopen? Check extension was reloaded after persistence commit + same job URL.
- **Feedback repair / 95% path:** `jobalytics_repair.py` → `api_server.py`
  repair endpoint → extension `REPAIR` handler. Run `verify_feedback_repair.py`.
  Never loosen validator or auto-add HIGH/BLOCKED terms.
- **Provider/model work:** `providers.py` + `provider_prompts.py` + `prompts/` +
  `llm_client.py`. Keep schema validation; keep stubs `**kwargs`-tolerant.
- **New JD-keyword behavior:** `jd_analyzer.py` (extraction) and `candidate_fit.py`
  (inclusion). Don't reintroduce "any noun" extraction or fake injection.

---

## 11. Quick commands

```bash
# Regressions (deterministic, no keys)
PYTHONPATH=. python scripts/verify_anticheat.py
PYTHONPATH=. python scripts/verify_90_pipeline.py
PYTHONPATH=. python scripts/verify_scoring_v2.py
PYTHONPATH=. python scripts/verify_feedback_repair.py
PYTHONPATH=. python -m pytest tests/test_api_parity.py tests/test_api_health.py -q

# Provider eval (needs real Kimi/NVIDIA keys in .env)
PYTHONPATH=. python scripts/evaluate_model_providers.py
PYTHONPATH=. python scripts/run_20_job_validation.py

# Run the API locally (serves /api + proxies Streamlit)
python api_server.py            # http://localhost:7860

# Deploy
git push origin main            # always
git push hf main                # ONLY for pipeline/API/Space changes (rebuilds Space)

# Live checks (Space must be Running; private Space may 404 without auth)
# GET  …/api/health                  → {"status":"ok"}
# POST …/api/repair-with-feedback    → 405 on GET means route exists; POST needs token + multipart
```

---

End of context. Code is the source of truth — update this doc when you finish a task.