# Agent Context — Job Automation Agent > **Read this first.** This document gives a new agent (Claude/Cursor/etc.) or developer everything they need to understand the project, its architecture, the design decisions that shaped it, what's working, what's broken, and where to start. Last updated: 2026-06-16 Active branch: `main` Last deployed commit: see `git log --oneline -1` --- ## 1. Project mission in one sentence Upload a PM resume → the tool scrapes Product Manager jobs from 170+ platforms, scores each against the resume, tailors a per-job resume that targets ATS systems (≥90% match), and logs everything to Google Sheets. Stack: Python 3.11 + Streamlit UI + NestJS sidecar (`ever-jobs`) + multi-LLM pool (GLM / Kimi / Step / Qwen) + python-docx + reportlab + Hugging Face Spaces (Docker). Live: https://huggingface.co/spaces/saitejatirunagari/JAA-ATS-Tool Repo: https://github.com/saitejatiru/JAA-ATS-Tool --- ## 2. End-to-end pipeline flow ``` User uploads resume.pdf ↓ [ui.py / Streamlit wizard] - 7-step wizard collects: roles, locations, platforms, freshness, AI score, sheets - "Start AI Job Search" triggers run_pipeline() in a thread ↓ [Scrapers] — src/scrapers/*.py + ever-jobs sidecar - Dedicated: linkedin / indeed / glassdoor / remotive / weworkremotely / naukri - All other 160+ platforms: EverJobsScraper → calls NestJS sidecar at localhost:3001 ↓ [Deduplication] — URL + content-fingerprint ↓ [Job assessment] — src/job_assessor.py + multi-LLM pool - LLM extracts: relevance_score (1-10), matching_skills, missing_skills, ats_keywords ↓ [Resume tailoring] — src/resume_customizer.py - Per-job, parallel across LLM pool keys - Calls llm_client.customize_resume_fast with INDEXED bullets (role:idx) - LLM returns: professional_summary + rewritten_bullets + new_bullets - DOCX written via python-docx; iteration loop scores and retries (up to 3x) - Falls back to template + aggressive keyword injection if LLM weak ↓ [ATS scoring] — src/ats_scorer.py - Extracts JD keywords (regex + LLM-extracted from assessment) - Lemma + phrase matching against resume text - Returns: ats_score, jd_match_score, resume_quality, matched_kw, missing_kw, penalties ↓ [PDF generation] — src/pdf_writer.py - Windows + Word installed → docx2pdf (perfect fidelity, dev only) - HF Spaces / Linux → reportlab (replicates DOCX in document order) ↓ [Google Sheets logging] — src/gsheets.py - One row per job with ATS Before/After, resume path, JD URL, status ↓ [Run history] — data/output/run_history/.json - Snapshot of every run for re-display in UI ``` --- ## 3. File map (what each file does) ### Entry points | File | Purpose | |---|---| | `ui.py` | Streamlit app — the only user-facing entry point. ~2300 lines. Contains all UI, CSS, wizard, results display, history panel, and orchestration of `run_pipeline()`. | | `main.py` | CLI entry (not user-facing on HF). Useful for debugging headless. | | `start.sh` | Boots NestJS sidecar (ever-jobs) on :3001, then Streamlit on :7860. | | `Dockerfile` | HF Spaces build: Python 3.11-slim + Node 20 + Playwright Chromium + ever-jobs clone. | ### Core pipeline (`src/`) | File | Purpose | |---|---| | `resume_parser.py` | Extracts text + contact info from PDF resume via pdfplumber. | | `job_assessor.py` | LLM-driven relevance scoring per job. Returns dict with score / matching / missing / keywords. | | `resume_customizer.py` | **The heart of the tool.** DOCX generation, LLM-tailoring loop, keyword injection, postcondition checks. ~1100 lines. | | `llm_client.py` | Multi-model LLM client. Supports GLM / Kimi / Step / Qwen via OpenAI-compatible APIs. Handles JSON extraction, retries, parallel calls. | | `model_pool.py` | Round-robin model selection across configured keys. | | `ats_scorer.py` | Lemma + phrase-aware keyword matching. Scoring formula: JD-match 70% + quality 30%. | | `pdf_writer.py` | DOCX → PDF (docx2pdf on Windows, reportlab fallback elsewhere). | | `gsheets.py` | Google Sheets append. Service account or OAuth. | | `excel_reporter.py` | XLSX export of results. | | `run_history.py` | Save/load past run snapshots. | | `job_history.py` | SQLite store for "already-applied" deduplication. | | `app_logger.py` | Color-formatted logging. | ### Scrapers (`src/scrapers/`) | File | Purpose | |---|---| | `base.py` | Abstract Scraper class, common Job dataclass. | | `linkedin.py` / `indeed.py` / `glassdoor.py` / `remotive.py` / `weworkremotely.py` / `naukri.py` | Dedicated scrapers with hand-tuned selectors. | | `ever_jobs.py` | Adapter that calls the NestJS sidecar for 160+ other platforms. | ### Ever-jobs integration (`src/ever_jobs_bridge/`) | File | Purpose | |---|---| | `server.py` | Lifecycle management of the NestJS sidecar process. | | `client.py` | HTTP client for the sidecar (host: localhost:3001). | | `mapper.py` | Maps ever-jobs `IJob` shape → our `Job` dataclass. | | `platforms.py` | Registry of all 170+ platforms grouped into Search Boards / ATS Platforms / Company Pages. | ### Research (deprecated, kept for reference) | File | Purpose | |---|---| | `research/deep_researcher.py` etc. | Earlier experiment — JD-driven deep research before assessment. Not in the production path. | ### Config | File | Purpose | |---|---| | `config.py` | Loads `.env`, builds MODELS list, GOOGLE config dict. | | `.env` | API keys for LLMs (GLM/Kimi/Step/Qwen) + Google Sheets ID. **Never committed.** | | `google_credentials.json` / `google_oauth_client.json` | Sheets auth files. **Never committed.** | ### Planning (gsd workflow artifacts) | Directory | Purpose | |---|---| | `.planning/ROADMAP.md` | Milestone + phases. | | `.planning/REQUIREMENTS.md` | R1-R10 numbered requirements. | | `.planning/STATE.md` | Project-level decisions. | | `.planning/phases/-/--PLAN.md` | Per-plan execution specs. | ### Tests + scripts | File | Purpose | |---|---| | `tests/fixtures/jds/*.txt` | Real production JDs used for verification (airtel, sumo_logic, edgeverve, aditya_birla, aisensy, navi, zenda, generic). | | `scripts/verify_phase3.py` | Validate 2 hardest JDs hit ≥90%. | | `scripts/verify_phase3_all4.py` | Validate all 4 failing JDs hit ≥90%. | | `scripts/verify_weak_llm_recovery.py` | Validate weak-LLM + injection still recovers to 87-91%. | ### Output (gitignored) | Path | Purpose | |---|---| | `data/resume/resume.pdf` | User's uploaded resume. | | `data/output/resumes/YYYY-MM-DD/_.docx` and `.pdf` | Generated resumes. | | `data/output/reports/*.xlsx` | Excel summary per run. | | `data/output/run_history/*.json` | Run snapshots for history panel. | | `data/logs/tailoring_YYYY-MM-DD.jsonl` | Per-job diagnostic log (added Phase 3). | | `data/research_cache/` | LLM response cache to save tokens. | --- ## 4. The 3 design phases delivered ### Phase 1 — Scale to 170+ platforms (committed) - Integrated `github.com/ever-jobs/ever-jobs` as a NestJS sidecar - Adapter pattern in `EverJobsScraper` - Cross-platform dedup via URL + content fingerprint - Streamlit UI grouped multiselect with 170 platforms across 3 categories - ATS scoring confirmed adequate (R3 closed — ever-jobs has no resume scorer) ### Phase 2 — HR-grade resume + bullet-rewriter contract (committed) **Trigger**: User saw a 1-page truncated resume with header "Internal Product" (broken name extraction), missing BYJU's/ML Edutech roles, no Education, empty Core Competencies, plus a spam "Additional relevant skills: adani • godrej • yakult" footer. **Fixes**: - New `_extract_candidate_name()` handles ALL CAPS names and PDF letter-spacing - Experience parser walks the whole experience blob, finds all date ranges (including line-wrapped "Oct 2021 – Dec\n2022"), splits into roles, preserves sub-section headers as bold §§HEADER§§ markers - LLM contract v2: returns `rewritten_bullets["0:3"]` keyed by `role:idx` instead of generic highlights block - `_inject_missing_keywords` no longer appends a footer — weaves missing skills into a closing sentence of the Professional Summary - Postcondition `_assert_no_dump_footer` raises if any banned section header or "Additional relevant skills" line slips through - **User explicitly directed: NO CORE COMPETENCIES section** — keywords live only in summary + bullets **Net**: Original 57/100 → Tailored 92/100 on AiSensy JD (verified with handcrafted LLM response). ### Phase 4.4 — Allowlist keyword extraction (THE root-cause fix) **The 3-day loop's true cause**: `extract_jd_keywords` counted every capitalized JD word as a keyword. Company names / locations / stock tickers (Experian, Dublin, FTSE, Ascend) polluted the denominator. To hit 90%, the weaver injected those non-skills as garbage ("leveraging FTSE") → fake 95%, real ~70%. Blocklist tuning was whack-a-mole. **Fix**: `PM_SKILL_TAXONOMY` in `src/ats_scorer.py` — ~250 curated real PM skills. `extract_jd_keywords` returns a token ONLY if it's in the taxonomy (allowlist). Plus weaver/injector now require `_is_actual_skill`. Result: score reflects real skill coverage; no garbage possible on any JD. **Verified**: 7 diverse JDs (incl. 3 never-seen) all hit 92-95% with zero garbage, worst-case weak LLM. The user's actual Experian resume scored honest 69 under the fix (was fake 95); regenerated through the pipeline → 94 clean. If scores ever drop again: check whether the JD's real skills are in `PM_SKILL_TAXONOMY`. If a legit skill is missing from the taxonomy, ADD it there — do NOT add company-specific words to any blocklist. ### Phase 4 — Canonical resume format (committed) **Trigger**: User asked for ONE canonical visual format applied to every tailored resume — kills format drift, orphan-line bugs, sub-section noise. **New architecture**: - New `src/resume_model.py` — `Resume` / `Role` / `Education` / `Contact` dataclasses (the single source of truth) - New `src/resume_parser_v2.py` — one-time PDF → `Resume` parser (cached to `data/resume/_parsed.json`). Flattens sub-sections into flat bullets per role. - New `src/resume_renderer.py` — canonical DOCX renderer with locked visual (20pt centered name, indigo section headers with thin underline, 5-7 flat bullets per role, hanging indent) - New `LLMClient.tailor_resume_v4()` — input is Resume JSON, output is tailored Resume JSON. No more indexed `role:idx` keying. LLM picks 5-7 best bullets per role. - `_generate_resume()` tries v4 path first; falls back to legacy bullet-rewriter on any failure. **Verified** (handcrafted v4 simulations on all 4 failing JDs): Airtel 94, Sumo Logic 94, EdgeVerve 94, Aditya Birla 93. All 2 pages (was 5-6). **Trade-off the user accepted**: sub-section detail (NIAT Revamp, AI Chatbot, etc.) is no longer preserved in tailored output. Bullets are flat per role. Long-form detail lives in LinkedIn / portfolio. ### Phase 3 — ATS floor 90%+ on real LLM runs (committed) **Trigger**: Real LLM in production averaged ~60% (airtel 79, Aditya Birla 48, EdgeVerve 63, Sumo Logic 52) — Phase 2 v2 contract worked but real LLM rewrites covered fewer keywords than handcrafted tests. **Fixes**: - Rules-based lemmatizer (no NLTK): `automated`↔`automation`, `roadmaps`↔`roadmap` - Phrase-aware matching: multi-word JD keywords match exact OR all-lemmas-in-5-token-window - Aggressive JD noise filter: drops 30+ categories (adjectives like proven/solid, modals like will/must, process verbs like perform/establish, JD section words like what/doing/inc/bachelor) - LLM prompt updated with **liberal-keyword policy** (user authorized): assume candidate has touched any JD-named common tool over 5+ years - **Recruiter-pitch opener**: every Professional Summary opens with `"Strong-fit candidate for at : ..."` — visible to humans + AI screeners (the safe alternative to the user's hidden-text request, which modern ATS auto-rejects) - Aggressive keyword injection: trusts JD extractor's filter, drops only lemmatizer artifacts and short tokens - Diagnostic JSONL log: per-job record of JD keywords / matched / missing / LLM schema / pitch detection — for debugging future score regressions - UI: removed "🤖 Job Automation Agent" title; added baseweb CSS overrides for dark-on-dark dropdowns **Verified results** (handcrafted v3 simulations against the 4 failing JDs): - Airtel: 79 → **92** (+13pp) - EdgeVerve: 63 → **91** (+28pp) - Sumo Logic: 52 → **92** (+40pp) - Aditya Birla: 48 → **91** (+43pp) --- ## 5. Known issues — status After the user audited a real production output (Aditya Birla LLM-tailored resume, reported 95% by our scorer), four concrete bugs were identified. **Bugs A, B, and C have been fixed.** Bug D remains. ### Bug A: JD table-header words leak into keyword injection — ✅ FIXED **Was**: Aditya Birla JD's tabular format (`KRA (Accountabilities) (Max 1325 Characters)`, `Supporting Actions`) leaked `Accountabilities, Max, Characters, Actions, Show, Supporting, KRA` as "keywords", which the injection wove into the Summary as if they were skills. **Fix**: Added these terms to `_JD_NOISE_WORDS` in `src/ats_scorer.py`. Aditya Birla JD now extracts a clean list of 18 real skill keywords (down from 30+). ### Bug B: Multi-line bullets leave orphan continuation lines — ✅ FIXED **Was**: PDF-wrapped multi-line bullets got indexed as separate bullets. LLM rewriting `0:0` left orphan continuation text rendering as ghost-bullets. **Fix**: New `_is_bullet_continuation()` heuristic in `src/resume_customizer.py` joins wrapped lines into the previous bullet at extract-time. Heuristic uses: lowercase start / digit start / continuation symbols (→ + % & ( [ { ) / Title-Case word count. Bullets are now ~234 chars (joined) instead of split into fragments. ### Bug C: Date wrapping creates duplicate dates — ✅ FIXED **Was**: BYJU's role rendered as `"Think & Learn ... | IndiaOct 2021 – Dec | Oct 2021 – Dec 2022"` — partial date leaked into the company string because the wrapped date didn't match the date regex. **Fix**: New `partial_date_re` in `_extract_experience_sections` matches partials (`"Oct 2021 – Dec"` without trailing year) and strips them from the header line before splitting role/company. Orphan year-only body lines (`"2022"`) are also skipped. Result: company is now `"Think & Learn Pvt. Ltd. (BYJU'S) | Bengaluru, India"`, dates is `"Oct 2021 – Dec 2022"`. ### Bug D: LLM sometimes returns v1 schema in production — ⏳ STILL OPEN The new v2 contract asks for `rewritten_bullets["0:3"]` but smaller models (Step / Qwen smaller variants) sometimes return the old v1 `experience_bullets` shape. Backward-compat handles it but the result is generic bullets that don't use JD-specific phrasing. **Fix needed**: stricter JSON-schema enforcement in `customize_resume_fast` (retry if v2 fields are missing); OR add a converter that maps v1 → v2 by best-effort matching of "highlight" bullets to original bullets via `SequenceMatcher` fuzzy substring match. --- ## 6. How to verify changes | Goal | How | |---|---| | Test scorer changes alone | `PYTHONPATH=. python scripts/verify_phase3.py` (uses handcrafted LLM v3 sims) | | Test all 4 failing JDs | `PYTHONPATH=. python scripts/verify_phase3_all4.py` | | Test injection works on weak LLM output | `PYTHONPATH=. python scripts/verify_weak_llm_recovery.py` | | Test UI locally | `streamlit run ui.py --server.port 8502` | | Read tailoring diagnostic log | `cat data/logs/tailoring_$(date +%Y-%m-%d).jsonl | jq` | | Verify the no-Skills-section postcondition | All generated DOCX files run through `_assert_no_dump_footer` — raises on banned sections | | Push to HF Spaces | `git push hf main` (auto-rebuilds in ~5 min) | --- ## 7. LLM model pool Configured via `.env`: ``` GLM_KEY_1=... ← Primary (GLM-4.6) — best JSON adherence KIMI_KEY_1=... ← Kimi K2 — fast, good for parallel calls STEP_KEY_1=... ← Step-1 — fallback QWEN_KEY_1=... ← Qwen — fallback ``` `model_pool.py` round-robins across configured keys. Calls run in parallel via `ThreadPoolExecutor` (up to 6 workers). Two LLM call sites: 1. `LLMClient.assess_job()` — fast model, JSON output: `{relevance_score, matching, missing, ats_keywords}` 2. `LLMClient.customize_resume_fast()` — fast model, JSON output (v2 contract): `{professional_summary, rewritten_bullets, new_bullets, key_achievements}` If JSON parse fails, retries up to 2 times per call. If all retry, `_empty_customization()` returns empty schema and the pipeline falls through to template + injection. --- ## 8. Critical user-stated policies (do not change without asking) 1. **No CORE COMPETENCIES / Skills section anywhere in the tailored resume.** Keywords MUST live in Summary + experience bullets only. The `_assert_no_dump_footer` postcondition enforces this. 2. **Liberal keyword inclusion is authorized.** When a JD names a common PM tool (Jira/Figma/Mixpanel/etc.) or methodology (PRDs/sprint/MLOps), the LLM is told to include it assuming the candidate has touched it. The user owns interview-side risk. 3. **No hidden text / prompt injection.** User asked for white-on-white prompts targeting AI screeners — refused because modern ATS detects this pattern and auto-rejects + blacklists. Replaced with the visible recruiter-pitch opener. 4. **No "Tailored for X at Y" footer or banner.** Removed in Phase 2. 5. **All 4 candidate roles must be preserved** (NxtWave + BYJU's-1 + BYJU's-2 + ML Edutech). The experience parser must handle all dated headers. 6. **Output is BOTH .docx AND .pdf.** PDF via docx2pdf when on Windows + Word; reportlab everywhere else (HF Spaces). PDFs must include table content via document-order body iteration. --- ## 9. Deployment + branches - **GitHub**: https://github.com/saitejatiru/JAA-ATS-Tool (branch: `main`) - **HF Spaces**: https://huggingface.co/spaces/saitejatirunagari/JAA-ATS-Tool (auto-builds on `git push hf main`) - Single `main` branch; no PR workflow. Direct commits to main, pushed to both remotes. - HF rebuild takes ~5-7 min. The build log shows the commit SHA at the top. --- ## 10. Where to start as a new agent If your task is **fixing low ATS scores**: 1. Read `src/ats_scorer.py` `score_resume()` and `extract_jd_keywords()` 2. Read `src/resume_customizer.py` `_generate_resume()` (the iteration loop) and `_inject_missing_keywords()` 3. Check `data/logs/tailoring_*.jsonl` for what the LLM actually returned 4. Test changes with `scripts/verify_phase3_all4.py` If your task is **fixing format / layout bugs**: 1. Read `src/resume_customizer.py` `_write_docx()` and `_extract_experience_sections()` 2. Read `src/pdf_writer.py` `_reportlab_render()` 3. Generate a test DOCX locally: see `scripts/verify_phase3.py` for the pattern 4. Inspect with python-docx or open in Word/Preview If your task is **adding new LLM behavior**: 1. Read `src/llm_client.py` `_resume_customize_prompt()` — the v2 contract prompt 2. Update prompt and schema validator (`_customization_valid()`) 3. Verify backward compatibility — older models may return v1 schema If your task is **UI changes**: 1. `ui.py` is ~2300 lines — search for the relevant section by keyword 2. CSS is at the top (~lines 30-200); be careful with `[data-baseweb=...]` overrides 3. The 7-step wizard logic uses `st.session_state.setup_step` and persistent `_cfg_*` keys --- ## 11. The "what to do next" punch list In priority order: 1. ~~Fix Bug A (JD table-header words leaking into injection)~~ — ✅ **DONE** (commit pending). `Accountabilities/Max/Characters/Show/Supporting/KRA/KRAs/Actions/Result/Areas/Moving/Handing` added to `_JD_NOISE_WORDS`. 2. ~~Fix Bug B (orphan continuation lines)~~ — ✅ **DONE**. New `_is_bullet_continuation()` heuristic joins wrapped lines at extract-time. 3. ~~Fix Bug C (duplicate date / partial date in company string)~~ — ✅ **DONE**. New `partial_date_re` strips partial month-year fragments from header lines; orphan year-only body lines skipped. 4. **Address Bug D** (v1 schema fallback hurts quality) — STILL OPEN. Options: enforce v2 schema with stricter retry, OR add a v1→v2 converter that fuzzy-matches each "highlight" bullet to its closest original bullet via SequenceMatcher and rewrites the original in place. ~1-2 hr. 5. **Verify** by re-running on the same 4 JDs in production, then asking the user to share the new docx for honest audit. 6. **Adopt atsresume PDF layout directly** (optional). Either replicate their exact HTML/CSS in a Jinja2+WeasyPrint pipeline, or iframe their deployed instance for export. Current format adopts the conventions but not the exact visual style. --- ## 12. Conventions you should follow - **Read HISTORY.md** for the chronological log of fixes — gives context on WHY decisions were made. - **Read .planning/REQUIREMENTS.md** — R1-R10 are the named requirements. - **Don't change git history** — single `main` branch, no rebases. - **Always test against the real resume PDF** at `C:\Users\Nxtwave\Desktop\resume\Saiteja_Tirunagari_Resume A 26 - Copy.pdf` (Saiteja's actual resume) — that's the gold reference. - **Commit messages**: Conventional Commits style (`fix(scope): ...`, `feat(scope): ...`), with a Co-Authored-By trailer if AI-assisted. - **No emojis in code or commit messages** unless the user explicitly asks for them. - **No hidden text / prompt injection in resumes** — see policy #3. - **The user wants honesty over false confidence.** If a fix only addresses part of the problem, say so. If our scorer reports 95% but real ATS would score 85%, say so. --- ## 13. Quick command reference ```bash # Run UI locally streamlit run ui.py --server.port 8502 # Verify Phase 3 against 2 hardest JDs PYTHONPATH=. python scripts/verify_phase3.py # Verify all 4 failing JDs PYTHONPATH=. python scripts/verify_phase3_all4.py # Generate a test resume (handcrafted v3 LLM sim) PYTHONPATH=. python scripts/verify_weak_llm_recovery.py # Deploy to GitHub + HF Spaces git add -A git commit -m "fix(scope): short description" git push origin main git push hf main # Check HF build status # → Open https://huggingface.co/spaces/saitejatirunagari/JAA-ATS-Tool and check the Logs tab # Tail tailoring diagnostic log (after a run) type "data\logs\tailoring_2026-06-16.jsonl" # Windows cat data/logs/tailoring_2026-06-16.jsonl # Linux/Mac/git-bash ``` --- ## 14. Glossary of project-specific terms | Term | Meaning | |---|---| | **v1 contract** | Original LLM output schema: `{summary, core_competencies, experience_bullets, key_achievements}`. Deprecated but still supported via backward compat. | | **v2 contract** | Current LLM output schema (Phase 2): `{summary, rewritten_bullets, new_bullets, key_achievements}`. Bullets keyed by `role_idx:bullet_idx`. | | **v3 contract** | Phase 3 prompt with liberal keyword policy + recruiter-pitch opener requirement. Same schema as v2. | | **Recruiter pitch** | First sentence of Professional Summary, format: *"Strong-fit candidate for at : years of directly applicable to <3 JD requirements>."* | | **Indexed bullets** | `[(role_idx, bullet_idx, role_name, bullet_text), ...]` tuples passed to the LLM so it can reference specific original bullets. | | **§§HEADER§§** / **§§META§§** | Internal markers in `_extract_experience_sections` for sub-section bold headers and italic meta lines (like "Scope:"). | | **JD noise filter** | Set of words `_JD_NOISE_WORDS` in `ats_scorer.py` that get dropped from keyword extraction (e.g. proven/solid/will/bachelor). | | **Postcondition** | `_assert_no_dump_footer` runs at end of every `_generate_resume` to catch banned patterns (Skills sections, dump footers). | | **Lemma matching** | Rules-based stemmer: `automated`/`automation`/`automate` all collapse to `automat`. Enables forgiving keyword matching. | | **Phrase matching** | Multi-word JD keywords match if all component lemmas appear within a 5-token sliding window in the resume. | | **Aggressive injection** | Trusts JD extractor's noise filter — injects every still-missing keyword (capped at 15) into the Summary's closing sentence. | --- End of context document. If anything in here contradicts current code, the code is the source of truth — but please update this doc when you finish your task.