JAA-ATS-Tool / README.md
saitejatirunagari's picture
feat: non-destructive tailoring + reliable PDF + persistent side panel + history (Phase 8)
8155d78
|
Raw
History Blame
22.1 kB
---
title: JAA ATS Tool
emoji: πŸ€–
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
license: mit
---
# Job Automation Agent β€” PM Edition
Automated Product Manager job search, AI-powered assessment, ATS-optimized resume generation, and Google Sheets reporting β€” all in one pipeline.
---
## What It Does
| Step | What Happens |
|------|--------------|
| 1 | Parses your PDF resume (Kimi-K2.6, ~5s) |
| 2 | Scrapes PM-only jobs from **LinkedIn, Indeed, Glassdoor, Remotive, WeWorkRemotely** β€” ALL selected roles Γ— locations, last N days |
| 3 | Filters non-PM roles at scrape time; dedup by URL + (title, company) + 30-day SQLite history; platform-level total cap |
| 4 | Fetches FULL job descriptions in bulk (parallel HTTP / single browser session) |
| 5 | Assesses ALL jobs using 7 parallel AI models (10-model pool via NVIDIA API) |
| 6 | Generates ATS-optimized resumes for ALL jobs β€” DOCX **and PDF**, After β‰₯ Before guaranteed, target 95% |
| 7 | Writes everything to your Google Sheet with direct job links |
| 8 | Saves local Excel report + resumes in `data/output/resumes/YYYY-MM-DD/` |
| 9 | Saves run to history β€” reload past runs in one click from the History panel |
---
## Quick Start
```powershell
# 1. Install dependencies
pip install -r requirements.txt
playwright install chromium
# 2. Copy and fill in your API keys
copy .env.example .env
# Edit .env with your NVIDIA_API_KEY, GOOGLE_SHEET_ID, etc.
# 3. Place your resume PDF
# Copy your resume to: data/resume/resume.pdf
# 4. Connect Google (one-time browser login)
python connect_google.py
# 5. Run a test (5 jobs)
# In config.py: ASSESSMENT["test_mode"] = True
python main.py
# 6. Run full production mode (all 100+ PM jobs)
# In config.py: ASSESSMENT["test_mode"] = False
python main.py
# 7. Or use the Streamlit UI
streamlit run ui.py
```
---
## ever-jobs Integration (160+ Platforms)
This project integrates the [ever-jobs](https://github.com/ever-jobs/ever-jobs) NestJS service, which provides REST API access to 160+ job board scrapers.
### Prerequisites
- **Docker Desktop** (preferred): [Install Docker Desktop](https://docs.docker.com/desktop/install/windows-install/)
- OR **Node.js 24.x** for npm subprocess fallback
### Setup (one-time)
```bash
# Clone ever-jobs to vendor/ directory (done automatically during setup)
git clone https://github.com/ever-jobs/ever-jobs.git vendor/ever-jobs --depth=1
# Start via Docker (preferred)
cd vendor/ever-jobs && docker compose up -d
# Verify API is running
curl http://localhost:3001/health
# OR from Python:
python -c "from src.ever_jobs_bridge.server import is_running; print(is_running())"
```
### Automatic Startup
The pipeline calls `ensure_running()` automatically before any ever-jobs platforms are scraped. It tries Docker first, falls back to `npm run start` if Docker is unavailable.
### Platform Selection
The UI has a single **"🌐 Job Platforms"** section with three groups. Selecting LinkedIn, Indeed, Glassdoor, Remotive, WeWorkRemotely, or Naukri uses their dedicated high-quality scrapers; all other platforms go through the ever-jobs REST API.
| Group | Count | Description |
|-------|-------|-------------|
| Search Boards | 94 | General job boards. India-relevant defaults pre-selected. |
| ATS Platforms | 37 | Greenhouse, Lever, Workday etc. β€” companies post jobs here. NOT resume scoring. |
| Company Pages | 39 | Direct career pages (Flipkart, Swiggy, Amazon, Google, etc.) |
**India default platforms:** linkedin, naukri, indeed, glassdoor, google, bdjobs, internshala, bayt, iimjobs, foundit
### ATS Clarification
> **Important:** "ATS" in ever-jobs means Applicant Tracking System **PLATFORMS**
> (tools companies use to post jobs, like Greenhouse or Lever).
> It does **NOT** mean ATS resume scoring.
>
> Our ATS resume scoring (`src/ats_scorer.py`) uses a 70% JD keyword match +
> 30% resume quality hybrid and is **NOT changed** by this integration.
### Performance Notes
| Selection | Expected Time |
|-----------|---------------|
| Default 10 platforms | ~1–3 minutes |
| 30 platforms | ~3–5 minutes |
| 100+ platforms | 5–10 minutes (warning shown in UI) |
Playwright-based scrapers (some company pages) are inherently slower than API-based boards.
### Cross-Platform Deduplication
In addition to URL-based dedup, a **content fingerprint** (SHA-256 of normalized title+company) catches cross-platform duplicates. For example, the same "Product Manager at Google" posting on LinkedIn AND Greenhouse (different URLs) is detected and deduplicated.
---
## Project Structure
```
Job Automation Agent/
β”œβ”€β”€ main.py # Main pipeline (6-step orchestrator)
β”œβ”€β”€ config.py # All configuration β€” models, platforms, ATS settings
β”œβ”€β”€ ui.py # Streamlit UI (4 tabs: Search, Results, Job Details, Research)
β”œβ”€β”€ connect_google.py # One-time Google OAuth setup
β”œβ”€β”€ setup_google.py # Service account alternative
β”œβ”€β”€ requirements.txt
β”œβ”€β”€ .env # API keys (never commit)
β”œβ”€β”€ .env.example # Template for .env
β”‚
β”œβ”€β”€ src/
β”‚ β”œβ”€β”€ resume_parser.py # PDF β†’ plain text (pdfplumber)
β”‚ β”œβ”€β”€ llm_client.py # GLM 5.1 wrapper (profile extract, resume customize)
β”‚ β”œβ”€β”€ model_pool.py # 10-model parallel AI pool (NVIDIA API)
β”‚ β”œβ”€β”€ job_assessor.py # Phase 1 keyword scoring + Phase 2 LLM assessment
β”‚ β”œβ”€β”€ job_history.py # SQLite dedup store (data/job_history.db)
β”‚ β”œβ”€β”€ ats_scorer.py # Hybrid ATS scoring (70% JD match + 30% quality)
β”‚ β”œβ”€β”€ resume_customizer.py # LLM-tailored DOCX resume generator
β”‚ β”œβ”€β”€ gsheets.py # Google Sheets + Drive upload
β”‚ β”œβ”€β”€ excel_reporter.py # Local Excel report
β”‚ β”œβ”€β”€ odysseus_llm_core.py # Deep research LLM core
β”‚ β”‚
β”‚ β”œβ”€β”€ ever_jobs_bridge/ # ever-jobs NestJS REST API adapter
β”‚ β”‚ β”œβ”€β”€ __init__.py # Package init
β”‚ β”‚ β”œβ”€β”€ server.py # Docker/npm server lifecycle (start/stop/health)
β”‚ β”‚ β”œβ”€β”€ client.py # HTTP client for POST /api/jobs/search
β”‚ β”‚ β”œβ”€β”€ mapper.py # IJob JSON β†’ Job dataclass mapper
β”‚ β”‚ └── platforms.py # 170 platform catalog with group metadata
β”‚ β”‚
β”‚ β”œβ”€β”€ scrapers/
β”‚ β”‚ β”œβ”€β”€ base.py # Job dataclass + BaseScraper + is_pm_role() filter
β”‚ β”‚ β”œβ”€β”€ linkedin.py # LinkedIn scraper (requests + BeautifulSoup)
β”‚ β”‚ β”œβ”€β”€ indeed.py # Indeed scraper (Playwright for JS rendering)
β”‚ β”‚ β”œβ”€β”€ glassdoor.py # Glassdoor scraper (Playwright)
β”‚ β”‚ β”œβ”€β”€ naukri.py # Naukri (disabled β€” blocked by Akamai)
β”‚ β”‚ └── ever_jobs.py # EverJobsScraper (REST adapter for 160+ platforms)
β”‚ β”‚
β”‚ └── research/
β”‚ β”œβ”€β”€ deep_researcher.py # Odysseus IterResearch engine (Thinkβ†’Searchβ†’Extractβ†’Synthesize)
β”‚ └── search.py # DuckDuckGo + Bing fallback, 12h cache
β”‚
└── data/
β”œβ”€β”€ resume/resume.pdf # Your resume (add this)
β”œβ”€β”€ job_history.db # Dedup SQLite DB (auto-created)
β”œβ”€β”€ research_cache/ # 12h DuckDuckGo result cache
└── output/
β”œβ”€β”€ resumes/ # Generated DOCX resumes (Company_JobTitle.docx)
└── reports/ # Excel reports
```
---
## Configuration (`config.py`)
### AI Models (10-model pool via NVIDIA API)
| Model | Speed | Phase 2 | Notes |
|-------|-------|---------|-------|
| Kimi-K2.6 | ~5s/batch | βœ… | Fastest, handles most work |
| Step-3.7-Flash | ~8-35s | βœ… | |
| Qwen3.5-397b | ~9s | βœ… | |
| Qwen3.5-122b-v2 | ~12s | βœ… | |
| GPT-OSS-120b | ~11s | βœ… | |
| Qwen3.5-122b | ~40s | βœ… | |
| DeepSeek-v4-Pro | ~42s | βœ… | |
| DeepSeek-v4-Flash | ~229s | ❌ | Too slow for phase 2 |
| GLM-5.1 | ~234s | ❌ | Used for resume parsing only |
| MiniMax-M2.7 | ~908s | ❌ | Blocked/rate-limited |
### Key Settings
```python
ASSESSMENT = {
"min_score_for_llm_resume": 6, # LLM-tailored resume for score >= this
"generate_all_resumes": True, # Template resume for ALL PM jobs
"max_llm_resumes": 30, # Max LLM resumes per run
"dedup_days": 30, # Skip jobs seen in last 30 days
"test_mode": True, # ← Set False for full production run
"test_jobs_limit": 5, # Max jobs in test mode
}
```
---
## ATS Scoring Method
Hybrid scoring: **70% JD Match + 30% Resume Quality**
- **JD Match (70%)**: Extract keywords FROM the specific job description β†’ match against resume using word-boundary regex (`(?<!\w)kw(?!\w)`) β€” same approach as Resume-Matcher
- **Resume Quality (30%)**: 6-factor structural analysis (measurable achievements, contact info, education, etc.)
- **Iterative optimization**: Up to 3 LLM attempts to push score up, gap report fed back each iteration
- **Fallback**: `_inject_missing_keywords()` weaves still-missing terms naturally
### Keyword extraction is JD-driven, skills-only, and honest (industry standard)
- Keywords are extracted **from each JD itself** (not a fixed stored list), but
**skills-only**: a term is kept only if it's a recognised skill/tool/method/
domain/soft-skill in our gazetteer. Prose nouns (Goals, Authority, Enterprise,
Productivity…) can never become "keywords" β€” exactly how real checkers like
Jobalytics work (they match against a curated skills list, not every word).
- **Dedicated, categorized SKILLS section** (Tools & Analytics / Methodologies /
Domains / Core Competencies) β€” the #1 ATS keyword vehicle, placed after the
summary, parsed and counted by Jobalytics/Resume Worded.
- **Quality over quantity**: cover the JD's real skills (~20–26), each 1–3Γ—,
across the skills section + summary + woven bullets. Best practice is 15–25
keywords at 60–80% coverage β€” NOT 100 stuffed terms.
- **No cap, distributed smartly.** Every meaningful missing keyword is kept β€”
woven into relevant bullets first (the ideal, never-penalised place), then any
remainder spread across several short sentences (each its own paragraph, ≀10
items). A single long comma-dump is *deliberately not* produced:
`_strip_keyword_spam` removes any 15+-separator line before scoring, exactly as
real ATS checkers and recruiters discount stuffing β€” so distributing keeps them
all counted without looking like spam.
- **Buzzwords removed.** Vague abstractions (innovation, solutions, tools,
leadership, leverage, scalable…) are never injected β€” real checkers like Resume
Worded penalise them.
- **Honest expectation:** 90%+ is reached on JDs that genuinely fit the
candidate's background. Out-of-domain JDs score honestly lower β€” that reflects
reality (and matches third-party checkers), rather than a faked number.
- The displayed score is a **conservative estimate** (leans low vs our raw
internal coverage); always verify the final resume on Jobalytics/Simplify.
**Benchmark results** (correctly differentiates β€” full resume + capable LLM):
- In-domain PM JDs (Airtel, generic PM) β†’ ~83–87
- Out-of-domain JDs (SecOps, lending) β†’ ~60–70, honestly
---
## Chrome Extension + JSON API
Apply one job at a time in the browser (no bulk scraping β†’ no IP-blocking). The
HF Space serves BOTH the Streamlit app and a JSON API from one codebase β€” so the
**score is identical** to the web app (proven by `tests/test_api_parity.py`).
**Setup:** load `extension/` unpacked β†’ Options β†’ **paste your resume LaTeX** (or
upload a PDF) + set API URL (`https://<your-space>.hf.space`) + `API_SECRET_TOKEN`.
Then open any job β†’ **Run** β†’ tailored resume download.
**Resume LaTeX (recommended):** paste your resume's LaTeX source in Options. When
present it is **prioritised over the PDF** β€” we extract its text for keyword
matching, inject the honestly-includable JD keywords **distributed across a Summary
sentence, Experience bullets, and one compact competencies line** (never a dump),
and compile it to a **clean PDF** (Tectonic on the Space) you download directly
(plus the modified `.tex`). This fixes the generated-PDF design issues because the
layout is your own LaTeX. Anti-faking is identical: certs, seniority, employers,
and specialised hands-on engineering terms are never injected.
Verify: `python scripts/verify_latex_resume.py`.
**Recruiter-grade placement + resilient Run (Phase 7, v1.4.0):** JD extraction now
strips page/extension UI chrome (Simplify/Jobalytics overlays) so junk like "Show
Match Details" / "People Clicked Apply" never becomes a keyword; the DOCX Skills
section is one capped line per category (18-28 items, no repeated "Core
Competencies:" dumps) with surplus keywords woven into Summary + Experience; and
clicking **Run** now survives the popup closing β€” the background owns the run and
the popup restores the spinner/result on reopen. Verify:
`python scripts/verify_clean_jd_extraction.py`, `verify_skills_distribution.py`,
`verify_resilient_run.py`.
**Trust + usability (Phase 8, v1.5.0):**
- **Download a real PDF, not just DOCX.** The DOCX path now always renders a PDF
sidecar (reportlab on Linux/HF) so the PDF button works even without LaTeX; the
LaTeX path compiles a clean PDF via Tectonic and, if compilation fails, tells you
why (no engine vs LaTeX error) while still offering the `.tex`.
- **Your resume is preserved β€” non-destructive tailoring is the default.** Role
titles, companies, dates, and your existing bullets are kept VERBATIM; keywords
are added only in the Professional Summary and as ≀3 appended lines at the end of
each experience. Nothing you wrote is renamed or rewritten.
- **Persistent left side panel (Jobalytics-style).** The UI is now a collapsible
panel docked to the left of the page instead of a popup that vanishes β€” click
**Run**, then click anywhere on the page; the run keeps going and stays visible.
- **History.** A History list in the panel shows your previous generations per job
(title/company/time/scores/status); click any entry to restore it and re-download
DOCX/PDF/.tex. Verify: `python scripts/verify_pdf_download.py`,
`verify_non_destructive.py`, `verify_side_panel.py`, `verify_history.py`.
**API endpoints** (`api_server.py`, port 7860, `X-Api-Token` header):
- `GET /api/health` β†’ `{status: ok}`
- `POST /api/generate` β€” multipart `jd_text` + (`resume_latex` **or** `resume` PDF;
LaTeX wins) β†’ tailored resume. PDF path returns base64 DOCX/PDF; LaTeX path
returns compiled `pdf_b64` + `tex_b64` + external coverage report.
- `POST /api/repair-with-feedback` β€” **External ATS Feedback Repair Mode**: paste
Jobalytics/Simplify feedback (score + missing keywords); every missing keyword
is risk-classified by `candidate_fit` and woven *honestly* into bullets/skills
(or injected into your LaTeX when `resume_latex` is supplied).
**Feedback-repair honesty rules** (never relaxed):
- LOW β†’ auto-weave; MEDIUM β†’ auto-weave + review flag; **HIGH β†’ not added,
returned as "needs your confirmation"; BLOCKED β†’ excluded.** No fake
skills/seniority/employers/degrees/certs.
- Scores come from the re-parsed exported file; READY needs internal AND
independent β‰₯ 90 AND ATS readability β‰₯ 90 AND parse validation.
**Statuses:** `READY_95_EXTERNAL_ALIGNED` (β‰₯90 both + pasted gaps mostly resolved)
Β· `READY_90_PLUS_REVIEW_RECOMMENDED` (medium-risk added) Β· `NEEDS_USER_INPUT`
(95 needs high-risk terms you must confirm) Β· `NOT_ELIGIBLE_LOW_FIT` (can't reach
90/95 without fabrication).
Verify: `PYTHONPATH=. python scripts/verify_feedback_repair.py` (deterministic;
proves blocked/high terms never enter the resume).
### Maximum ATS Mode (User-Confirmed Skill Expansion)
For your target role family (Product / Product Manager / AI Product Manager /
SaaS / B2B), the uploaded resume is treated as an **incomplete base profile**.
Enable **Maximum ATS Mode** (extension toggle, on by default; API field
`maximum_ats_mode=1`) to treat normal PM/Product/AI/agile vocabulary β€” *AI, ML,
generative AI, prompt design, model evaluation, QA, experimentation, data-driven,
analytics, roadmap, product strategy, product ownership, agile, scrum, user
stories, acceptance criteria, PRD, stakeholder management, SaaS, B2B, enterprise
platform, product discovery, go-to-market, retention, growth*, etc. β€” as
**user-confirmed / interview-supportable** and aggressively place them across
Summary, Skills, Key Achievements and Experience. It targets **95** external ATS
and treats **anything below 90** for in-family roles as a failure state to keep
repairing (unless the only gaps are genuinely blocked).
**Still never fabricated** (hard boundaries, unchanged): degrees, certifications/
licenses, employers, titles, years/seniority, regulated credentials, and
specialized hands-on engineering/security tools (those surface as **"needs your
confirmation"**, with a one-click **Confirm & regenerate**).
- Both endpoints accept `maximum_ats_mode` / `user_confirmed_expansion`,
`confirmed_terms`, `target_external_score` (default 95). Backward compatible.
- Responses include a rich **coverage report** (per keyword: category, risk,
disposition, the resume section it was placed in, and why anything was
excluded), `still_missing_repairable`, and a plain-English `below_target_explanation`.
- Confirmed expansion terms persist to the **Candidate Vault** (`user_confirmed`)
so future resumes treat them as safe.
**Maximum-ATS statuses:** `READY_MAX_ATS_95_PLUS` (gates pass + external coverage
β‰₯ 95) Β· `READY_90_PLUS_EXTERNAL_ALIGNED` (gates pass + coverage β‰₯ 90) Β·
`BELOW_TARGET_REPAIRABLE` (below target, remaining gaps LOW/MEDIUM β†’ keep
repairing) Β· `NEEDS_USER_CONFIRMATION` (only high-risk-but-supportable terms left).
**External-coverage is the success signal (not the internal score).** Our internal
scorer uses a narrow taxonomy (β†’ 90%+ easily); external checkers extract a broad
40-46 term set (β†’ can be ~54% on the same resume). So in Maximum ATS Mode the
system builds a broad Jobalytics-style **expected** set (`src/external_ats.py`),
**physically guarantees** every *includable* term into the exported DOCX (Skills
verbatim + woven into Experience bullets; HIGH/BLOCKED stay gated), re-parses the
file, and **measures coverage from the export**. Status is then `READY_MAX_ATS_95_PLUS`
(β‰₯95) / `READY_90_PLUS_EXTERNAL_ALIGNED` (β‰₯90) / `BELOW_TARGET_REPAIRABLE` β€” an
internal-96 / external-54 result is treated as a bug, never "done". Both endpoints
return `external_coverage` + a per-term `coverage_report` (keyword / found in export
/ section / why-missing) and log `maximum_ats_mode`.
Verify (deterministic, no keys):
- `PYTHONPATH=. python scripts/verify_maximum_ats.py` β€” PM/AI terms become
user-confirmed; certs/seniority/engineering stay gated; coverage improves.
- `PYTHONPATH=. python scripts/verify_max_ats_coverage.py` β€” the 26/46 live-failure
regression: exported DOCX covers β‰₯90% of includable PM terms; credentials/fake
seniority excluded; status driven by external coverage.
---
## Google Sheet Columns
| Column | Description |
|--------|-------------|
| Batch Date | When the run happened |
| Rank | Score rank within this batch |
| Job Title / Company / Location | Job details |
| Platform | LinkedIn / Indeed / Glassdoor |
| Relevance Score | AI assessment (1–10) |
| ATS Before (%) | ATS score on original resume |
| ATS After (%) | ATS score on tailored resume |
| ATS Improvement | After βˆ’ Before |
| Resume Quality | Structural quality score |
| Priority | High / Medium / Low |
| Matching / Missing Skills | Gap analysis |
| AI Recommendation | LLM reasoning |
| Apply Link | Direct job URL (clickable) |
| Resume Link | Google Drive link to tailored resume |
| Application Status | Dropdown: Not Applied β†’ Offer |
---
## Environment Variables (`.env`)
```
NVIDIA_API_KEY=nvapi-... # GLM 5.1 + primary key
NVIDIA_API_KEY_2=nvapi-... # DeepSeek-v4-Pro, MiniMax
NVIDIA_API_KEY_3=nvapi-... # Kimi-K2.6
NVIDIA_API_KEY_4=nvapi-... # Qwen3.5-122b
NVIDIA_API_KEY_5=nvapi-... # GPT-OSS-120b
NVIDIA_API_KEY_6=nvapi-... # DeepSeek-v4-Flash
NVIDIA_API_KEY_7=nvapi-... # Qwen3.5-397b, Qwen3.5-122b-v2
NVIDIA_API_KEY_8=nvapi-... # Step-3.7-Flash
GOOGLE_SHEET_ID=1Ehxt3eo... # Your Google Sheet ID
RESUME_PATH=data/resume/resume.pdf
```
---
## Logging & Debugging
Every pipeline run writes a timestamped log to `data/logs/run_YYYY-MM-DD_HH-MM-SS.log`.
To diagnose failures:
1. Run a search from the UI
2. Switch to the **πŸ“‹ Logs** tab
3. Errors show in red, warnings in yellow
4. Use **Download Full Log File** to share or inspect offline
5. Previous runs are also listed in the selector
The log captures:
- Every scrape attempt (role, location, raw result count)
- Full Python tracebacks on any exception
- All `print()` output from scrapers and LLM clients
- Playwright browser output
---
## Known Issues / Pending
| Issue | Status | Notes |
|-------|--------|-------|
| Google Drive upload `'Client' object has no attribute 'auth'` | Pending fix | gspread auth method mismatch |
| LLM resumes = 0 (GLM timeout during customization) | Pending fix | Switch to Kimi/Step for resume generation |
| Naukri blocked by Akamai | Permanent skip | Returns 406 / "Access Denied" with Playwright |
| Google OAuth "Access blocked" | Fixed | Add email as test user at GCP console |
---
## Running the UI
```powershell
streamlit run ui.py
# Opens at http://localhost:8501
# Tabs:
# 1. Search β€” configure and run the full pipeline
# 2. Results β€” view all assessed jobs with scores
# 3. Job Details β€” expand any job for full AI breakdown
# 4. Deep Research β€” Odysseus engine to research companies
```