logs

README.md

@@ -164,4 +164,34 @@ Requires a built vector index at `data/processed/index/`. See
 
 This app surfaces public salary statistics and the published text of EU
 Directive 2023/970. It is **not legal advice**. For binding interpretation,
-talk to your union or a lawyer.
+talk to your union or a lawyer.
+
+# Development log
+
+### Jun 9 — Setup
+Created the Hugging Face Space inside `build-small-hackathon`, joined the org, downloaded the EU Directive 2023/970 and the salary statistics PDF. Scoped v1 to Danish engineers under IDA (the engineer union) only; Djøf (white-collar jobs union) and France left as future work. Wired in ML-Intern, and Gradio Server as the core stack.
+
+### Jun 10 — Knowledge extraction
+Sent ML-Intern to parse both sources on HF Jobs (a10g-large GPU):
+- **EU Directive** — Nemotron Parse over rasterized pages → 24 pages, 366 elements, JSON-Schema validated. Caught and removed the `TableInsertionLogitsProcessor`, which was forcing tabular output and mangling prose recitals into fake tables.
+- **IDA + Djøf lønstatistik** — separate extractors (lattice for IDA, header-anchored for Djøf's borderless tables), normalized into typed records with derived experience years and citation-bearing RAG text. 266 non-salary rows dropped during cleanup.
+
+Both outputs published as private HF Datasets.
+
+### Jun 14 — Index and retrieval
+Embedded all chunks on Modal (local CPU too slow for the Nemotron 1B encoder), assembled a FAISS `IndexFlatIP` locally. Hit HF's 10MB non-LFS file ceiling — installed git-lfs and shipped the 20MB `index.faiss` as a pointer.
+
+### Jun 15 — RAG, frontend, chat
+Four major pieces in one push:
+1. **Dual-source Reciprocal Rank Fusion** — query the directive and IDA independently, fuse the ranked lists.
+2. **Typed salary layer** — `IdaSalaryRecord` schema, `Profile` / `Dashboard` models, wildcard matching, specificity ranking, reliability-tier gating, 2026 temporal projection.
+3. **Re-extracted IDA tables** with a panel-aware `pdfplumber` parser, recovering multi-axis data (region, gender × leadership, public position) that v1 had flattened — 559 records. This restored the gender pay-gap data, the core of the equal-pay feature. Biggest data win of the project.
+4. **Custom frontend** — hand-authored HTML/CSS/JS SPA served by `gradio.Server` (no Gradio Blocks at the root), profile-driven dashboard with hand-drawn SVG visuals, Branch A (employed) and Branch B (job-seeking) onboarding flows, contextual rights chat. Sage/ink Scandinavian palette.
+
+GPU warmup triggered on the first wizard answer so Tiny Aya is ready by the time the user reaches the chat. Added token streaming, then removed it for a cleaner single response.
+
+### Jun 15–16 — The number bug
+Salaries rendering ~10× too small. The small multilingual LLM was reading thousands separators as decimal points: first the Danish `47.010` → `4,701`, then the en-US `47,300` → `4,730` after switching format. Fix: bare integers with no separators in everything the model reads (`rag_text`, injected context), plus a "copy digits exactly" system instruction. Regenerated `chunks.jsonl` only — FAISS rows stayed aligned, so no re-embed needed.
+
+### Jun 16 — Submission
+Scrubbed a write token that had leaked into the git remote URL. Final README pass — motivation, IDA scope, Djøf and France as future work, track/sponsor/achievement tags, demo video, social links. Added a "coming soon: Djøf" dashboard banner. Last issue was a ZeroGPU container prestart-hook error on the Space — diagnosed as platform infrastructure (no code changes resolved it), fixed by a Factory reboot.