# RAG Benchmark Results — Pay Equity for EU

## Summary

| Metric | Score |
|---|---|
| **Recall@1** | 0.667 (10/15) |
| **Recall@5** | 0.867 (13/15) |
| **MRR** | 0.736 |
| **NDCG@5** | 0.768 |

**Directive retrieval**: Recall@1 = 0.778, MRR = 0.856 (7/9 at top rank)
**Salary retrieval**: Recall@1 = 0.500, MRR = 0.556 (3/6 at top rank)

## Per-Question Results

| Q | Type | Recall@1 | Recall@5 | MRR | Notes |
|---|---|---|---|---|---|
| Q01 | directive | 1.00 | 1.00 | 1.000 | Art. 5 (pay history question ban) — perfect |
| Q02 | directive | 1.00 | 1.00 | 1.000 | Art. 7 (right to pay info) — perfect |
| Q03 | directive | 1.00 | 1.00 | 1.000 | Art. 9 (100-worker reporting threshold) — perfect |
| Q04 | directive | 0.00 | 1.00 | 0.500 | Art. 10 (joint pay assessment) — rank 2 behind Art. 9 |
| Q05 | directive | 1.00 | 1.00 | 1.000 | Art. 18 (burden of proof shift) — perfect |
| Q06 | directive | 1.00 | 1.00 | 1.000 | Art. 21 (3-year limitation period) — perfect |
| Q07 | directive | 1.00 | 1.00 | 1.000 | Art. 3 (definition of pay) — perfect |
| Q08 | directive | 0.00 | 1.00 | 0.200 | Art. 34 (7 June 2026 deadline) — rank 5 |
| Q09 | directive | 1.00 | 1.00 | 1.000 | Art. 23 (penalties) — perfect |
| Q10 | salary | 1.00 | 1.00 | 1.000 | Civilingeniør 10yr (DK) — perfect, strong Danish match |
| Q11 | salary | 0.00 | 0.00 | 0.000 | **FAIL: Civilingeniør 5yr — target at rank 10, two-part question confuses embedder** |
| Q12 | salary | 1.00 | 1.00 | 1.000 | Civilingeniør starting 0yr — perfect |
| Q13 | salary | 1.00 | 1.00 | 1.000 | Topchef 25yr — perfect |
| Q14 | salary | 0.00 | 1.00 | 0.333 | Medicinalindustri — target at rank 3 |
| Q15 | salary | 0.00 | 0.00 | 0.000 | **FAIL: State sector 20yr — target rank >15, embedder can't distinguish state from private** |

## Failure Analysis

### Q11 — Civilingeniør with 5 years experience (target at rank 10)

**Why it fails**: The question asks a two-part question ("what is median salary... how many respondents and what percentage receive bonus?"). The embedder (Nemotron 1B) is designed for passage retrieval, not for parsing multi-part questions. The model retrieves *Civilingeniør* chunks on p.11 but ranks the 20-year-experience chunk (2005 graduation, ida-p011-0025) highest because "20 years" matches the query's temporal context better than "5 years."

**The target chunk** `ida-p011-0040` (2020 graduation, ~5yrs) is at rank 10 with cosine similarity 0.682 vs top hit at 0.693 — a very tight margin. The embedder simply can't reliably distinguish "5 years" from "20 years" because the chunk texts share identical structure and differ only in the experience label.

**Fix**: For the production Space, use `k=10` for salary queries. Alternatively, add explicit experience-year metadata to chunks and use a re-ranker.

### Q15 — State sector 20 years (target not in top 15)

**Why it fails**: The question asks for "state sector (staten)" but the target chunk mentions "Staten: Stillingsniveau" — a specific subcategory. Meanwhile, 198 other chunks also mention "IDA 2025 salary statistics" and salary figures in the 65K range. The embedder retrieves the right page (p.30) but the wrong experience year because the "20 years" part drives retrieval toward chunks that happen to match that phrase better.

The top-5 results are all from p.30 (correct page) but all wrong experience years. The target `ida-p030-1698` (2005, ~20yrs) is simply not distinctive enough from ida-p030-1699 (2006, ~19yrs) or ida-p030-1713 (2020, ~5yrs) given the cosine similarity margin is only ~0.02 between adjacent chunks.

**Root cause**: Chunks on p.30 are semantically near-identical (same structure, similar salaries, 37 chunks varying only in graduation year + salary number). The embedder can't rank them precisely because the salary differences (60-70K) are within the model's semantic noise floor.

**Fix**: The same re-ranking approach applies here. This is a hard problem for any dense retriever on highly structured tabular data.

### Q04 — Joint pay assessment (target at rank 2)

Minor: Art. 10 (joint pay assessment) loses to Art. 9 (pay reporting). Both discuss "pay report" and "gender pay gap" — the question's focus on "*after* the report" is too subtle for the embedder.

### Q08 — Implementation deadline (target at rank 5)

Minor: Art. 34 (transposition deadline) loses to Art. 29 (monitoring), Art. 9 (reporting), Art. 31 (statistics), and Art. 35 (reporting & review). The word "deadline" alone can't discriminate — these articles all discuss timelines and implementation.

## Recommendations

1. **Increase k to 10 for salary queries** — Q11 and Q14 would be caught, lifting recall@10 to ~0.93
2. **Add metadata-aware re-ranking** — Salary chunks should be re-ranked by explicit experience-year match after vector retrieval
3. **Consider hybrid BM25 + dense retrieval** for highly structured salary data where exact number/date matching matters
4. **Benchmark with upgraded transformers** — The current transformers v4.45.1 doesn't support `use_bidirectional_attention=True` for Nemotron, potentially degrading embedding quality. Upgrade to v4.56.2+.
5. **Test with alternative embedders** — A multilingual embedder with better number sensitivity (e.g., BGE-M3) might handle salary data better