# SolidPrivacy Scrub — Workpackages

This file translates `ROADMAP.md` into executable workpackages.

Use:

- `PROJECT_PROMPT.md` for full worker instructions and operating rules.
- `PROJECT_PROMPT_SHORT.md` for the compact ChatGPT Project Instructions version.
- `ROADMAP.md` for product direction and phase order.
- `WORKPACKAGES.md` for immediate execution planning and parallelization.
- `CHANGELOG.md` for implementation history.

---

## Mandatory worker start sequence

Every worker must start by reading, in order:

1. `PROJECT_PROMPT.md`
2. `ROADMAP.md`
3. `WORKPACKAGES.md`
4. `CHANGELOG.md`

If the active repository is not `solidprivacy-nl/scrub`, stop and report the mismatch.

Every worker must end with a handover summary and write that summary to:

```text
handover/workpackages/
```

Filename format:

```text
handover/workpackages/YYYYMMDD_HHMM_<workpackage_slug>.md
```

---

## Current execution principle

Avoid parallel edits to the same Streamlit UI patch area.

Parallel work is safe for:

- pure helper modules;
- tests;
- specifications;
- documentation;
- non-UI architecture work.

Parallel work is risky for:

- `presidio_streamlit.py`;
- `fix_streamlit_nested_expanders.py`;
- export/download UI blocks;
- shared replacement table flow.

UI integration should therefore happen sequentially.

---

## Completed prerequisite

### WP0 — v12.3 stabilization check

Status: completed by user verification.

Evidence:

- GitHub Actions tests green.
- GitHub to Hugging Face sync green.
- App reloaded successfully.
- pandas Index truth-value error gone.
- Simplified review table working.
- Technical details available in separate expander.

---

## Completed UI workpackages

### WP1 — v12.4 Review guidance text

Status: completed.

Outcome:

- Review workflow guidance is visible.
- Export semantics were not changed.

---

### WP2 — v12.5 Final review summary

Status: completed and formally closed after verification.

Implemented files:

- `review_summary.py`
- `tests/test_review_summary.py`
- `tests/test_review_summary_ui_patch.py`
- `fix_streamlit_nested_expanders.py`

Implemented behavior:

- The app shows `Eindcontrole vóór download` before the download/export section.
- The summary is advisory and does not change export/download semantics.

Outcome:

- v12.5 is complete.

---

### WP3 — v12.6 Export sanity checks

Status: completed and formally closed.

Implemented files:

- `export_sanity.py`
- `tests/test_export_sanity.py`
- `tests/test_export_sanity_ui_patch.py`
- `fix_streamlit_nested_expanders.py`

Implemented behavior:

- The app shows `Extra exportcontrole` near `Eindcontrole vóór download` before the download/export section.
- The warning block is advisory only.
- Downloads are not blocked.
- TXT, CSV, DOCX and PDF export behavior is not changed.

Outcome:

- v12.6 is closed.
- v12 Review UX line is complete from WP1 through WP3.

---

## Completed strategic workpackages

### WP4 — v13.0 Scrub Key specification and pure model

Status: completed.

Implemented files:

- `SCRUB_KEY_SPEC.md`
- `scrub_key.py`
- `tests/test_scrub_key.py`

Outcome:

- v13.0 model/spec is complete.

---

### WP4B / WP4B-FIX — v13.1 Scrub Key JSON export UI and mapping hotfix

Status: completed and app-verified.

Implemented files:

- `fix_streamlit_nested_expanders.py`
- `tests/test_scrub_key_ui_patch.py`

Implemented behavior:

- The app shows a `Scrub Key (JSON)` section in the download/export flow.
- The app shows a pseudonymization / reversibility warning.
- The app shows `Download Scrub Key (.json)`.
- The Scrub Key JSON download works after the mapping hotfix.

Outcome:

- v13.1 Scrub Key JSON export is complete.

---

### WP5 — v13.2 Scrub Key import/reload helper and tests

Status: completed.

Implemented files:

- `scrub_key_import.py`
- `tests/test_scrub_key_import.py`

Outcome:

- v13.2 helper layer was ready for UI integration and is now part of the completed v13.2 import/reload flow.

---

### WP6 — v13.2 Scrub Key import/reload UI integration

Status: completed and app-verified.

Implemented files:

- `fix_streamlit_nested_expanders.py`
- `tests/test_scrub_key_import_ui_patch.py`

Implemented behavior:

- Adds a `Scrub Key laden` section near the existing `Scrub Key (JSON)` export block.
- Allows upload of a Scrub Key `.json` file or pasted Scrub Key JSON.
- Validates the imported key using `build_scrub_key_import_result(...)` before loading.
- Shows pseudonymization/reversibility and local-protection warnings.
- Preserves the existing `Download Scrub Key (.json)` export block.

Outcome:

- v13.2 Scrub Key import/reload UI is completed, app-verified and closed.

---

### WP7A / WP7B / WP7B-FINAL — v13.3 Deterministic reinsert helper

Status: completed and formally closed after Actions/sync verification.

Implemented files:

- `scrub_key_reinsert.py`
- `tests/test_scrub_key_reinsert.py`

Implemented helper behavior:

- Added `detect_placeholders(text)`.
- Added `build_reinsert_mapping(scrub_key)`.
- Added `reinsert_from_scrub_key(text, scrub_key)`.
- Reuses existing `validate_scrub_key(...)` validation.
- Reports validation issues, mapping item count, active item count, excluded item count, replacement count, missing placeholders, unknown placeholders and duplicate placeholders.
- Reports `local_only=True`, `ai_processing=False` and `cloud_processing=False`.

Validation evidence:

- Local targeted validation: `PYTHONPATH=. pytest -q tests/test_scrub_key.py tests/test_scrub_key_import.py tests/test_scrub_key_reinsert.py` → 25 passed.
- Coordinator verification evidence confirmed Tests #106-#109 and Sync #120-#123 green.

Outcome:

- v13.3 deterministic reinsert helper is completed and formally closed.

---

### WP8 — v13.3 Deterministic reinsert UI planning

Status: implemented; reinsert UI implementation completed in WP8B and app-verified in WP8C.

Implemented files:

- `REINSERT_UI_SPEC.md`
- `WORKPACKAGES.md`
- `CHANGELOG.md`
- `handover/workpackages/20260607_1900_v13_3_reinsert_ui_planning.md`

Outcome:

- v13.3 deterministic reinsert UI was planned before implementation.

---

### WP8B / WP8C — v13.3 Deterministic reinsert UI implementation and app verification closeout

Status: completed and app-verified after Actions/sync verification.

Implemented files:

- `fix_streamlit_nested_expanders.py`
- `tests/test_scrub_key_reinsert_ui_patch.py`
- `tests/test_scrub_key_ui_patch.py`
- `WORKPACKAGES.md`
- `CHANGELOG.md`
- `handover/workpackages/20260607_1915_v13_3_reinsert_ui_implementation.md`
- `handover/workpackages/20260607_1930_v13_3_reinsert_ui_app_closeout.md`

Implemented behavior:

- Adds `Originele waarden terugzetten` after the existing `Scrub Key laden` / Scrub Key area.
- Lets the user paste scrubbed or AI-generated text into `Plak hier de tekst waarin u originele waarden lokaal wilt terugzetten`.
- Requires explicit button action: `Zet originele waarden lokaal terug`.
- Calls `reinsert_from_scrub_key(reinsert_input_text, active_reinsert_scrub_key)`.
- Shows `Herstelde tekst`.
- Adds `Download herstelde tekst (.txt)`.
- Shows `Controleverslag terugzetten` with mapping item count, active item count, excluded item count, replacement count, placeholders not found, unknown placeholders, duplicate placeholders, validation issues, local-only status, no-AI status and no-cloud status.
- Shows warning that restored text can again contain personal or confidential information and must be reviewed before sharing.
- Uses the imported validated Scrub Key from session state when available, otherwise the current Scrub Key built from reviewed replacement rows.

Validation status:

- Local targeted validation recorded in WP8B:
  - `PYTHONPATH=. pytest -q tests/test_scrub_key.py tests/test_scrub_key_import.py tests/test_scrub_key_reinsert.py tests/test_scrub_key_reinsert_ui_patch.py tests/test_scrub_key_import_ui_patch.py tests/test_scrub_key_ui_patch.py` → 57 passed.
- GitHub Actions: green based on coordinator evidence:
  - Tests #120 green — commit `7725182`.
  - Tests #121 green — commit `84f5312`.
  - Tests #122 green — commit `1a8e87e`.
- Hugging Face sync: green based on coordinator evidence:
  - Sync to Hugging Face Space #134 green — commit `7725182`.
  - Sync to Hugging Face Space #135 green — commit `84f5312`.
  - Sync to Hugging Face Space #136 green — commit `1a8e87e`.
- App verification: confirmed by coordinator/user.

App verification evidence:

- `Originele waarden terugzetten` is visible.
- Warning about sensitive/confidential information is visible.
- Local-only / no-AI / no-cloud text is visible.
- Text input works.
- Button `Zet originele waarden lokaal terug` works.
- Reinsert works with a valid Scrub Key.
- Placeholders are correctly restored.
- Result message appears: `37 waarde(n) lokaal teruggezet.`
- `Herstelde tekst` appears with restored original values.

Closeout notes:

- Restored output may contain sensitive/confidential information again.
- Existing Scrub Key export remains available.
- Existing Scrub Key import/reload remains available.
- Existing TXT, CSV, DOCX and PDF scrubbed downloads remain available based on prior verification and no intentional export changes.
- No AI calls were added.
- No cloud processing was added.
- No automatic document rehydration was added.
- No DOCX/PDF reinsert was added.
- No existing scrubbed export/download behavior was intentionally changed.

Boundaries preserved:

- No code files changed in WP8C closeout.
- No direct edit to `presidio_streamlit.py`.
- No AI calls.
- No cloud processing.
- No automatic document rehydration.
- No DOCX/PDF reinsert added.
- No TXT, CSV, DOCX or PDF scrubbed export behavior changed.
- No Scrub Key JSON export behavior intentionally changed.
- No Scrub Key import/reload behavior intentionally changed except storing the validated imported key in session state for reinsert use.
- No silent overwrite of existing review rows.
- No secrets, tokens or real personal data stored.

Outcome:

- v13.3 deterministic local reinsert UI is completed, app-verified and formally closed.

---

### WP9 — AI-output / document reinsert workflow UX and architecture review

Status: completed; review-only workpackage.

Added files:

- `AI_OUTPUT_REINSERT_WORKFLOW_REVIEW.md`
- `handover/workpackages/20260608_0000_ai_output_reinsert_workflow_review.md`

Changed files:

- `WORKPACKAGES.md`
- `CHANGELOG.md`

Outcome:

- Pasted-text reinsert remains a safe baseline and fallback, but is not sufficient as the final legal-document workflow.
- The first three obvious ideas were explicitly challenged:
  - keeping only pasted-text reinsert;
  - immediately adding broad PDF/DOCX upload reinsert;
  - placing anonymize and de-anonymize in one combined long screen.
- Recommended UX direction is a two-mode interface:
  - `Anonimiseren`;
  - `Originele waarden terugzetten`.
- Recommended implementation sequence is phased:
  - keep pasted text;
  - add TXT upload/download reinsert;
  - add DOCX reinsert helper and tests;
  - add DOCX reinsert UI;
  - investigate PDF text extraction only;
  - consider PDF output only after reliability review.
- Recommended architecture remains local-only, deterministic and helper-first.
- No code files, UI behavior, export/download behavior, Scrub Key export/import behavior, AI calls or cloud processing were changed.

Validation:

- Tests: not applicable; planning/review-only workpackage.
- App verification: not applicable; no UI behavior changed.

---

## Active / next recommended workpackage

### WP10 — v13.4 TXT/DOCX reinsert foundation helper and tests

Status: recommended next implementation workpackage; not started here.

Goal:

- Prepare document-level reinsert without changing UI first.
- Add pure helper/test foundation for TXT and DOCX reinsert.
- Keep the existing pasted-text reinsert path available.
- Keep PDF out of implementation scope except for later research.

Recommended scope:

- Add pure TXT file/text reinsert helper behavior if useful beyond existing text helper.
- Add a pure DOCX placeholder replacement helper or detailed helper contract.
- Add synthetic tests for TXT and DOCX placeholder replacement.
- Reuse the existing `reinsert_from_scrub_key(...)` mapping behavior.
- Return audit summary fields for document-level reinsert.
- Do not edit UI in the first helper package.
- Do not edit `fix_streamlit_nested_expanders.py` or `presidio_streamlit.py`.
- Do not change existing scrubbed export/download semantics.
- Do not add AI calls.
- Do not add cloud processing.
- Do not add PDF reinsert implementation.
- Do not store secrets, tokens or real personal data.

Recommended later UI workpackage after WP10:

```text
WP11 — v13.5 Two-mode reinsert UI planning
```

Purpose:

- Plan the future `Anonimiseren` / `Originele waarden terugzetten` mode structure before Streamlit UI changes.
- Avoid parallel edits to the current patch-based UI flow.

---

## Recommended execution order

1. Start WP10 as helper/test-only document-level reinsert foundation.
2. Keep PDF full reinsert out of scope.
3. After helper validation, plan the two-mode UI separately.
4. Only then implement TXT/DOCX reinsert UI sequentially.
5. Keep AI/cloud behavior out unless explicitly approved.
6. Preserve export/download and Scrub Key import/export semantics.