CHANGELOG.md

# Changelog — SolidPrivacy Scrub

This changelog records meaningful product, architecture, workflow and recognizer changes for the Scrub legal document scrubber.

Conventions:

- Keep this file explicit and human-readable.
- Group changes by project phase / version.
- Record what changed, why it changed, and what is intentionally not changed.
- Do not use this as a substitute for tests; every recognizer hardening step should also add or update regression tests.

---

## Current development rule

From v10 onward, recognizer work follows this order:

1. Add or update synthetic regression cases.
2. Add or update tests.
3. Change recognizer / scanner logic.
4. Verify GitHub Actions tests are green.
5. Let GitHub sync to Hugging Face automatically.
6. Test the app in Hugging Face.

For UI/UX-only work, prefer pure helper modules and tests before touching Streamlit UI flow.

---

## v13.3 — Deterministic reinsert UI implementation

Status: implemented; awaiting GitHub Actions, Hugging Face sync and app verification.

Purpose:

- Add the deterministic local reinsert UI described in `REINSERT_UI_SPEC.md`.
- Let the user paste scrubbed or AI-generated text and locally restore mapped placeholders using a validated Scrub Key.
- Show restored text, an audit summary and a `.txt` download for restored text.
- Keep the step local and deterministic, with no AI calls and no cloud processing.

Files added or changed:

- `fix_streamlit_nested_expanders.py`
- `tests/test_scrub_key_reinsert_ui_patch.py`
- `tests/test_scrub_key_ui_patch.py`
- `WORKPACKAGES.md`
- `CHANGELOG.md`
- `handover/workpackages/20260607_1915_v13_3_reinsert_ui_implementation.md`

Main changes:

- Added import/wiring for the verified helper:
  - `from scrub_key_reinsert import reinsert_from_scrub_key`.
- Added the section `Originele waarden terugzetten` after the existing Scrub Key import/reload area.
- Added the input label `Plak hier de tekst waarin u originele waarden lokaal wilt terugzetten`.
- Added the explicit action button `Zet originele waarden lokaal terug`.
- Added local helper call:
  - `reinsert_from_scrub_key(reinsert_input_text, active_reinsert_scrub_key)`.
- Added output label `Herstelde tekst`.
- Added `Download herstelde tekst (.txt)` with `text/plain` output.
- Added `Controleverslag terugzetten` audit summary.
- Added audit rendering for item count, active item count, excluded item count, replacement count, placeholders not found, unknown placeholders, duplicate placeholders, validation issues, local-only status, AI-processing status and cloud-processing status.
- Added visible warning that restored text may again contain personal or confidential information and must be reviewed before sharing.
- Added local/no-AI/no-cloud wording.
- Stores a successfully imported Scrub Key in `st.session_state["active_scrub_key"]` for local reinsert use.
- Falls back to the currently built Scrub Key from reviewed replacement rows if no imported key is active.

Testing and validation:

- Added `tests/test_scrub_key_reinsert_ui_patch.py`.
- Updated `tests/test_scrub_key_ui_patch.py` so it no longer forbids the intentionally added reinsert flow, while still guarding no-AI, no-cloud and no-automatic-document-rehydration boundaries.
- Local targeted validation on a reconstructed subset passed:
  - `PYTHONPATH=. pytest -q tests/test_scrub_key.py tests/test_scrub_key_import.py tests/test_scrub_key_reinsert.py tests/test_scrub_key_reinsert_ui_patch.py tests/test_scrub_key_import_ui_patch.py tests/test_scrub_key_ui_patch.py` → 57 passed.
- Full repository test suite was not run from this connector environment.
- GitHub Actions are pending for WP8B commits.
- Hugging Face sync is pending for WP8B commits.
- App verification is pending because UI behavior changed.

Intentionally not changed:

- No direct edit to `presidio_streamlit.py`.
- No edit to `scrub_key_reinsert.py`.
- No edit to `scrub_key.py`.
- No edit to `scrub_key_import.py`.
- No AI calls.
- No cloud processing.
- No automatic document rehydration.
- No DOCX/PDF reinsert.
- No TXT, CSV, DOCX or PDF scrubbed export behavior changed.
- No Scrub Key JSON export behavior intentionally changed.
- No Scrub Key import/reload behavior intentionally changed except storing the validated imported key in session state for reinsert use.
- No silent overwrite of existing review rows.
- No secrets, tokens or real personal data.

Outcome:

- v13.3 deterministic local reinsert UI is implemented and awaits external verification and app testing.

---

## v13.3 — Deterministic reinsert UI planning

Status: implemented; reinsert UI implementation completed in WP8B and awaits verification.

Purpose:

- Plan deterministic local reinsert UI before changing Streamlit UI code.
- Define where the UI should appear, what labels it should use, what state it should rely on, and which warnings and audit fields it must show.
- Keep AI-output behavior separate unless explicitly approved.

Files added or changed:

- `REINSERT_UI_SPEC.md`
- `WORKPACKAGES.md`
- `CHANGELOG.md`
- `handover/workpackages/20260607_1900_v13_3_reinsert_ui_planning.md`

Main planning decisions:

- Future UI block should appear after the existing Scrub Key import/reload section near the download/export area.
- Suggested section label: `Originele waarden terugzetten`.
- Suggested input label: `Plak hier de tekst waarin u originele waarden lokaal wilt terugzetten`.
- Suggested action button: `Zet originele waarden lokaal terug`.
- Suggested output label: `Herstelde tekst`.
- Suggested download label: `Download herstelde tekst (.txt)`.
- Future UI should call `reinsert_from_scrub_key(text, scrub_key)` and render the helper result instead of duplicating reinsert logic.
- Future UI should use a validated Scrub Key from import/reload first, or a key built from the current reviewed replacement table if no imported key exists.
- Reinsertion must require a separate visible user action and must not happen automatically when a Scrub Key is loaded.
- First UI implementation should produce restored text and `.txt` download only; no DOCX/PDF rehydration in the first UI step.

Required warnings specified:

- Reinsertion restores original sensitive values.
- Restored text may again contain personal or confidential information.
- Restored output must be manually reviewed before sharing.
- A Scrub Key is reversible/pseudonymization, not full anonymization.
- The key must remain local and protected.
- No AI/cloud processing is involved in the local reinsert step.

Required audit summary specified:

- mapping item count;
- active item count;
- excluded item count;
- replacement count;
- placeholders not found;
- unknown placeholders;
- duplicate placeholders;
- validation issues;
- local-only status;
- no-AI status;
- no-cloud status.

Testing and validation:

- Tests: not applicable; planning/specification-only workpackage.
- App verification: not applicable; no UI behavior changed in WP8 planning.

Intentionally not changed in planning phase:

- No UI code changed.
- No direct edit to `fix_streamlit_nested_expanders.py`.
- No direct edit to `presidio_streamlit.py`.
- No tests added or changed.
- No AI calls.
- No cloud processing.
- No automatic document rehydration.
- No TXT, CSV, DOCX or PDF export behavior changed.
- No Scrub Key JSON export behavior changed.
- No Scrub Key import/reload behavior changed.
- No secrets, tokens or real personal data.

Outcome:

- v13.3 deterministic reinsert UI was planned and then implemented in WP8B.

---

## v13.3 — Deterministic reinsert helper verification reconciliation

Status: completed and formally closed after Actions/sync verification.

Purpose:

- Administratively reconcile the v13.3 deterministic reinsert helper after coordinator-provided verification evidence.
- Replace the previous `awaiting coordinator verification of Actions/sync` status with formal closeout.
- Confirm that the helper remains pure, local and deterministic.
- Preserve the boundary that no UI, AI-output flow, cloud processing or export/download behavior change was added.

Verification evidence:

- Tests #106 green — commit `5854dbf`.
- Sync to Hugging Face Space #120 green — commit `5854dbf`.
- Tests #107 green — commit `43ecad4`.
- Sync to Hugging Face Space #121 green — commit `43ecad4`.
- Tests #108 green — commit `6e4ec9b`.
- Sync to Hugging Face Space #122 green — commit `6e4ec9b`.
- Tests #109 green — commit `eaf036a`.
- Sync to Hugging Face Space #123 green — commit `eaf036a`.

Validation status:

- GitHub Actions: green based on coordinator evidence.
- Hugging Face sync: green based on coordinator evidence.
- App verification: not applicable, helper-only package.

Outcome:

- v13.3 deterministic reinsert helper is completed and formally closed.

---

## v13.2 — Scrub Key import/reload UI app verification closeout

Status: completed, app-verified and closed.

Purpose:

- Administratively close the v13.2 Scrub Key import/reload UI after app verification.
- Record that the implemented import/reload flow works in the Hugging Face app.
- Preserve the boundary that this phase is import/reload only and does not add AI-output reinsert.

Technical evidence already recorded:

- Tests #89 green — commit `83353e4`.
- Tests #90 green — commit `4a1ef55`.
- Sync to Hugging Face Space #104 green — commit `4a1ef55`.
- Tests #91 green — commit `4d8bfe9`.
- Sync to Hugging Face Space #105 green — commit `4d8bfe9`.
- Tests #92 green — commit `ff8321f`.
- Sync to Hugging Face Space #106 green — commit `ff8321f`.

App verification:

- Confirmed by coordinator/user.
- `Scrub Key laden` works.
- Scrub Key import/reload UI is visible.
- Upload/paste import flow works.
- Pseudonymization/reversibility warning is visible.
- Existing `Download Scrub Key (.json)` remains visible.
- Existing TXT, CSV, DOCX and PDF downloads remain available.

Outcome:

- v13.2 Scrub Key import/reload UI is completed, app-verified and closed.

---

## Earlier completed work

- v13.2 Scrub Key import/reload UI integration.
- v13.2 Scrub Key import/reload helper and tests.
- v13.1 Scrub Key JSON export UI closeout.
- v12.6 Export sanity checks closeout.
- v13.0 Scrub Key specification and pure model.
- v12.5 Final review summary.
- v12.4 Review guidance text.
- Project governance setup.
- v12.3 Review table simplification.
- v12.2 Review focus filters.
- v12.1 Review table status model.
- v11.2 Dutch recognizer integration tests.
- v11.1 Legal reference recognizer hardening.
- v10 Regression test layer.
- v9.1 UI polish and baseline stabilization.
- v9 Dutch Legal UI Layer.

---

## Planned later phase — v13 and beyond

Possible directions:

- Deterministic reinsert UI verification and closeout.
- AI-output reinsert workflow review.
- Further recognizer expansion by legal domain.