# Changelog — SolidPrivacy Scrub This changelog records meaningful product, architecture, workflow and recognizer changes for the Scrub legal document scrubber. Conventions: - Keep this file explicit and human-readable. - Group changes by project phase / version. - Record what changed, why it changed, and what is intentionally not changed. - Do not use this as a substitute for tests; every recognizer hardening step should also add or update regression tests. --- ## Current development rule From v10 onward, recognizer work follows this order: 1. Add or update synthetic regression cases. 2. Add or update tests. 3. Change recognizer / scanner logic. 4. Verify GitHub Actions tests are green. 5. Let GitHub sync to Hugging Face automatically. 6. Test the app in Hugging Face. For UI/UX-only work, prefer pure helper modules and tests before touching Streamlit UI flow. --- ## v13.2 — Scrub Key import/reload UI closeout Status: completed and formally closed after Actions/sync verification; app verification pending user/coordinator confirmation. Purpose: - Administratively close the v13.2 Scrub Key import/reload UI workpackage. - Record coordinator evidence that GitHub Actions tests and GitHub to Hugging Face sync are green. - Preserve the boundary that this phase is import/reload only and does not add AI-output reinsert. Files added or changed: - `WORKPACKAGES.md` - `CHANGELOG.md` - `handover/workpackages/20260607_1715_v13_2_scrub_key_import_ui_closeout.md` Closeout evidence: - Tests #89 green — commit `83353e4`. - Tests #90 green — commit `4a1ef55`. - Sync to Hugging Face Space #104 green — commit `4a1ef55`. - Tests #91 green — commit `4d8bfe9`. - Sync to Hugging Face Space #105 green — commit `4d8bfe9`. - Tests #92 green — commit `ff8321f`. - Sync to Hugging Face Space #106 green — commit `ff8321f`. Closeout notes: - Scrub Key import/reload UI was implemented. - GitHub Actions tests are green based on coordinator evidence. - GitHub to Hugging Face sync is green based on coordinator evidence. - Existing Scrub Key JSON export remains in place. - Import/reload remains local and uses existing helper logic. - The key remains pseudonymization/reversible and must be protected. - Existing TXT, CSV, DOCX and PDF downloads were not intentionally changed. App verification: - App verification: pending user/coordinator confirmation. Intentionally not changed in closeout: - No code files changed. - No tests changed. - No direct edit to `fix_streamlit_nested_expanders.py`. - No direct edit to `presidio_streamlit.py`. - No edit to `scrub_key.py`. - No edit to `scrub_key_import.py`. - No edit to `tests/*`. - No AI-output reinsert behavior. - No automatic document rehydration. - No change to TXT, CSV, DOCX or PDF export/download behavior. - No cloud processing. - No secrets, tokens or real personal data. --- ## v13.2 — Scrub Key import/reload UI integration Status: implemented; Actions/sync later verified green in closeout. Purpose: - Integrate the existing Scrub Key import/reload helper into the Streamlit startup patch flow. - Let the user upload or paste a previously exported Scrub Key JSON. - Validate imported Scrub Keys before loading mappings into the current review/replacement table. - Keep import/reload local and separate from deterministic reinsert or AI-output workflows. Files added or changed: - `fix_streamlit_nested_expanders.py` - `tests/test_scrub_key_import_ui_patch.py` - `WORKPACKAGES.md` - `CHANGELOG.md` - `handover/workpackages/20260607_1645_v13_2_scrub_key_import_ui.md` Main changes: - Added the `Scrub Key laden` UI section near the existing `Scrub Key (JSON)` export block. - Added upload support for Scrub Key JSON files. - Added paste support for Scrub Key JSON text. - Reused the existing `build_scrub_key_import_result(...)` helper instead of duplicating parsing or validation logic. - Added visible validation feedback before mapping rows are loaded. - Added warnings that a Scrub Key makes values locally reversible and is pseudonymization, not full anonymization. - Added warning text that the key must stay local and protected and should not be shared with AI services or third parties unless consciously intended and allowed. - Loaded validated mapping rows into the current replacement table only after the visible `Valideer en laad Scrub Key` user action. - Kept the existing `Download Scrub Key (.json)` export block. Testing: - Added `tests/test_scrub_key_import_ui_patch.py`. - Local targeted validation on the reconstructed connector subset passed: - `PYTHONPATH=. pytest -q tests/test_scrub_key.py` → 6 passed. - `PYTHONPATH=. pytest -q tests/test_scrub_key_import.py` → 8 passed. - `PYTHONPATH=. pytest -q tests/test_scrub_key_import_ui_patch.py` → 9 passed. - `PYTHONPATH=. pytest -q tests/test_scrub_key_ui_patch.py` → 12 passed. - `PYTHONPATH=. pytest -q` on the available subset → 35 passed. Intentionally not changed: - No direct edit to `presidio_streamlit.py`. - No deterministic reinsert behavior. - No AI-output flow. - No automatic document rehydration. - No silent replacement of review rows without a visible user action. - No change to TXT, CSV, DOCX or PDF export behavior. - No change to existing Scrub Key JSON export behavior. - No cloud processing. - No server-side Scrub Key storage. - No secrets, tokens or real personal data. --- ## v13.2 — Scrub Key import/reload helper and tests Status: helper and tests implemented; coordinator evidence reported green checks before UI integration. Purpose: - Prepare reliable import/reload of a previously saved Scrub Key JSON file. - Keep this work pure and helper-only before UI integration. - Prepare future mapping reuse, session continuation and deterministic reinsert work. - Preserve local-only handling and warning language for sensitive Scrub Keys. Files added or changed: - `scrub_key_import.py` - `tests/test_scrub_key_import.py` - `WORKPACKAGES.md` - `CHANGELOG.md` - `handover/workpackages/20260607_1605_v13_2_scrub_key_import_helper.md` Main changes: - Added `scrub_key_import.py` as a pure helper module. - Added `validate_scrub_key_import_text(json_text)` for safe validation errors. - Added `normalise_scrub_key_items(scrub_key)` to convert Scrub Key items into review-table-like mapping rows. - Added `build_scrub_key_import_result(json_text)` with a UI-friendly result shape: - `ok`; - `errors`; - `warnings`; - `scrub_key`; - `mapping_rows`; - `item_count`; - `reversible`; - `privacy_model`; - `document_label`. - Reused the existing `scrub_key_from_json(...)` and `validate_scrub_key(...)` model helpers. - Added safe Dutch user-facing errors for empty input, invalid JSON syntax, invalid top-level structure and invalid Scrub Key content. - Added a local-only privacy warning explaining that imported Scrub Keys make replacements locally reversible and should not be shared with AI services or third parties unless consciously intended and allowed. Testing: - Added `tests/test_scrub_key_import.py`. - Tests cover valid Scrub Key JSON import, normalized row mapping, privacy warning, empty JSON text, invalid JSON syntax, invalid top-level format, structural validation errors, no input mutation and synthetic Dutch legal values only. - Coordinator evidence supplied for green checks before UI integration: - Tests #83 green; - Tests #84 green; - Sync to Hugging Face Space #98 green; - Tests #85 green; - Sync to Hugging Face Space #99 green; - Tests #86 green; - Sync to Hugging Face Space #100 green. Intentionally not changed in helper phase: - No UI changes. - No direct edit to `fix_streamlit_nested_expanders.py`. - No direct edit to `presidio_streamlit.py`. - No edit to existing Scrub Key export UI. - No reinsert behavior. - No AI-output flow. - No cloud processing. - No server-side Scrub Key storage. - No real personal data in tests. - No change to TXT, CSV, DOCX or PDF download behavior. --- ## v13.1 — Scrub Key JSON export UI closeout Status: completed and app verified after mapping hotfix. Purpose: - Close v13.1 after confirming that the local Scrub Key JSON export works in the Hugging Face app. - Record that the earlier `original_value` validation error was fixed. - Preserve the boundary that Scrub Key import/reload and reinsert are not implemented yet. Files added or changed in the full v13.1 line: - `fix_streamlit_nested_expanders.py` - `tests/test_scrub_key_ui_patch.py` - `WORKPACKAGES.md` - `CHANGELOG.md` - `handover/workpackages/20260607_1535_v13_1_scrub_key_json_export_ui.md` - `handover/workpackages/20260607_1535_v13_1_scrub_key_ui_mapping_hotfix.md` - `handover/workpackages/20260607_1550_v13_1_scrub_key_json_export_closeout.md` Main changes: - Added a `Scrub Key (JSON)` section to the download/export flow. - Added a `Download Scrub Key (.json)` button. - Added warning text explaining that the Scrub Key makes replaced values locally reversible. - Added warning text that this is pseudonymization, not full anonymization. - Added warning text not to share the key with AI services or third parties unless consciously intended and allowed. - Added timestamp creation in the UI/export layer so the pure `scrub_key.py` model remains deterministic and side-effect free. - Added the mapping hotfix so app review-table rows are converted into Scrub Key model rows before calling `build_scrub_key(...)`. Mapping hotfix: - `find` → `original_value` - `replace_with` → `placeholder` - `entity_type` → `entity_type` - `type_label` → `type_label` - `source` → `source` - `review_status` → `review_status` - `include` → `include` Testing and verification: - Initial v13.1 UI implementation: - `Tests #73` green for commit `9d349bb`. - `Sync to Hugging Face Space #87` green for commit `9d349bb`. - Mapping hotfix: - `Tests #78` green for commit `8d33941`. - `Sync to Hugging Face Space #92` green for commit `8d33941`. - User verified in the Hugging Face app: - `Scrub Key (JSON)` section is visible. - pseudonymization / reversibility warning is visible. - `Item has empty required field: original_value` is gone. - `Download Scrub Key (.json)` is visible. - Scrub Key JSON download works. Intentionally not changed: - No edit to `scrub_key.py` in the mapping hotfix. - No direct edit to `presidio_streamlit.py`. - No Scrub Key import/reload. - No reinsert UI. - No AI-output flow. - No cloud processing. - No server-side Scrub Key storage. - No change to TXT, CSV, DOCX or PDF download behavior. - No change to existing replacement/export semantics. - No `st.stop()` or export blocking behavior added. Outcome: - v13.1 Scrub Key JSON export is complete. --- ## v12.6 — Export sanity checks closeout Status: completed and administratively closed after coordinator closeout instruction. Purpose: - Close the v12.6 export sanity workpackage after helper and UI integration were completed. - Record that the UI warning block is advisory only. - Preserve the documented boundary that export/download behavior was not changed. Files added or changed in the full v12.6 line: - `export_sanity.py` - `tests/test_export_sanity.py` - `tests/test_export_sanity_ui_patch.py` - `fix_streamlit_nested_expanders.py` - `WORKPACKAGES.md` - `CHANGELOG.md` - `handover/workpackages/20260607_1405_v12_6_export_sanity_helper.md` - `handover/workpackages/20260607_1445_v12_6_export_sanity_ui.md` - `handover/workpackages/20260607_1515_v12_6_export_sanity_closeout.md` Main changes: - Added pure helper logic for advisory export readiness checks. - Added Dutch user-facing warning text for unchecked `Controle nodig` rows, candidate rows not included, no replacements selected, required user review and export not guaranteeing full anonymization. - Integrated the existing `export_sanity.py` helper into the Streamlit startup patch flow. - Added UI text for `Extra exportcontrole` near the existing v12.5 `Eindcontrole vóór download` block. - Added patch-level tests to guard that the export sanity helper is wired into the UI patch. - Administratively closed WP3C without changing code files. Testing and verification: - Helper validation recorded in WP3A handover: `PYTHONPATH=. pytest -q tests/test_export_sanity.py tests/test_review_summary.py` → 12 passed. - Coordinator reconciled helper verification: `Tests #58` green, `Sync to Hugging Face Space #72` green, commit `b0bf8ae`. - WP3C was administrative closeout only; no new local pytest run was performed by that worker. Intentionally not changed: - No code files changed in WP3C closeout. - No direct edit to `presidio_streamlit.py`. - No export/download blocking. - No change to which rows are included in export. - No change to TXT, CSV, DOCX or PDF download logic. - No Scrub Key logic added in v12.6. - No reinsert workflow added in v12.6. - No cloud processing introduced. Outcome: - v12.6 is closed. - v12 Review UX is complete through guidance, final review summary and export sanity warnings. --- ## Earlier completed work - v13.0 Scrub Key specification and pure model. - v12.5 Final review summary. - v12.4 Review guidance text. - Project governance setup. - v12.3 Review table simplification. - v12.2 Review focus filters. - v12.1 Review table status model. - v11.2 Dutch recognizer integration tests. - v11.1 Legal reference recognizer hardening. - v10 Regression test layer. - v9.1 UI polish and baseline stabilization. - v9 Dutch Legal UI Layer. --- ## Planned later phase — v13 and beyond Possible directions: - Deterministic reinsert helper. - AI-output reinsert. - Further recognizer expansion by legal domain.