scrub

Sleeping

App Files Files Community

scrub / CHANGELOG.md

solidprivacy-nl

Record deterministic reinsert UI implementation

84f5312 3 months ago

preview code

Raw

History Blame

10.8 kB

Changelog — SolidPrivacy Scrub

This changelog records meaningful product, architecture, workflow and recognizer changes for the Scrub legal document scrubber.

Conventions:

Keep this file explicit and human-readable.
Group changes by project phase / version.
Record what changed, why it changed, and what is intentionally not changed.
Do not use this as a substitute for tests; every recognizer hardening step should also add or update regression tests.

Current development rule

From v10 onward, recognizer work follows this order:

Add or update synthetic regression cases.
Add or update tests.
Change recognizer / scanner logic.
Verify GitHub Actions tests are green.
Let GitHub sync to Hugging Face automatically.
Test the app in Hugging Face.

For UI/UX-only work, prefer pure helper modules and tests before touching Streamlit UI flow.

v13.3 — Deterministic reinsert UI implementation

Status: implemented; awaiting GitHub Actions, Hugging Face sync and app verification.

Purpose:

Add the deterministic local reinsert UI described in REINSERT_UI_SPEC.md.
Let the user paste scrubbed or AI-generated text and locally restore mapped placeholders using a validated Scrub Key.
Show restored text, an audit summary and a .txt download for restored text.
Keep the step local and deterministic, with no AI calls and no cloud processing.

Files added or changed:

fix_streamlit_nested_expanders.py
tests/test_scrub_key_reinsert_ui_patch.py
tests/test_scrub_key_ui_patch.py
WORKPACKAGES.md
CHANGELOG.md
handover/workpackages/20260607_1915_v13_3_reinsert_ui_implementation.md

Main changes:

Added import/wiring for the verified helper:
- from scrub_key_reinsert import reinsert_from_scrub_key.
Added the section Originele waarden terugzetten after the existing Scrub Key import/reload area.
Added the input label Plak hier de tekst waarin u originele waarden lokaal wilt terugzetten.
Added the explicit action button Zet originele waarden lokaal terug.
Added local helper call:
- reinsert_from_scrub_key(reinsert_input_text, active_reinsert_scrub_key).
Added output label Herstelde tekst.
Added Download herstelde tekst (.txt) with text/plain output.
Added Controleverslag terugzetten audit summary.
Added audit rendering for item count, active item count, excluded item count, replacement count, placeholders not found, unknown placeholders, duplicate placeholders, validation issues, local-only status, AI-processing status and cloud-processing status.
Added visible warning that restored text may again contain personal or confidential information and must be reviewed before sharing.
Added local/no-AI/no-cloud wording.
Stores a successfully imported Scrub Key in st.session_state["active_scrub_key"] for local reinsert use.
Falls back to the currently built Scrub Key from reviewed replacement rows if no imported key is active.

Testing and validation:

Added tests/test_scrub_key_reinsert_ui_patch.py.
Updated tests/test_scrub_key_ui_patch.py so it no longer forbids the intentionally added reinsert flow, while still guarding no-AI, no-cloud and no-automatic-document-rehydration boundaries.
Local targeted validation on a reconstructed subset passed:
- PYTHONPATH=. pytest -q tests/test_scrub_key.py tests/test_scrub_key_import.py tests/test_scrub_key_reinsert.py tests/test_scrub_key_reinsert_ui_patch.py tests/test_scrub_key_import_ui_patch.py tests/test_scrub_key_ui_patch.py → 57 passed.
Full repository test suite was not run from this connector environment.
GitHub Actions are pending for WP8B commits.
Hugging Face sync is pending for WP8B commits.
App verification is pending because UI behavior changed.

Intentionally not changed:

No direct edit to presidio_streamlit.py.
No edit to scrub_key_reinsert.py.
No edit to scrub_key.py.
No edit to scrub_key_import.py.
No AI calls.
No cloud processing.
No automatic document rehydration.
No DOCX/PDF reinsert.
No TXT, CSV, DOCX or PDF scrubbed export behavior changed.
No Scrub Key JSON export behavior intentionally changed.
No Scrub Key import/reload behavior intentionally changed except storing the validated imported key in session state for reinsert use.
No silent overwrite of existing review rows.
No secrets, tokens or real personal data.

Outcome:

v13.3 deterministic local reinsert UI is implemented and awaits external verification and app testing.

v13.3 — Deterministic reinsert UI planning

Status: implemented; reinsert UI implementation completed in WP8B and awaits verification.

Purpose:

Plan deterministic local reinsert UI before changing Streamlit UI code.
Define where the UI should appear, what labels it should use, what state it should rely on, and which warnings and audit fields it must show.
Keep AI-output behavior separate unless explicitly approved.

Files added or changed:

REINSERT_UI_SPEC.md
WORKPACKAGES.md
CHANGELOG.md
handover/workpackages/20260607_1900_v13_3_reinsert_ui_planning.md

Main planning decisions:

Future UI block should appear after the existing Scrub Key import/reload section near the download/export area.
Suggested section label: Originele waarden terugzetten.
Suggested input label: Plak hier de tekst waarin u originele waarden lokaal wilt terugzetten.
Suggested action button: Zet originele waarden lokaal terug.
Suggested output label: Herstelde tekst.
Suggested download label: Download herstelde tekst (.txt).
Future UI should call reinsert_from_scrub_key(text, scrub_key) and render the helper result instead of duplicating reinsert logic.
Future UI should use a validated Scrub Key from import/reload first, or a key built from the current reviewed replacement table if no imported key exists.
Reinsertion must require a separate visible user action and must not happen automatically when a Scrub Key is loaded.
First UI implementation should produce restored text and .txt download only; no DOCX/PDF rehydration in the first UI step.

Required warnings specified:

Reinsertion restores original sensitive values.
Restored text may again contain personal or confidential information.
Restored output must be manually reviewed before sharing.
A Scrub Key is reversible/pseudonymization, not full anonymization.
The key must remain local and protected.
No AI/cloud processing is involved in the local reinsert step.

Required audit summary specified:

mapping item count;
active item count;
excluded item count;
replacement count;
placeholders not found;
unknown placeholders;
duplicate placeholders;
validation issues;
local-only status;
no-AI status;
no-cloud status.

Testing and validation:

Tests: not applicable; planning/specification-only workpackage.
App verification: not applicable; no UI behavior changed in WP8 planning.

Intentionally not changed in planning phase:

No UI code changed.
No direct edit to fix_streamlit_nested_expanders.py.
No direct edit to presidio_streamlit.py.
No tests added or changed.
No AI calls.
No cloud processing.
No automatic document rehydration.
No TXT, CSV, DOCX or PDF export behavior changed.
No Scrub Key JSON export behavior changed.
No Scrub Key import/reload behavior changed.
No secrets, tokens or real personal data.

Outcome:

v13.3 deterministic reinsert UI was planned and then implemented in WP8B.

v13.3 — Deterministic reinsert helper verification reconciliation

Status: completed and formally closed after Actions/sync verification.

Purpose:

Administratively reconcile the v13.3 deterministic reinsert helper after coordinator-provided verification evidence.
Replace the previous awaiting coordinator verification of Actions/sync status with formal closeout.
Confirm that the helper remains pure, local and deterministic.
Preserve the boundary that no UI, AI-output flow, cloud processing or export/download behavior change was added.

Verification evidence:

Tests #106 green — commit 5854dbf.
Sync to Hugging Face Space #120 green — commit 5854dbf.
Tests #107 green — commit 43ecad4.
Sync to Hugging Face Space #121 green — commit 43ecad4.
Tests #108 green — commit 6e4ec9b.
Sync to Hugging Face Space #122 green — commit 6e4ec9b.
Tests #109 green — commit eaf036a.
Sync to Hugging Face Space #123 green — commit eaf036a.

Validation status:

GitHub Actions: green based on coordinator evidence.
Hugging Face sync: green based on coordinator evidence.
App verification: not applicable, helper-only package.

Outcome:

v13.3 deterministic reinsert helper is completed and formally closed.

v13.2 — Scrub Key import/reload UI app verification closeout

Status: completed, app-verified and closed.

Purpose:

Administratively close the v13.2 Scrub Key import/reload UI after app verification.
Record that the implemented import/reload flow works in the Hugging Face app.
Preserve the boundary that this phase is import/reload only and does not add AI-output reinsert.

Technical evidence already recorded:

Tests #89 green — commit 83353e4.
Tests #90 green — commit 4a1ef55.
Sync to Hugging Face Space #104 green — commit 4a1ef55.
Tests #91 green — commit 4d8bfe9.
Sync to Hugging Face Space #105 green — commit 4d8bfe9.
Tests #92 green — commit ff8321f.
Sync to Hugging Face Space #106 green — commit ff8321f.

App verification:

Confirmed by coordinator/user.
Scrub Key laden works.
Scrub Key import/reload UI is visible.
Upload/paste import flow works.
Pseudonymization/reversibility warning is visible.
Existing Download Scrub Key (.json) remains visible.
Existing TXT, CSV, DOCX and PDF downloads remain available.

Outcome:

v13.2 Scrub Key import/reload UI is completed, app-verified and closed.

Earlier completed work

v13.2 Scrub Key import/reload UI integration.
v13.2 Scrub Key import/reload helper and tests.
v13.1 Scrub Key JSON export UI closeout.
v12.6 Export sanity checks closeout.
v13.0 Scrub Key specification and pure model.
v12.5 Final review summary.
v12.4 Review guidance text.
Project governance setup.
v12.3 Review table simplification.
v12.2 Review focus filters.
v12.1 Review table status model.
v11.2 Dutch recognizer integration tests.
v11.1 Legal reference recognizer hardening.
v10 Regression test layer.
v9.1 UI polish and baseline stabilization.
v9 Dutch Legal UI Layer.

Planned later phase — v13 and beyond

Possible directions:

Deterministic reinsert UI verification and closeout.
AI-output reinsert workflow review.
Further recognizer expansion by legal domain.