scrub

Running

App Files Files Community

scrub / ROADMAP.md

solidprivacy-nl

Add central product development roadmap

ceee13f 3 months ago

preview code

Raw

History Blame

15 kB

SolidPrivacy Scrub — Product & Development Roadmap

This document is the central reference for future Scrub development.

Use it together with CHANGELOG.md:

ROADMAP.md explains the bigger picture: direction, priorities, phases and why we are doing the current work.
CHANGELOG.md records what has actually changed, when, and in which version/fase.

The roadmap should be updated whenever the product strategy or development sequence changes meaningfully.

1. Product vision

Scrub is evolving from a technical Presidio demo into a local-first professional document scrubber for confidential Dutch documents.

The primary starting market is:

Scrub Legal
A local Dutch legal scrubber for process documents, case files and AI use.

The broader product direction is:

A Dutch local-first privacy scrubber for professional confidential documents.

The long-term product should help professionals safely prepare documents for:

AI use;
external sharing;
internal analysis;
training examples;
reporting;
controlled publication.

The key promise:

Sensitive information stays local.
The user remains in control.
The document stays readable.

2. Core product principle

The product is not just a generic anonymizer.

The real problem is:

How can a professional safely use or share confidential documents without losing context, legal meaning or auditability?

Therefore Scrub must optimize for:

local processing;
context preservation;
human review;
consistent placeholders;
auditability;
domain-specific recognition;
safe export;
eventually desktop/offline deployment.

3. Strategic lessons from external products

Two external products shaped this roadmap:

anonym.plus;
CamoText.

3.1 Lessons from anonym.plus

Important concepts to learn from:

offline-first as a main trust message;
desktop installer as real product form;
clear processing pipeline;
multiple anonymization operators;
presets/profiles;
entity recognition catalogue;
batch processing;
local encrypted vault/settings;
transparent security limitations;
documentation and demos as part of the product.

Relevant product lesson:

Do not sell only detection. Sell a trustworthy local workflow.

3.2 Lessons from CamoText

CamoText sharpened the roadmap further because it is focused on AI-safe document preparation.

Important concepts to learn from:

position the product as an AI-safety workflow;
desktop-first / offline-first;
human-in-the-loop review;
anonymization key / mapping file;
reinsert original terms into AI output;
priorities / categories / exclusions;
category-level review actions;
redaction mode as a clear export mode;
metadata-free clean output;
batch mode for folders;
CLI/headless mode later;
observable local-security validation.

Most important roadmap addition from CamoText:

Scrub → Review → Scrub Key → AI → Reinsert → Export → Audit

4. What differentiates Scrub

Scrub should not try to win as a generic international PII anonymizer.

The strongest differentiator is:

Dutch domain-specific confidential document scrubbing.

For the first product line, this means:

Dutch legal documents
Dutch legal identifiers
Dutch case/document references
Dutch process roles
legal context preservation
AI-ready readable output

Examples of context that must remain readable:

slachtoffer;
minderjarige;
verzoeker;
verweerder;
eiser;
gemachtigde;
rechtbank;
zaaknummer label;
claim context;
incident context.

The product should mask or replace the sensitive value, not the legal meaning of the sentence.

5. Current status

Current development status at the time of this roadmap update:

v9      Dutch Legal UI Layer                         completed
v9.1    UI polish                                    completed
v10     Regression test foundation                   completed
v11.1   Legal reference hardening / audit layer      completed
v11.2   Dutch recognizer integration tests           completed
v12.1   Review status model                          completed
v12.2   Review focus filters                         completed
v12.3   Review table simplification                  implemented; pending final verification after bugfix

Important recent bugfix:

v12.3 introduced table configuration using pandas DataFrame columns.
A pandas Index cannot be boolean-tested.
This was fixed by converting available_columns explicitly to list/set.

Immediate verification before further work:

GitHub Actions Tests must be green.
GitHub → Hugging Face sync must be green.
Hugging Face app must reload successfully.
The same legal test example must no longer show the pandas Index truth-value error.

6. Development governance

From v10 onward, recognizer work must follow this sequence:

Add or update synthetic regression cases.
Add or update tests.
Change recognizer/scanner logic.
Verify GitHub Actions tests are green.
Let GitHub sync to Hugging Face automatically.
Test the app in Hugging Face.
Update CHANGELOG.md.
If the strategic roadmap changes, update ROADMAP.md.

For UI/UX-only work:

Add pure helper modules where possible.
Add tests for helper logic.
Patch UI.
Verify GitHub Actions tests.
Verify Hugging Face app.
Update changelog.

7. Current development line — v12 Review UX

The current line of work is v12: make the review workflow safer and easier for legal users.

v12.1 — Review status model

Status: completed.

Added statuses:

Automatisch vervangen;
Controle nodig;
Handmatig toegevoegd;
Onthouden vervanging.

Purpose:

Help users understand what each row means before export.

v12.2 — Review focus filters

Status: completed.

Added filters:

Toon alles;
Alleen controle nodig;
Alleen juridische referenties;
Alleen namen/adressen;
Alleen lage zekerheid.

Important design rule:

Filters are focus views only. The full replacement table remains the source of truth.

v12.3 — Review table simplification

Status: implemented; pending final verification after bugfix.

Main table should focus on:

Meenemen;
Onthouden;
Status;
Gevonden tekst;
Vervangen door;
Type gegeven;
Zekerheid.

Technical fields should move to:

Technische details bij de vervangtabel

8. Next immediate phase — finish v12

v12.4 — Review guidance text

Goal:

Make the review workflow self-explanatory.

Planned scope:

explain that only checked rows are included in export;
explain that Controle nodig rows are not automatically safe;
explain the focus filter is only a view, not the export scope;
explain technical details are for audit/debugging;
add clearer guidance around AI usage: scrub first, then use AI.

Non-goals:

no recognizer changes;
no export semantics change;
no desktop/MSI work.

v12.5 — Final review summary

Goal:

Show a final export readiness summary before downloads.

Planned summary:

automatically detected rows;
rows needing review;
manually added rows;
remembered replacements;
checked rows included in export;
unchecked rows excluded from export;
open candidate warning.

v12.6 — Export sanity checks

Goal:

Warn users before exporting if risk remains.

Planned checks:

warning if Controle nodig rows remain unchecked;
warning if candidate rows exist but are not included;
warning if no replacements are selected;
warning if export mode implies redaction vs pseudonymization risk;
reminder that user review remains required.

9. Next strategic phase — v13 Scrub Key / Reinsert

This is the most important strategic addition after the v12 review flow.

Inspired by CamoText’s anonymization-key and reinsert workflow.

v13.1 — Scrub Key JSON export

Goal:

Create a local mapping file for replacements.

A Scrub Key should contain:

original value;
placeholder;
entity type;
user-facing type label;
source;
review status;
include/exclude state;
timestamp;
optional project/dossier label.

v13.2 — Scrub Key import/reload

Goal:

Allow users to reuse a previously saved mapping.

Use cases:

consistent names across multiple documents;
same client/case over several files;
continue work later;
reinsert AI output.

v13.3 — AI-output reinsert

Goal:

Paste AI-generated output back into Scrub and locally restore original terms.

Workflow:

scrub original document;
send scrubbed text to AI;
paste AI output back into Scrub;
load Scrub Key;
reinsert original values locally.

v13.4 — Pseudonymization warnings

Goal:

Make it clear that reversible mapping is pseudonymization, not true anonymization.

Warnings should explain:

if a Scrub Key exists, the text may be reversible;
key security matters;
do not share the key with external parties unless intended.

10. v14 — Manual output review / highlight workflow

Goal:

Allow users to manually mark text from the output/review area and replace it everywhere.

Planned scope:

search in scrubbed output;
manually add selected text to replacement table;
choose replacement type;
replace selected text everywhere;
add manual replacement to Scrub Key.

Why this matters:

Legal users often see missing sensitive terms while reading, not while editing a table.

11. v15 — Document hygiene and metadata-clean export

Goal:

Produce clean outputs that do not leak metadata or hidden document content.

DOCX priorities:

remove document metadata;
remove author information where possible;
handle comments;
handle tracked changes policy;
include headers/footers in scrubbing;
preserve basic layout where feasible;
produce new clean output file.

PDF priorities:

remove metadata where possible;
support text-based PDF;
warn when scanned/OCR content is not processed;
explicitly state limitations.

This phase is strategically important because legal documents often contain hidden metadata.

12. v16 — Desktop/local proof of concept

Goal:

Prove that Scrub can run locally outside Hugging Face.

Preferred direction:

Python backend remains local;
frontend can be Streamlit initially, then desktop wrapper;
portable Windows build first;
MSI later;
no internet required for core processing.

Possible technical paths:

Streamlit local launcher;
Tauri + local Python service;
Electron + local Python service;
PyInstaller/Nuitka for local packaging experiments.

Success criteria:

app starts locally;
sample document can be scrubbed offline;
no cloud calls required;
Hugging Face no longer needed for real-world use.

13. v17 — Legal profiles / vertical profiles

Original order had legal profiles before Scrub Key, but after CamoText review the order changed:

First: finish review workflow and Scrub Key.
Then: expand domain profiles.

Planned Legal profiles:

algemeen juridisch;
familierecht;
strafrecht;
arbeidsrecht;
bestuursrecht;
vreemdelingenrecht;
letselschade / verzekering;
huurrecht / vastgoed;
medisch-juridisch.

Each profile should have:

own example texts;
own recognizer emphasis;
own false-positive guards;
own review guidance;
own regression tests.

14. v18 — Batch / dossiermap

Goal:

Process multiple documents or full case folders.

Planned scope:

input folder;
output folder;
preserve folder structure;
scrub keys per file or per dossier;
summary report;
ZIP export;
later parallel processing.

Batch should not come before single-document flow is reliable.

15. v19 — CLI / automation

Goal:

Support headless and enterprise workflows.

Possible commands:

scrublegal --input dossier.docx --output dossier_scrubbed.docx --profile arbeidsrecht
scrublegal --input-dir zaakmap --output-dir zaakmap_ai --key-dir keys
scrublegal --reinsert ai_output.docx --key zaak_key.json

This is useful for:

power users;
IT-managed workflows;
batch processing;
future integrations;
AI-agent workflows.

16. v20 — Broader vertical markets

Scrub Core should remain one engine, with vertical profiles on top.

Potential future verticals:

Scrub Legal;
Scrub Zorg;
Scrub HR / Arbo;
Scrub Claims / Verzekering;
Scrub Gemeente / Sociaal Domein;
Scrub Finance / Accountancy;
Scrub Research.

Do not build separate apps too early.

Architecture:

Scrub Core
  + profile: Legal
  + profile: Zorg
  + profile: HR/Arbo
  + profile: Claims
  + profile: Gemeente
  + profile: Finance
  + profile: Research

Each profile should add:

recognizers;
examples;
false-positive guards;
UI copy;
exports/audit labels;
tests.

17. Product architecture target

Current prototype architecture:

Hugging Face Space
Streamlit UI
Presidio/spaCy recognizers
Dutch legal recognizers
Candidate scanner
Review table
Exports
GitHub Actions tests
GitHub → Hugging Face sync

Target architecture:

Local desktop app
Local recognition engine
Local review workflow
Local Scrub Key vault/files
Local exports
Optional CLI
No required cloud processing

The intermediate architecture can remain Streamlit-based while we validate workflow and recognizers.

18. Security and trust principles

For the final product:

no document upload to third-party cloud;
no model training on user documents;
no telemetry containing document content;
clear warning when using cloud AI outside Scrub;
local-only processing as default;
metadata-aware exports;
audit report;
clear distinction between anonymization, pseudonymization and redaction.

Future security validation should include:

offline mode demonstration;
network traffic check;
clear file storage locations;
local key storage explanation;
user-controlled deletion.

19. Current next action

Before new roadmap work starts:

Verify the latest v12.3 pandas Index bugfix.
Confirm GitHub Actions Tests are green.
Confirm GitHub → Hugging Face sync is green.
Reload the app.
Confirm the simplified table and technical details expander work.

Then continue with:

v12.4 — Review guidance text

20. Maintenance rule for this roadmap

Update this file when:

the development sequence changes;
external product research changes priorities;
a new major phase is introduced;
a phase is completed and its status changes;
we decide to target a new vertical market;
desktop/MSI direction changes.

Do not use this file for every small code change. Use CHANGELOG.md for implementation history.