Spaces:
Running
Running
Commit ·
bffab6a
1
Parent(s): 870cb34
Rebuild clean MVP document fidelity hardening branch
Browse files- DECISION_LOG.md +31 -0
- RELEASE_NOTES.md +9 -0
- RISK_REGISTER.md +2 -2
- handover/workpackages/20260717_2230_mvp_document_hygiene_fidelity_hardening.md +78 -0
- mvp_document_fidelity_report.py +138 -0
- mvp_phase6_document_cases.py +39 -6
- output/validation/mvp_document_fidelity_pr_validation.json +28 -0
- output/validation/mvp_document_fidelity_pr_validation.log +8 -0
- output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json +312 -0
- reinsert_mode_ui.py +3 -3
- scripts/run_mvp_document_hygiene_fidelity_report.py +36 -0
- scrub_key_document_reinsert.py +112 -45
- tests/test_mvp_document_fidelity_pr_final_contracts.py +42 -0
- tests/test_mvp_document_fidelity_report.py +111 -0
- tests/test_mvp_document_fidelity_ui_copy.py +40 -0
- tests/test_mvp_document_hygiene_fidelity_hardening.py +203 -0
- tests/test_mvp_phase6_e2e_synthetic_validation_matrix.py +9 -8
- workpackage_claims/scrub_wp_mvp_document_hygiene_fidelity_hardening.md +52 -0
DECISION_LOG.md
CHANGED
|
@@ -4,6 +4,37 @@ This file records accepted strategic, product and architecture decisions.
|
|
| 4 |
|
| 5 |
---
|
| 6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
## 2026-07-17 — D029 — Current Phase 6 matrix does not justify a recognizer fix
|
| 8 |
|
| 9 |
Status: accepted evidence-routing decision
|
|
|
|
| 4 |
|
| 5 |
---
|
| 6 |
|
| 7 |
+
## 2026-07-17 — D030 — Restore existing DOCX header and footer text during deterministic reinsert
|
| 8 |
+
|
| 9 |
+
Status: accepted implementation decision
|
| 10 |
+
|
| 11 |
+
Decision:
|
| 12 |
+
|
| 13 |
+
```text
|
| 14 |
+
Extend the existing local deterministic DOCX reinsert helper to process WordprocessingML text nodes in word/header*.xml and word/footer*.xml in addition to word/document.xml.
|
| 15 |
+
```
|
| 16 |
+
|
| 17 |
+
Reason:
|
| 18 |
+
|
| 19 |
+
- The scrubbed DOCX export already replaces reviewed values in body, table, header and footer paragraphs.
|
| 20 |
+
- The Phase 6 matrix showed that reinsert restored body/table values but left header/footer placeholders behind.
|
| 21 |
+
- The gap belongs to document fidelity and reinsert scope, not detection or recognizer behavior.
|
| 22 |
+
|
| 23 |
+
Boundaries:
|
| 24 |
+
|
| 25 |
+
- Process only existing body, header and footer WordprocessingML text nodes.
|
| 26 |
+
- Preserve unrelated OOXML package parts byte-for-byte where they are not rewritten.
|
| 27 |
+
- Do not claim support for comments, tracked-change-only parts, footnotes/endnotes, text boxes, metadata or placeholders split across text nodes.
|
| 28 |
+
- Do not add OCR or restored-PDF behavior.
|
| 29 |
+
- Keep processing local, deterministic and Scrub Key driven.
|
| 30 |
+
|
| 31 |
+
Evidence:
|
| 32 |
+
|
| 33 |
+
- `output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json`
|
| 34 |
+
- `output/validation/mvp_phase6_false_negative_gap_triage.json`
|
| 35 |
+
|
| 36 |
+
---
|
| 37 |
+
|
| 38 |
## 2026-07-17 — D029 — Current Phase 6 matrix does not justify a recognizer fix
|
| 39 |
|
| 40 |
Status: accepted evidence-routing decision
|
RELEASE_NOTES.md
CHANGED
|
@@ -1,3 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
## 2026-07-16 — Handmatige aanvulling compacter
|
| 2 |
|
| 3 |
- `Gemiste waarde toevoegen` gebruikt bij openen een compactere invoerregel voor waarde, type en vervanging.
|
|
|
|
| 1 |
+
## 2026-07-17 — DOCX-herstel omvat nu kop- en voetteksten
|
| 2 |
+
|
| 3 |
+
- Bij het terugzetten van originele waarden in een DOCX worden nu ook bestaande kop- en voetteksten meegenomen.
|
| 4 |
+
- Hoofdtekst en tabellen blijven ondersteund.
|
| 5 |
+
- Opmerkingen, alleen-in-wijzigingen aanwezige tekst, voetnoten/eindnoten, tekstvakken, metadata en placeholders die over meerdere Word-tekstnodes zijn verdeeld, blijven buiten deze versie.
|
| 6 |
+
- PDF-herstel blijft beperkt tot herstelde TXT; OCR en een hersteld PDF-bestand zijn niet toegevoegd.
|
| 7 |
+
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
## 2026-07-16 — Handmatige aanvulling compacter
|
| 11 |
|
| 12 |
- `Gemiste waarde toevoegen` gebruikt bij openen een compactere invoerregel voor waarde, type en vervanging.
|
RISK_REGISTER.md
CHANGED
|
@@ -73,7 +73,7 @@ Risk:
|
|
| 73 |
DOCX metadata, comments, tracked changes, headers, footers or hidden content contain sensitive data that is not scrubbed or cleaned.
|
| 74 |
```
|
| 75 |
|
| 76 |
-
DOCX hygiene audit remains report-only. Export grouping keeps audit details available and does not imply a clean-DOCX guarantee. The review debug collapse line explicitly keeps audit details available rather than removing them. The Phase 6 synthetic DOCX case now records header/footer findings and the existing main-document-only reinsert boundary as reproducible evidence for the document-hygiene hardening package. Gap triage classifies this as document fidelity and reinsert scope and routes it to `SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING`.
|
| 77 |
|
| 78 |
---
|
| 79 |
|
|
@@ -134,7 +134,7 @@ Risk:
|
|
| 134 |
Users may assume PDF support means complete restored PDF reinsert or OCR, while the approved scope is text-based extraction to restored TXT only.
|
| 135 |
```
|
| 136 |
|
| 137 |
-
PDF limitations must remain clear in export/reinsert copy. The Phase 6 text-based PDF case verifies the current restored-TXT-only path and explicitly records that restored PDF and OCR are unsupported. Gap triage retains this as an explicit product boundary and does not authorize OCR or restored-PDF work.
|
| 138 |
|
| 139 |
---
|
| 140 |
|
|
|
|
| 73 |
DOCX metadata, comments, tracked changes, headers, footers or hidden content contain sensitive data that is not scrubbed or cleaned.
|
| 74 |
```
|
| 75 |
|
| 76 |
+
DOCX hygiene audit remains report-only. Export grouping keeps audit details available and does not imply a clean-DOCX guarantee. The review debug collapse line explicitly keeps audit details available rather than removing them. The Phase 6 synthetic DOCX case now records header/footer findings and the existing main-document-only reinsert boundary as reproducible evidence for the document-hygiene hardening package. Gap triage classifies this as document fidelity and reinsert scope and routes it to `SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING`. The hardening package now restores placeholders in existing DOCX header and footer XML parts while retaining hygiene reporting and explicit unsupported-part warnings.
|
| 77 |
|
| 78 |
---
|
| 79 |
|
|
|
|
| 134 |
Users may assume PDF support means complete restored PDF reinsert or OCR, while the approved scope is text-based extraction to restored TXT only.
|
| 135 |
```
|
| 136 |
|
| 137 |
+
PDF limitations must remain clear in export/reinsert copy. The Phase 6 text-based PDF case verifies the current restored-TXT-only path and explicitly records that restored PDF and OCR are unsupported. Gap triage retains this as an explicit product boundary and does not authorize OCR or restored-PDF work. Document-fidelity hardening preserves that boundary unchanged.
|
| 138 |
|
| 139 |
---
|
| 140 |
|
handover/workpackages/20260717_2230_mvp_document_hygiene_fidelity_hardening.md
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Handover — SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
|
| 2 |
+
|
| 3 |
+
Repository worked in: solidprivacy-nl/scrub
|
| 4 |
+
|
| 5 |
+
## Workpackage title
|
| 6 |
+
|
| 7 |
+
SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
|
| 8 |
+
|
| 9 |
+
## Status
|
| 10 |
+
|
| 11 |
+
Completed / ready for app verification.
|
| 12 |
+
|
| 13 |
+
## Files added
|
| 14 |
+
|
| 15 |
+
- `mvp_document_fidelity_report.py`
|
| 16 |
+
- `scripts/run_mvp_document_hygiene_fidelity_report.py`
|
| 17 |
+
- `tests/test_mvp_document_hygiene_fidelity_hardening.py`
|
| 18 |
+
- `tests/test_mvp_document_fidelity_report.py`
|
| 19 |
+
- `tests/test_mvp_document_fidelity_ui_copy.py`
|
| 20 |
+
- `output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json`
|
| 21 |
+
- `handover/workpackages/20260717_2230_mvp_document_hygiene_fidelity_hardening.md`
|
| 22 |
+
|
| 23 |
+
## Files changed
|
| 24 |
+
|
| 25 |
+
- `scrub_key_document_reinsert.py`
|
| 26 |
+
- `reinsert_mode_ui.py`
|
| 27 |
+
- `mvp_phase6_document_cases.py`
|
| 28 |
+
- `tests/test_mvp_phase6_e2e_synthetic_validation_matrix.py`
|
| 29 |
+
- `CHANGELOG.md`
|
| 30 |
+
- `WORKPACKAGES.md`
|
| 31 |
+
- `RELEASE_NOTES.md`
|
| 32 |
+
- `DECISION_LOG.md`
|
| 33 |
+
- `RISK_REGISTER.md`
|
| 34 |
+
- `workpackage_claims/scrub_wp_mvp_document_hygiene_fidelity_hardening.md`
|
| 35 |
+
|
| 36 |
+
## Tests
|
| 37 |
+
|
| 38 |
+
- Existing TXT/DOCX Scrub Key document-reinsert suite.
|
| 39 |
+
- New body/table/header/footer end-to-end DOCX tests.
|
| 40 |
+
- OOXML package preservation and malformed-header fail-safe tests.
|
| 41 |
+
- Current Phase 6 synthetic matrix tests.
|
| 42 |
+
- Source-level DOCX/PDF capability-copy contract tests.
|
| 43 |
+
- Fidelity report and PDF-boundary tests.
|
| 44 |
+
- Existing DOCX hygiene and document-tool tests.
|
| 45 |
+
- Python compilation and `git diff --check`.
|
| 46 |
+
|
| 47 |
+
## Validation status
|
| 48 |
+
|
| 49 |
+
- DOCX header/footer finding resolved: True.
|
| 50 |
+
- Resolved findings: 1.
|
| 51 |
+
- Remaining findings: 1.
|
| 52 |
+
- PDF TXT-only/no-OCR boundary preserved: True.
|
| 53 |
+
- Local-only deterministic processing retained.
|
| 54 |
+
- Human review remains required.
|
| 55 |
+
- Production readiness remains false.
|
| 56 |
+
|
| 57 |
+
## GitHub Actions status
|
| 58 |
+
|
| 59 |
+
Pending final PR validation.
|
| 60 |
+
|
| 61 |
+
## Hugging Face sync status
|
| 62 |
+
|
| 63 |
+
Pending after merge.
|
| 64 |
+
|
| 65 |
+
## App verification status
|
| 66 |
+
|
| 67 |
+
Required after sync because DOCX reinsert behavior changed.
|
| 68 |
+
|
| 69 |
+
## Remaining risks
|
| 70 |
+
|
| 71 |
+
- Comments, tracked-change-only parts, footnotes/endnotes, text boxes and metadata remain unsupported by reinsert.
|
| 72 |
+
- Placeholders split across Word text nodes remain unsupported.
|
| 73 |
+
- PDF remains restored TXT only; no OCR or restored PDF output.
|
| 74 |
+
- DOCX hygiene audit remains report-only and does not guarantee a clean document.
|
| 75 |
+
|
| 76 |
+
## Next recommended step
|
| 77 |
+
|
| 78 |
+
After app verification, start `SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION`.
|
mvp_document_fidelity_report.py
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Evidence report for MVP Phase 6 document hygiene and fidelity hardening."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import json
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
from typing import Any, Mapping
|
| 8 |
+
|
| 9 |
+
from mvp_phase6_validation_report import build_validation_report
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
REPORT_SCHEMA = "solidprivacy.mvp_phase6_document_fidelity_hardening_report"
|
| 13 |
+
REPORT_SCHEMA_VERSION = "1.0"
|
| 14 |
+
FIXED_TIMESTAMP = "2026-07-17T20:30:00Z"
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def build_document_fidelity_report(
|
| 18 |
+
manifest: Mapping[str, Any],
|
| 19 |
+
baseline_report: Mapping[str, Any],
|
| 20 |
+
triage_report: Mapping[str, Any],
|
| 21 |
+
) -> dict[str, Any]:
|
| 22 |
+
current = build_validation_report(manifest)
|
| 23 |
+
docx_case = next(
|
| 24 |
+
item for item in current["cases"]
|
| 25 |
+
if item["id"] == "docx_mixed_structure_roundtrip"
|
| 26 |
+
)
|
| 27 |
+
pdf_case = next(
|
| 28 |
+
item for item in current["cases"]
|
| 29 |
+
if item["id"] == "pdf_text_based_to_txt_reinsert"
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
docx_resolved = (
|
| 33 |
+
docx_case["roundtrip_complete"] is True
|
| 34 |
+
and docx_case["residual_placeholders"] == []
|
| 35 |
+
and docx_case["header_footer_roundtrip_values_present"] is True
|
| 36 |
+
and docx_case["audit_expectations_met"] is True
|
| 37 |
+
and any(
|
| 38 |
+
part.startswith("word/header")
|
| 39 |
+
for part in docx_case["processed_parts"]
|
| 40 |
+
)
|
| 41 |
+
and any(
|
| 42 |
+
part.startswith("word/footer")
|
| 43 |
+
for part in docx_case["processed_parts"]
|
| 44 |
+
)
|
| 45 |
+
)
|
| 46 |
+
pdf_boundary_preserved = (
|
| 47 |
+
pdf_case["roundtrip_text_equal"] is True
|
| 48 |
+
and pdf_case["reinsert_output_type"] == "txt"
|
| 49 |
+
and pdf_case["restored_pdf_supported"] is False
|
| 50 |
+
and pdf_case["ocr_supported"] is False
|
| 51 |
+
)
|
| 52 |
+
|
| 53 |
+
baseline_docx_gap = next(
|
| 54 |
+
item
|
| 55 |
+
for item in baseline_report["evidence_gaps"]
|
| 56 |
+
if item["category"] == "known_docx_reinsert_limitation"
|
| 57 |
+
)
|
| 58 |
+
triage_docx = next(
|
| 59 |
+
item
|
| 60 |
+
for item in triage_report["classifications"]
|
| 61 |
+
if item["source_category"] == "known_docx_reinsert_limitation"
|
| 62 |
+
)
|
| 63 |
+
|
| 64 |
+
resolved_findings = []
|
| 65 |
+
if docx_resolved:
|
| 66 |
+
resolved_findings.append(
|
| 67 |
+
{
|
| 68 |
+
"source_case_id": baseline_docx_gap["case_id"],
|
| 69 |
+
"source_category": baseline_docx_gap["category"],
|
| 70 |
+
"triage_classification": triage_docx["classification"],
|
| 71 |
+
"resolution": "DOCX body, tables, headers and footers restore deterministically from the existing Scrub Key.",
|
| 72 |
+
"resolved_placeholders": docx_case[
|
| 73 |
+
"resolved_header_footer_placeholders"
|
| 74 |
+
],
|
| 75 |
+
"processed_parts": docx_case["processed_parts"],
|
| 76 |
+
}
|
| 77 |
+
)
|
| 78 |
+
|
| 79 |
+
remaining_findings = [
|
| 80 |
+
{
|
| 81 |
+
"case_id": pdf_case["id"],
|
| 82 |
+
"category": "known_pdf_reinsert_limitation",
|
| 83 |
+
"status": "explicit_product_boundary",
|
| 84 |
+
"reinsert_output_type": "txt",
|
| 85 |
+
"restored_pdf_supported": False,
|
| 86 |
+
"ocr_supported": False,
|
| 87 |
+
}
|
| 88 |
+
]
|
| 89 |
+
|
| 90 |
+
return {
|
| 91 |
+
"schema": REPORT_SCHEMA,
|
| 92 |
+
"schema_version": REPORT_SCHEMA_VERSION,
|
| 93 |
+
"generated_at": FIXED_TIMESTAMP,
|
| 94 |
+
"source_baseline_report": (
|
| 95 |
+
"output/validation/mvp_phase6_synthetic_validation_report.json"
|
| 96 |
+
),
|
| 97 |
+
"source_triage_report": (
|
| 98 |
+
"output/validation/mvp_phase6_false_negative_gap_triage.json"
|
| 99 |
+
),
|
| 100 |
+
"synthetic_data_only": True,
|
| 101 |
+
"human_review_required": True,
|
| 102 |
+
"production_ready": False,
|
| 103 |
+
"production_readiness_claim": False,
|
| 104 |
+
"local_only": True,
|
| 105 |
+
"ai_processing": False,
|
| 106 |
+
"cloud_processing": False,
|
| 107 |
+
"ocr_processing": False,
|
| 108 |
+
"docx_header_footer_reinsert_resolved": docx_resolved,
|
| 109 |
+
"pdf_boundary_preserved": pdf_boundary_preserved,
|
| 110 |
+
"resolved_finding_count": len(resolved_findings),
|
| 111 |
+
"remaining_finding_count": len(remaining_findings),
|
| 112 |
+
"resolved_findings": resolved_findings,
|
| 113 |
+
"remaining_findings": remaining_findings,
|
| 114 |
+
"current_validation_report": current,
|
| 115 |
+
"next_recommended_package": (
|
| 116 |
+
"SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION"
|
| 117 |
+
),
|
| 118 |
+
}
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def write_document_fidelity_report(
|
| 122 |
+
manifest: Mapping[str, Any],
|
| 123 |
+
baseline_report: Mapping[str, Any],
|
| 124 |
+
triage_report: Mapping[str, Any],
|
| 125 |
+
output_path: str | Path,
|
| 126 |
+
) -> dict[str, Any]:
|
| 127 |
+
report = build_document_fidelity_report(
|
| 128 |
+
manifest,
|
| 129 |
+
baseline_report,
|
| 130 |
+
triage_report,
|
| 131 |
+
)
|
| 132 |
+
output = Path(output_path)
|
| 133 |
+
output.parent.mkdir(parents=True, exist_ok=True)
|
| 134 |
+
output.write_text(
|
| 135 |
+
json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
| 136 |
+
encoding="utf-8",
|
| 137 |
+
)
|
| 138 |
+
return report
|
mvp_phase6_document_cases.py
CHANGED
|
@@ -49,6 +49,21 @@ def build_docx_case(case: Mapping[str, Any]) -> bytes:
|
|
| 49 |
return output.getvalue()
|
| 50 |
|
| 51 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
|
| 53 |
source_docx = build_docx_case(case)
|
| 54 |
source_upload = UploadedBytes(f"{case['id']}.docx", source_docx)
|
|
@@ -75,18 +90,31 @@ def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
|
|
| 75 |
str(case.get("header_text") or ""),
|
| 76 |
str(case.get("footer_text") or ""),
|
| 77 |
]
|
| 78 |
-
|
| 79 |
placeholder
|
| 80 |
for original, placeholder in common["replacement_map"].items()
|
| 81 |
if any(original in container for container in header_footer_text)
|
| 82 |
)
|
| 83 |
residuals = detect_placeholders(restored_text)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
body_values = [
|
| 85 |
original
|
| 86 |
for original in common["replacement_map"]
|
| 87 |
if not any(original in container for container in header_footer_text)
|
| 88 |
]
|
| 89 |
body_values_present = all(value in restored_text for value in body_values)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
expected_findings = set(case.get("expected_hygiene_findings", []) or [])
|
| 92 |
observed_findings = {
|
|
@@ -95,19 +123,21 @@ def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
|
|
| 95 |
if finding.get("id")
|
| 96 |
}
|
| 97 |
audit_met = expected_findings.issubset(observed_findings)
|
| 98 |
-
|
| 99 |
limitation_copy = " ".join(reinsert_result.get("limitations", [])).lower()
|
| 100 |
limitation_met = all(
|
| 101 |
-
|
| 102 |
for term in case.get("expected_reinsert_limitations", []) or []
|
| 103 |
)
|
|
|
|
| 104 |
|
| 105 |
status = (
|
| 106 |
"pass_with_known_limitations"
|
| 107 |
if body_values_present
|
|
|
|
| 108 |
and audit_met
|
| 109 |
-
and residuals_documented
|
| 110 |
and limitation_met
|
|
|
|
| 111 |
and not common["scrub_key_validation_issues"]
|
| 112 |
else "fail"
|
| 113 |
)
|
|
@@ -127,14 +157,17 @@ def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
|
|
| 127 |
"scrub_key_validation_issues": common["scrub_key_validation_issues"],
|
| 128 |
"reinsert_replacement_count": reinsert_result.get("replacement_count"),
|
| 129 |
"body_roundtrip_values_present": body_values_present,
|
| 130 |
-
"
|
|
|
|
|
|
|
| 131 |
"residual_placeholders": residuals,
|
|
|
|
| 132 |
"hygiene_severity": hygiene_report.get("summary", {}).get("severity"),
|
| 133 |
"hygiene_findings": sorted(observed_findings),
|
| 134 |
"audit_expectations_met": audit_met,
|
| 135 |
"known_limitations": list(reinsert_result.get("limitations", [])),
|
| 136 |
"limitation_expectations_met": limitation_met,
|
| 137 |
-
"roundtrip_complete":
|
| 138 |
"local_only": True,
|
| 139 |
"ai_processing": False,
|
| 140 |
"cloud_processing": False,
|
|
|
|
| 49 |
return output.getvalue()
|
| 50 |
|
| 51 |
|
| 52 |
+
def _expected_scope_is_supported(
|
| 53 |
+
term: str,
|
| 54 |
+
processed_parts: list[str],
|
| 55 |
+
limitation_copy: str,
|
| 56 |
+
) -> bool:
|
| 57 |
+
normalized = str(term).lower()
|
| 58 |
+
if normalized in limitation_copy:
|
| 59 |
+
return True
|
| 60 |
+
if normalized in {"header", "headers"}:
|
| 61 |
+
return any(part.startswith("word/header") for part in processed_parts)
|
| 62 |
+
if normalized in {"footer", "footers"}:
|
| 63 |
+
return any(part.startswith("word/footer") for part in processed_parts)
|
| 64 |
+
return False
|
| 65 |
+
|
| 66 |
+
|
| 67 |
def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
|
| 68 |
source_docx = build_docx_case(case)
|
| 69 |
source_upload = UploadedBytes(f"{case['id']}.docx", source_docx)
|
|
|
|
| 90 |
str(case.get("header_text") or ""),
|
| 91 |
str(case.get("footer_text") or ""),
|
| 92 |
]
|
| 93 |
+
formerly_expected_residuals = sorted(
|
| 94 |
placeholder
|
| 95 |
for original, placeholder in common["replacement_map"].items()
|
| 96 |
if any(original in container for container in header_footer_text)
|
| 97 |
)
|
| 98 |
residuals = detect_placeholders(restored_text)
|
| 99 |
+
resolved_header_footer_placeholders = sorted(
|
| 100 |
+
placeholder
|
| 101 |
+
for placeholder in formerly_expected_residuals
|
| 102 |
+
if placeholder not in residuals
|
| 103 |
+
)
|
| 104 |
body_values = [
|
| 105 |
original
|
| 106 |
for original in common["replacement_map"]
|
| 107 |
if not any(original in container for container in header_footer_text)
|
| 108 |
]
|
| 109 |
body_values_present = all(value in restored_text for value in body_values)
|
| 110 |
+
header_footer_values = [
|
| 111 |
+
original
|
| 112 |
+
for original in common["replacement_map"]
|
| 113 |
+
if any(original in container for container in header_footer_text)
|
| 114 |
+
]
|
| 115 |
+
header_footer_values_present = all(
|
| 116 |
+
value in restored_text for value in header_footer_values
|
| 117 |
+
)
|
| 118 |
|
| 119 |
expected_findings = set(case.get("expected_hygiene_findings", []) or [])
|
| 120 |
observed_findings = {
|
|
|
|
| 123 |
if finding.get("id")
|
| 124 |
}
|
| 125 |
audit_met = expected_findings.issubset(observed_findings)
|
| 126 |
+
processed_parts = list(reinsert_result.get("processed_parts", []))
|
| 127 |
limitation_copy = " ".join(reinsert_result.get("limitations", [])).lower()
|
| 128 |
limitation_met = all(
|
| 129 |
+
_expected_scope_is_supported(term, processed_parts, limitation_copy)
|
| 130 |
for term in case.get("expected_reinsert_limitations", []) or []
|
| 131 |
)
|
| 132 |
+
roundtrip_complete = not residuals
|
| 133 |
|
| 134 |
status = (
|
| 135 |
"pass_with_known_limitations"
|
| 136 |
if body_values_present
|
| 137 |
+
and header_footer_values_present
|
| 138 |
and audit_met
|
|
|
|
| 139 |
and limitation_met
|
| 140 |
+
and roundtrip_complete
|
| 141 |
and not common["scrub_key_validation_issues"]
|
| 142 |
else "fail"
|
| 143 |
)
|
|
|
|
| 157 |
"scrub_key_validation_issues": common["scrub_key_validation_issues"],
|
| 158 |
"reinsert_replacement_count": reinsert_result.get("replacement_count"),
|
| 159 |
"body_roundtrip_values_present": body_values_present,
|
| 160 |
+
"header_footer_roundtrip_values_present": header_footer_values_present,
|
| 161 |
+
"expected_residual_placeholders": formerly_expected_residuals,
|
| 162 |
+
"resolved_header_footer_placeholders": resolved_header_footer_placeholders,
|
| 163 |
"residual_placeholders": residuals,
|
| 164 |
+
"processed_parts": processed_parts,
|
| 165 |
"hygiene_severity": hygiene_report.get("summary", {}).get("severity"),
|
| 166 |
"hygiene_findings": sorted(observed_findings),
|
| 167 |
"audit_expectations_met": audit_met,
|
| 168 |
"known_limitations": list(reinsert_result.get("limitations", [])),
|
| 169 |
"limitation_expectations_met": limitation_met,
|
| 170 |
+
"roundtrip_complete": roundtrip_complete,
|
| 171 |
"local_only": True,
|
| 172 |
"ai_processing": False,
|
| 173 |
"cloud_processing": False,
|
output/validation/mvp_document_fidelity_pr_validation.json
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ai_processing": false,
|
| 3 |
+
"branch": "scrub-mvp-document-hygiene-fidelity-hardening",
|
| 4 |
+
"cloud_processing": false,
|
| 5 |
+
"compile_passed": true,
|
| 6 |
+
"diff_check_passed": true,
|
| 7 |
+
"generated_at": "2026-07-17T13:13:02.792165+00:00",
|
| 8 |
+
"head_sha": "4b07e520296f14b7bde2323cad8ce28bbb4d79c5",
|
| 9 |
+
"human_review_required": true,
|
| 10 |
+
"main_synchronized": true,
|
| 11 |
+
"ocr_processing": false,
|
| 12 |
+
"overall_passed": true,
|
| 13 |
+
"production_ready": false,
|
| 14 |
+
"pytest_passed": true,
|
| 15 |
+
"schema": "solidprivacy.mvp_document_fidelity_pr_validation",
|
| 16 |
+
"schema_version": "1.0",
|
| 17 |
+
"synthetic_data_only": true,
|
| 18 |
+
"tests": [
|
| 19 |
+
"tests/test_mvp_document_hygiene_fidelity_hardening.py",
|
| 20 |
+
"tests/test_mvp_document_fidelity_report.py",
|
| 21 |
+
"tests/test_mvp_document_fidelity_ui_copy.py",
|
| 22 |
+
"tests/test_mvp_document_fidelity_pr_final_contracts.py",
|
| 23 |
+
"tests/test_scrub_key_document_reinsert.py",
|
| 24 |
+
"tests/test_mvp_phase6_e2e_synthetic_validation_matrix.py",
|
| 25 |
+
"tests/test_mvp_phase6_false_negative_gap_triage.py",
|
| 26 |
+
"tests/test_docx_hygiene_audit.py"
|
| 27 |
+
]
|
| 28 |
+
}
|
output/validation/mvp_document_fidelity_pr_validation.log
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
main_synchronized=true
|
| 2 |
+
...................................................... [100%]
|
| 3 |
+
54 passed in 5.56s
|
| 4 |
+
|
| 5 |
+
compile_status=0
|
| 6 |
+
pytest_status=0
|
| 7 |
+
diff_status=0
|
| 8 |
+
overall_status=0
|
output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json
ADDED
|
@@ -0,0 +1,312 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ai_processing": false,
|
| 3 |
+
"cloud_processing": false,
|
| 4 |
+
"current_validation_report": {
|
| 5 |
+
"ai_processing": false,
|
| 6 |
+
"case_count": 3,
|
| 7 |
+
"cases": [
|
| 8 |
+
{
|
| 9 |
+
"ai_processing": false,
|
| 10 |
+
"cloud_processing": false,
|
| 11 |
+
"detection": {
|
| 12 |
+
"ai_processing": false,
|
| 13 |
+
"analyzer_result_count": 13,
|
| 14 |
+
"candidate_result_count": 0,
|
| 15 |
+
"cloud_processing": false,
|
| 16 |
+
"context_preservation_met": true,
|
| 17 |
+
"detection_expectations_met": true,
|
| 18 |
+
"expected_values": [
|
| 19 |
+
"DOS-2026-778899",
|
| 20 |
+
"ZK-WOON-55091",
|
| 21 |
+
"CLNT-2026-0042",
|
| 22 |
+
"mila.testerveld@example.com",
|
| 23 |
+
"0612345678",
|
| 24 |
+
"NL91ABNA0417164300"
|
| 25 |
+
],
|
| 26 |
+
"local_only": true,
|
| 27 |
+
"missing_expected_values": [],
|
| 28 |
+
"preserved_terms": [
|
| 29 |
+
"minderjarige",
|
| 30 |
+
"arts",
|
| 31 |
+
"getuige"
|
| 32 |
+
],
|
| 33 |
+
"removed_preserved_terms": [],
|
| 34 |
+
"rows": [
|
| 35 |
+
{
|
| 36 |
+
"end": 235,
|
| 37 |
+
"entity_type": "NL_PHONE_NUMBER",
|
| 38 |
+
"score": 1.0,
|
| 39 |
+
"source": "DutchPhoneRecognizer",
|
| 40 |
+
"start": 225,
|
| 41 |
+
"text": "0612345678"
|
| 42 |
+
},
|
| 43 |
+
{
|
| 44 |
+
"end": 168,
|
| 45 |
+
"entity_type": "NL_POSTCODE",
|
| 46 |
+
"score": 0.8,
|
| 47 |
+
"source": "PatternRecognizer",
|
| 48 |
+
"start": 161,
|
| 49 |
+
"text": "1234 AB"
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"end": 261,
|
| 53 |
+
"entity_type": "NL_IBAN",
|
| 54 |
+
"score": 0.9,
|
| 55 |
+
"source": "PatternRecognizer",
|
| 56 |
+
"start": 243,
|
| 57 |
+
"text": "NL91ABNA0417164300"
|
| 58 |
+
},
|
| 59 |
+
{
|
| 60 |
+
"end": 176,
|
| 61 |
+
"entity_type": "NL_ADDRESS",
|
| 62 |
+
"score": 0.66,
|
| 63 |
+
"source": "PatternRecognizer",
|
| 64 |
+
"start": 127,
|
| 65 |
+
"text": "Testerveld woont aan Testlaan 12, 1234 AB Testdam"
|
| 66 |
+
},
|
| 67 |
+
{
|
| 68 |
+
"end": 81,
|
| 69 |
+
"entity_type": "NL_LEGAL_CASE_NUMBER",
|
| 70 |
+
"score": 0.86,
|
| 71 |
+
"source": "NL_LEGAL_CASE_NUMBER_capture_recognizer",
|
| 72 |
+
"start": 68,
|
| 73 |
+
"text": "ZK-WOON-55091"
|
| 74 |
+
},
|
| 75 |
+
{
|
| 76 |
+
"end": 55,
|
| 77 |
+
"entity_type": "NL_DOSSIER_NUMBER",
|
| 78 |
+
"score": 0.83,
|
| 79 |
+
"source": "NL_DOSSIER_NUMBER_capture_recognizer",
|
| 80 |
+
"start": 40,
|
| 81 |
+
"text": "DOS-2026-778899"
|
| 82 |
+
},
|
| 83 |
+
{
|
| 84 |
+
"end": 105,
|
| 85 |
+
"entity_type": "NL_CLIENT_NUMBER",
|
| 86 |
+
"score": 0.86,
|
| 87 |
+
"source": "NL_CLIENT_NUMBER_capture_recognizer",
|
| 88 |
+
"start": 96,
|
| 89 |
+
"text": "CLNT-2026"
|
| 90 |
+
},
|
| 91 |
+
{
|
| 92 |
+
"end": 137,
|
| 93 |
+
"entity_type": "NL_LEGAL_PARTY_NAME",
|
| 94 |
+
"score": 0.79,
|
| 95 |
+
"source": "NL_LEGAL_PARTY_NAME_capture_recognizer",
|
| 96 |
+
"start": 122,
|
| 97 |
+
"text": "Mila Testerveld"
|
| 98 |
+
},
|
| 99 |
+
{
|
| 100 |
+
"end": 24,
|
| 101 |
+
"entity_type": "NL_COURT_OR_AUTHORITY",
|
| 102 |
+
"score": 0.7,
|
| 103 |
+
"source": "PatternRecognizer",
|
| 104 |
+
"start": 7,
|
| 105 |
+
"text": "Rechtbank Testdam"
|
| 106 |
+
},
|
| 107 |
+
{
|
| 108 |
+
"end": 110,
|
| 109 |
+
"entity_type": "NL_CLIENT_REFERENCE",
|
| 110 |
+
"score": 0.9,
|
| 111 |
+
"source": "DutchContextualReferenceRecognizer",
|
| 112 |
+
"start": 96,
|
| 113 |
+
"text": "CLNT-2026-0042"
|
| 114 |
+
},
|
| 115 |
+
{
|
| 116 |
+
"end": 213,
|
| 117 |
+
"entity_type": "EMAIL_ADDRESS",
|
| 118 |
+
"score": 1.0,
|
| 119 |
+
"source": "EmailRecognizer",
|
| 120 |
+
"start": 186,
|
| 121 |
+
"text": "mila.testerveld@example.com"
|
| 122 |
+
}
|
| 123 |
+
]
|
| 124 |
+
},
|
| 125 |
+
"document_type": "txt",
|
| 126 |
+
"export_import_types": [
|
| 127 |
+
"docx",
|
| 128 |
+
"pdf"
|
| 129 |
+
],
|
| 130 |
+
"export_placeholders_visible": true,
|
| 131 |
+
"id": "txt_legal_core_roundtrip",
|
| 132 |
+
"known_limitations": [],
|
| 133 |
+
"local_only": true,
|
| 134 |
+
"manual_additions": [
|
| 135 |
+
{
|
| 136 |
+
"is_valid": true,
|
| 137 |
+
"manual_type": "Persoon",
|
| 138 |
+
"message": "",
|
| 139 |
+
"placeholder": "[PERSOON_HANDMATIG_01]",
|
| 140 |
+
"value": "Mila Testerveld"
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"is_valid": true,
|
| 144 |
+
"manual_type": "Anders",
|
| 145 |
+
"message": "",
|
| 146 |
+
"placeholder": "[WAARDE_HANDMATIG_01]",
|
| 147 |
+
"value": "lantaarnbloem"
|
| 148 |
+
}
|
| 149 |
+
],
|
| 150 |
+
"reinsert_replacement_count": 8,
|
| 151 |
+
"review_row_count": 8,
|
| 152 |
+
"roundtrip_exact": true,
|
| 153 |
+
"scrub_key_item_count": 8,
|
| 154 |
+
"scrub_key_validation_issues": [],
|
| 155 |
+
"scrub_report_requires_manual_review": true,
|
| 156 |
+
"scrubbed_text": "Aan de Rechtbank Testdam.\nDossiernummer [DOSSIERNUMMER_01].\nZaaknummer [ZAAKREFERENTIE_01].\nClientnummer [CLIENTNUMMER_01].\nVerzoeker [PERSOON_HANDMATIG_01] woont aan Testlaan 12, 1234 AB Testdam.\nE-mail: [EMAIL_01].\nTelefoon: [TELEFOON_01].\nIBAN: [IBAN_01].\nDe minderjarige sprak met de arts en de getuige.\nInterne testwaarde [WAARDE_HANDMATIG_01] bleef onopgemerkt.",
|
| 157 |
+
"source_import_type": "txt",
|
| 158 |
+
"status": "pass"
|
| 159 |
+
},
|
| 160 |
+
{
|
| 161 |
+
"ai_processing": false,
|
| 162 |
+
"audit_expectations_met": true,
|
| 163 |
+
"body_roundtrip_values_present": true,
|
| 164 |
+
"cloud_processing": false,
|
| 165 |
+
"document_type": "docx",
|
| 166 |
+
"expected_residual_placeholders": [
|
| 167 |
+
"[CLIENT_REFERENTIE_01]",
|
| 168 |
+
"[OVERIGE_REFERENTIE_01]"
|
| 169 |
+
],
|
| 170 |
+
"header_footer_roundtrip_values_present": true,
|
| 171 |
+
"hygiene_findings": [
|
| 172 |
+
"footers_detected",
|
| 173 |
+
"headers_detected"
|
| 174 |
+
],
|
| 175 |
+
"hygiene_severity": "high",
|
| 176 |
+
"id": "docx_mixed_structure_roundtrip",
|
| 177 |
+
"known_limitations": [
|
| 178 |
+
"DOCX helper processes word/document.xml plus word/header*.xml and word/footer*.xml text nodes.",
|
| 179 |
+
"Normal body paragraphs, body tables and header/footer text are supported.",
|
| 180 |
+
"Headers and footers are processed together with the main document body.",
|
| 181 |
+
"Placeholders split across multiple Word runs/text nodes are not restored in this version.",
|
| 182 |
+
"Comments, tracked-change-only parts, footnotes/endnotes, text boxes and metadata are not processed in this version."
|
| 183 |
+
],
|
| 184 |
+
"limitation_expectations_met": true,
|
| 185 |
+
"local_only": true,
|
| 186 |
+
"processed_parts": [
|
| 187 |
+
"word/document.xml",
|
| 188 |
+
"word/header1.xml",
|
| 189 |
+
"word/header2.xml",
|
| 190 |
+
"word/header3.xml",
|
| 191 |
+
"word/footer1.xml",
|
| 192 |
+
"word/footer2.xml",
|
| 193 |
+
"word/footer3.xml"
|
| 194 |
+
],
|
| 195 |
+
"reinsert_replacement_count": 6,
|
| 196 |
+
"residual_placeholders": [],
|
| 197 |
+
"resolved_header_footer_placeholders": [
|
| 198 |
+
"[CLIENT_REFERENTIE_01]",
|
| 199 |
+
"[OVERIGE_REFERENTIE_01]"
|
| 200 |
+
],
|
| 201 |
+
"restored_import_type": "docx",
|
| 202 |
+
"restored_text": "Verzoeker Mila Testerveld behandelt zaak ZK-WOON-55091.\nDe minderjarige blijft als juridische rol leesbaar.\nDossier DOS-2026-778899\nClient CLNT-2026-0042\nVERTROUWELIJK CL-FAM-55201\nPagina TEST-FOOTER-001",
|
| 203 |
+
"review_row_count": 6,
|
| 204 |
+
"roundtrip_complete": true,
|
| 205 |
+
"scrub_key_item_count": 6,
|
| 206 |
+
"scrub_key_validation_issues": [],
|
| 207 |
+
"scrubbed_import_type": "docx",
|
| 208 |
+
"scrubbed_text": "Verzoeker [PERSOON_01] behandelt zaak [ZAAKREFERENTIE_01].\nDe minderjarige blijft als juridische rol leesbaar.\nDossier [DOSSIERNUMMER_01]\nClient [CLIENTNUMMER_01]\nVERTROUWELIJK [CLIENT_REFERENTIE_01]\nPagina [OVERIGE_REFERENTIE_01]",
|
| 209 |
+
"source_import_type": "docx",
|
| 210 |
+
"source_text": "Verzoeker Mila Testerveld behandelt zaak ZK-WOON-55091.\nDe minderjarige blijft als juridische rol leesbaar.\nDossier DOS-2026-778899\nClient CLNT-2026-0042\nVERTROUWELIJK CL-FAM-55201\nPagina TEST-FOOTER-001",
|
| 211 |
+
"status": "pass_with_known_limitations"
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"ai_processing": false,
|
| 215 |
+
"cloud_processing": false,
|
| 216 |
+
"document_type": "pdf",
|
| 217 |
+
"id": "pdf_text_based_to_txt_reinsert",
|
| 218 |
+
"known_limitations": [
|
| 219 |
+
"PDF reinsert produces restored TXT only.",
|
| 220 |
+
"No restored PDF output is supported.",
|
| 221 |
+
"OCR and scanned/image-only PDFs are unsupported."
|
| 222 |
+
],
|
| 223 |
+
"limitation_contract_met": true,
|
| 224 |
+
"local_only": true,
|
| 225 |
+
"ocr_supported": false,
|
| 226 |
+
"reinsert_output_type": "txt",
|
| 227 |
+
"reinsert_replacement_count": 3,
|
| 228 |
+
"restored_pdf_supported": false,
|
| 229 |
+
"restored_text": "Zaak ZK-WOON-55091 betreft Mila Testerveld. Dossier DOS-2026-778899.",
|
| 230 |
+
"review_row_count": 3,
|
| 231 |
+
"roundtrip_text_equal": true,
|
| 232 |
+
"scrub_key_item_count": 3,
|
| 233 |
+
"scrub_key_validation_issues": [],
|
| 234 |
+
"scrubbed_import_type": "pdf",
|
| 235 |
+
"scrubbed_text": "Zaak [ZAAKREFERENTIE_01] betreft [PERSOON_01]. Dossier [DOSSIERNUMMER_01].",
|
| 236 |
+
"source_import_type": "pdf",
|
| 237 |
+
"source_text": "Zaak ZK-WOON-55091 betreft Mila Testerveld. Dossier DOS-2026-778899.",
|
| 238 |
+
"status": "pass_with_known_limitations"
|
| 239 |
+
}
|
| 240 |
+
],
|
| 241 |
+
"cloud_processing": false,
|
| 242 |
+
"evidence_gap_count": 1,
|
| 243 |
+
"evidence_gaps": [
|
| 244 |
+
{
|
| 245 |
+
"case_id": "pdf_text_based_to_txt_reinsert",
|
| 246 |
+
"category": "known_pdf_reinsert_limitation",
|
| 247 |
+
"value": "restored TXT only; no OCR or restored PDF"
|
| 248 |
+
}
|
| 249 |
+
],
|
| 250 |
+
"failing_case_count": 0,
|
| 251 |
+
"failing_cases": [],
|
| 252 |
+
"generated_at": "2026-07-17T18:20:00Z",
|
| 253 |
+
"human_review_required": true,
|
| 254 |
+
"local_only": true,
|
| 255 |
+
"manifest_schema": "solidprivacy.mvp_phase6_validation_matrix",
|
| 256 |
+
"manifest_schema_version": "1.0",
|
| 257 |
+
"next_recommended_package": "SCRUB-WP_MVP_FALSE_NEGATIVE_GAP_TRIAGE",
|
| 258 |
+
"ocr_processing": false,
|
| 259 |
+
"production_readiness_claim": false,
|
| 260 |
+
"production_ready": false,
|
| 261 |
+
"schema": "solidprivacy.mvp_phase6_validation_report",
|
| 262 |
+
"schema_version": "1.0",
|
| 263 |
+
"synthetic_data_only": true
|
| 264 |
+
},
|
| 265 |
+
"docx_header_footer_reinsert_resolved": true,
|
| 266 |
+
"generated_at": "2026-07-17T20:30:00Z",
|
| 267 |
+
"human_review_required": true,
|
| 268 |
+
"local_only": true,
|
| 269 |
+
"next_recommended_package": "SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION",
|
| 270 |
+
"ocr_processing": false,
|
| 271 |
+
"pdf_boundary_preserved": true,
|
| 272 |
+
"production_readiness_claim": false,
|
| 273 |
+
"production_ready": false,
|
| 274 |
+
"remaining_finding_count": 1,
|
| 275 |
+
"remaining_findings": [
|
| 276 |
+
{
|
| 277 |
+
"case_id": "pdf_text_based_to_txt_reinsert",
|
| 278 |
+
"category": "known_pdf_reinsert_limitation",
|
| 279 |
+
"ocr_supported": false,
|
| 280 |
+
"reinsert_output_type": "txt",
|
| 281 |
+
"restored_pdf_supported": false,
|
| 282 |
+
"status": "explicit_product_boundary"
|
| 283 |
+
}
|
| 284 |
+
],
|
| 285 |
+
"resolved_finding_count": 1,
|
| 286 |
+
"resolved_findings": [
|
| 287 |
+
{
|
| 288 |
+
"processed_parts": [
|
| 289 |
+
"word/document.xml",
|
| 290 |
+
"word/header1.xml",
|
| 291 |
+
"word/header2.xml",
|
| 292 |
+
"word/header3.xml",
|
| 293 |
+
"word/footer1.xml",
|
| 294 |
+
"word/footer2.xml",
|
| 295 |
+
"word/footer3.xml"
|
| 296 |
+
],
|
| 297 |
+
"resolution": "DOCX body, tables, headers and footers restore deterministically from the existing Scrub Key.",
|
| 298 |
+
"resolved_placeholders": [
|
| 299 |
+
"[CLIENT_REFERENTIE_01]",
|
| 300 |
+
"[OVERIGE_REFERENTIE_01]"
|
| 301 |
+
],
|
| 302 |
+
"source_case_id": "docx_mixed_structure_roundtrip",
|
| 303 |
+
"source_category": "known_docx_reinsert_limitation",
|
| 304 |
+
"triage_classification": "document_fidelity_and_reinsert_scope"
|
| 305 |
+
}
|
| 306 |
+
],
|
| 307 |
+
"schema": "solidprivacy.mvp_phase6_document_fidelity_hardening_report",
|
| 308 |
+
"schema_version": "1.0",
|
| 309 |
+
"source_baseline_report": "output/validation/mvp_phase6_synthetic_validation_report.json",
|
| 310 |
+
"source_triage_report": "output/validation/mvp_phase6_false_negative_gap_triage.json",
|
| 311 |
+
"synthetic_data_only": true
|
| 312 |
+
}
|
reinsert_mode_ui.py
CHANGED
|
@@ -209,9 +209,9 @@ def render_reinsert_mode() -> None:
|
|
| 209 |
st.warning(CONFIDENTIAL_OUTPUT_WARNING)
|
| 210 |
st.caption("Upload een DOCX-bestand met placeholders. Er wordt geen AI- of cloudverwerking gebruikt.")
|
| 211 |
st.info(
|
| 212 |
-
"
|
| 213 |
-
"
|
| 214 |
-
"over meerdere tekstfragmenten zijn gesplitst worden nog niet volledig ondersteund."
|
| 215 |
)
|
| 216 |
docx_reinsert_file = st.file_uploader(
|
| 217 |
"Upload een DOCX-bestand met placeholders",
|
|
|
|
| 209 |
st.warning(CONFIDENTIAL_OUTPUT_WARNING)
|
| 210 |
st.caption("Upload een DOCX-bestand met placeholders. Er wordt geen AI- of cloudverwerking gebruikt.")
|
| 211 |
st.info(
|
| 212 |
+
"DOCX-terugzetten ondersteunt normale documenttekst, tabellen en bestaande kop- en voetteksten. "
|
| 213 |
+
"Opmerkingen, bijgehouden wijzigingen, voetnoten/eindnoten, tekstvakken, metadata en placeholders "
|
| 214 |
+
"die door Word over meerdere tekstfragmenten zijn gesplitst worden nog niet volledig ondersteund."
|
| 215 |
)
|
| 216 |
docx_reinsert_file = st.file_uploader(
|
| 217 |
"Upload een DOCX-bestand met placeholders",
|
scripts/run_mvp_document_hygiene_fidelity_report.py
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
from mvp_document_fidelity_report import write_document_fidelity_report
|
| 7 |
+
from mvp_phase6_validation_manifest import load_validation_manifest
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
MANIFEST = Path("test_cases/mvp_phase6/validation_manifest.json")
|
| 11 |
+
BASELINE = Path("output/validation/mvp_phase6_synthetic_validation_report.json")
|
| 12 |
+
TRIAGE = Path("output/validation/mvp_phase6_false_negative_gap_triage.json")
|
| 13 |
+
OUTPUT = Path(
|
| 14 |
+
"output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json"
|
| 15 |
+
)
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def _json(path: Path) -> dict:
|
| 19 |
+
return json.loads(path.read_text(encoding="utf-8"))
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def main() -> None:
|
| 23 |
+
report = write_document_fidelity_report(
|
| 24 |
+
load_validation_manifest(MANIFEST),
|
| 25 |
+
_json(BASELINE),
|
| 26 |
+
_json(TRIAGE),
|
| 27 |
+
OUTPUT,
|
| 28 |
+
)
|
| 29 |
+
print(
|
| 30 |
+
f"Wrote {OUTPUT}: resolved={report['resolved_finding_count']}, "
|
| 31 |
+
f"remaining={report['remaining_finding_count']}."
|
| 32 |
+
)
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
if __name__ == "__main__":
|
| 36 |
+
main()
|
scrub_key_document_reinsert.py
CHANGED
|
@@ -5,11 +5,15 @@ for TXT and DOCX inputs. They deliberately avoid Streamlit UI integration,
|
|
| 5 |
PDF handling, AI calls, remote processing, file-system persistence and export
|
| 6 |
semantic changes.
|
| 7 |
|
| 8 |
-
DOCX support
|
| 9 |
-
-
|
| 10 |
-
-
|
| 11 |
-
-
|
| 12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
"""
|
| 14 |
|
| 15 |
from __future__ import annotations
|
|
@@ -17,19 +21,23 @@ from __future__ import annotations
|
|
| 17 |
from io import BytesIO
|
| 18 |
from typing import Any
|
| 19 |
from zipfile import ZIP_DEFLATED, BadZipFile, ZipFile
|
|
|
|
| 20 |
import xml.etree.ElementTree as ET
|
| 21 |
|
| 22 |
from scrub_key_reinsert import reinsert_from_scrub_key
|
| 23 |
|
| 24 |
DOCX_MAIN_DOCUMENT = "word/document.xml"
|
|
|
|
|
|
|
| 25 |
WORDPROCESSINGML_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
| 26 |
XML_NS = "http://www.w3.org/XML/1998/namespace"
|
| 27 |
|
| 28 |
DOCX_LIMITATIONS = [
|
| 29 |
-
"DOCX helper processes
|
| 30 |
-
"Normal body paragraphs
|
|
|
|
| 31 |
"Placeholders split across multiple Word runs/text nodes are not restored in this version.",
|
| 32 |
-
"
|
| 33 |
]
|
| 34 |
|
| 35 |
ET.register_namespace("w", WORDPROCESSINGML_NS)
|
|
@@ -48,11 +56,7 @@ def _with_document_metadata(result: dict[str, Any], document_type: str) -> dict[
|
|
| 48 |
|
| 49 |
|
| 50 |
def reinsert_text_document(text: str, scrub_key: dict[str, Any]) -> dict[str, Any]:
|
| 51 |
-
"""Reinsert placeholders in plain text and return restored text plus audit.
|
| 52 |
-
|
| 53 |
-
This is a document-level wrapper around ``reinsert_from_scrub_key`` for plain
|
| 54 |
-
text/TXT use cases. It does not mutate the supplied Scrub Key.
|
| 55 |
-
"""
|
| 56 |
result = _with_document_metadata(reinsert_from_scrub_key(text, scrub_key), "txt")
|
| 57 |
result["content"] = result.get("text", "")
|
| 58 |
result["limitations"] = []
|
|
@@ -64,11 +68,7 @@ def reinsert_txt_bytes(
|
|
| 64 |
scrub_key: dict[str, Any],
|
| 65 |
encoding: str = "utf-8",
|
| 66 |
) -> dict[str, Any]:
|
| 67 |
-
"""Decode TXT bytes, reinsert placeholders and return restored text/bytes.
|
| 68 |
-
|
| 69 |
-
Decoding is strict by default so invalid input is reported instead of being
|
| 70 |
-
silently changed. No file-system writes are performed.
|
| 71 |
-
"""
|
| 72 |
if not isinstance(content, (bytes, bytearray)):
|
| 73 |
result = reinsert_text_document("", scrub_key)
|
| 74 |
result["validation_issues"] = ["TXT content must be bytes."]
|
|
@@ -111,6 +111,9 @@ def _docx_validation_result(content: bytes, message: str) -> dict[str, Any]:
|
|
| 111 |
"cloud_processing": False,
|
| 112 |
"limitations": list(DOCX_LIMITATIONS),
|
| 113 |
"unsupported_parts": list(DOCX_LIMITATIONS),
|
|
|
|
|
|
|
|
|
|
| 114 |
}
|
| 115 |
|
| 116 |
|
|
@@ -118,23 +121,50 @@ def _word_text_nodes(root: ET.Element) -> list[ET.Element]:
|
|
| 118 |
return list(root.iter(f"{{{WORDPROCESSINGML_NS}}}t"))
|
| 119 |
|
| 120 |
|
| 121 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
output = BytesIO()
|
| 123 |
-
with ZipFile(BytesIO(content), "r") as source_package, ZipFile(
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
for entry in source_package.infolist():
|
| 125 |
-
data =
|
|
|
|
|
|
|
| 126 |
target_package.writestr(entry, data)
|
| 127 |
return output.getvalue()
|
| 128 |
|
| 129 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
def reinsert_docx_bytes(content: bytes, scrub_key: dict[str, Any]) -> dict[str, Any]:
|
| 131 |
-
"""Reinsert placeholders in
|
| 132 |
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
restored DOCX bytes and an audit summary. The input bytes and Scrub Key are
|
| 137 |
-
not mutated.
|
| 138 |
"""
|
| 139 |
if not isinstance(content, (bytes, bytearray)):
|
| 140 |
return _docx_validation_result(b"", "DOCX content must be bytes.")
|
|
@@ -149,30 +179,67 @@ def reinsert_docx_bytes(content: bytes, scrub_key: dict[str, Any]) -> dict[str,
|
|
| 149 |
original_content,
|
| 150 |
"DOCX package is missing word/document.xml.",
|
| 151 |
)
|
| 152 |
-
|
|
|
|
| 153 |
except BadZipFile:
|
| 154 |
-
return _docx_validation_result(
|
|
|
|
|
|
|
|
|
|
| 155 |
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
except ET.ParseError as exc:
|
| 159 |
-
return _docx_validation_result(original_content, f"DOCX document.xml could not be parsed: {exc}")
|
| 160 |
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
if not audit_result.get("validation_issues"):
|
| 166 |
-
for
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 175 |
audit_result["docx_bytes"] = restored_docx
|
| 176 |
audit_result["limitations"] = list(DOCX_LIMITATIONS)
|
| 177 |
audit_result["unsupported_parts"] = list(DOCX_LIMITATIONS)
|
|
|
|
|
|
|
|
|
|
| 178 |
return audit_result
|
|
|
|
| 5 |
PDF handling, AI calls, remote processing, file-system persistence and export
|
| 6 |
semantic changes.
|
| 7 |
|
| 8 |
+
DOCX support covers text nodes in:
|
| 9 |
+
- ``word/document.xml``;
|
| 10 |
+
- ``word/header*.xml``;
|
| 11 |
+
- ``word/footer*.xml``.
|
| 12 |
+
|
| 13 |
+
Normal body paragraphs, body tables and header/footer text are restored. The
|
| 14 |
+
helper still does not restore placeholders split across multiple Word text
|
| 15 |
+
nodes, comments, tracked-change-only parts, footnotes/endnotes, text boxes or
|
| 16 |
+
metadata.
|
| 17 |
"""
|
| 18 |
|
| 19 |
from __future__ import annotations
|
|
|
|
| 21 |
from io import BytesIO
|
| 22 |
from typing import Any
|
| 23 |
from zipfile import ZIP_DEFLATED, BadZipFile, ZipFile
|
| 24 |
+
import re
|
| 25 |
import xml.etree.ElementTree as ET
|
| 26 |
|
| 27 |
from scrub_key_reinsert import reinsert_from_scrub_key
|
| 28 |
|
| 29 |
DOCX_MAIN_DOCUMENT = "word/document.xml"
|
| 30 |
+
DOCX_HEADER_RE = re.compile(r"^word/header[^/]*\.xml$")
|
| 31 |
+
DOCX_FOOTER_RE = re.compile(r"^word/footer[^/]*\.xml$")
|
| 32 |
WORDPROCESSINGML_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
| 33 |
XML_NS = "http://www.w3.org/XML/1998/namespace"
|
| 34 |
|
| 35 |
DOCX_LIMITATIONS = [
|
| 36 |
+
"DOCX helper processes word/document.xml plus word/header*.xml and word/footer*.xml text nodes.",
|
| 37 |
+
"Normal body paragraphs, body tables and header/footer text are supported.",
|
| 38 |
+
"Headers and footers are processed together with the main document body.",
|
| 39 |
"Placeholders split across multiple Word runs/text nodes are not restored in this version.",
|
| 40 |
+
"Comments, tracked-change-only parts, footnotes/endnotes, text boxes and metadata are not processed in this version.",
|
| 41 |
]
|
| 42 |
|
| 43 |
ET.register_namespace("w", WORDPROCESSINGML_NS)
|
|
|
|
| 56 |
|
| 57 |
|
| 58 |
def reinsert_text_document(text: str, scrub_key: dict[str, Any]) -> dict[str, Any]:
|
| 59 |
+
"""Reinsert placeholders in plain text and return restored text plus audit."""
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
result = _with_document_metadata(reinsert_from_scrub_key(text, scrub_key), "txt")
|
| 61 |
result["content"] = result.get("text", "")
|
| 62 |
result["limitations"] = []
|
|
|
|
| 68 |
scrub_key: dict[str, Any],
|
| 69 |
encoding: str = "utf-8",
|
| 70 |
) -> dict[str, Any]:
|
| 71 |
+
"""Decode TXT bytes, reinsert placeholders and return restored text/bytes."""
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
if not isinstance(content, (bytes, bytearray)):
|
| 73 |
result = reinsert_text_document("", scrub_key)
|
| 74 |
result["validation_issues"] = ["TXT content must be bytes."]
|
|
|
|
| 111 |
"cloud_processing": False,
|
| 112 |
"limitations": list(DOCX_LIMITATIONS),
|
| 113 |
"unsupported_parts": list(DOCX_LIMITATIONS),
|
| 114 |
+
"processed_parts": [],
|
| 115 |
+
"processed_part_count": 0,
|
| 116 |
+
"part_texts": {},
|
| 117 |
}
|
| 118 |
|
| 119 |
|
|
|
|
| 121 |
return list(root.iter(f"{{{WORDPROCESSINGML_NS}}}t"))
|
| 122 |
|
| 123 |
|
| 124 |
+
def _supported_docx_xml_parts(names: set[str]) -> list[str]:
|
| 125 |
+
parts = [DOCX_MAIN_DOCUMENT]
|
| 126 |
+
parts.extend(sorted(name for name in names if DOCX_HEADER_RE.match(name)))
|
| 127 |
+
parts.extend(sorted(name for name in names if DOCX_FOOTER_RE.match(name)))
|
| 128 |
+
return parts
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def _serialize_package_with_xml_parts(
|
| 132 |
+
content: bytes,
|
| 133 |
+
replacement_parts: dict[str, bytes],
|
| 134 |
+
) -> bytes:
|
| 135 |
output = BytesIO()
|
| 136 |
+
with ZipFile(BytesIO(content), "r") as source_package, ZipFile(
|
| 137 |
+
output,
|
| 138 |
+
"w",
|
| 139 |
+
ZIP_DEFLATED,
|
| 140 |
+
) as target_package:
|
| 141 |
for entry in source_package.infolist():
|
| 142 |
+
data = replacement_parts.get(entry.filename)
|
| 143 |
+
if data is None:
|
| 144 |
+
data = source_package.read(entry.filename)
|
| 145 |
target_package.writestr(entry, data)
|
| 146 |
return output.getvalue()
|
| 147 |
|
| 148 |
|
| 149 |
+
def _parse_supported_parts(
|
| 150 |
+
package: ZipFile,
|
| 151 |
+
part_names: list[str],
|
| 152 |
+
) -> tuple[dict[str, ET.Element], str | None]:
|
| 153 |
+
roots: dict[str, ET.Element] = {}
|
| 154 |
+
for part_name in part_names:
|
| 155 |
+
try:
|
| 156 |
+
roots[part_name] = ET.fromstring(package.read(part_name))
|
| 157 |
+
except ET.ParseError as exc:
|
| 158 |
+
return {}, f"DOCX {part_name} could not be parsed: {exc}"
|
| 159 |
+
return roots, None
|
| 160 |
+
|
| 161 |
+
|
| 162 |
def reinsert_docx_bytes(content: bytes, scrub_key: dict[str, Any]) -> dict[str, Any]:
|
| 163 |
+
"""Reinsert placeholders in supported DOCX body/header/footer text nodes.
|
| 164 |
|
| 165 |
+
The input bytes and Scrub Key are not mutated. All supported OOXML parts are
|
| 166 |
+
validated before any output package is produced. Audit counts and unresolved
|
| 167 |
+
placeholders cover the combined supported text surface.
|
|
|
|
|
|
|
| 168 |
"""
|
| 169 |
if not isinstance(content, (bytes, bytearray)):
|
| 170 |
return _docx_validation_result(b"", "DOCX content must be bytes.")
|
|
|
|
| 179 |
original_content,
|
| 180 |
"DOCX package is missing word/document.xml.",
|
| 181 |
)
|
| 182 |
+
part_names = _supported_docx_xml_parts(names)
|
| 183 |
+
roots, parse_issue = _parse_supported_parts(docx_package, part_names)
|
| 184 |
except BadZipFile:
|
| 185 |
+
return _docx_validation_result(
|
| 186 |
+
original_content,
|
| 187 |
+
"DOCX content is not a valid OOXML package.",
|
| 188 |
+
)
|
| 189 |
|
| 190 |
+
if parse_issue:
|
| 191 |
+
return _docx_validation_result(original_content, parse_issue)
|
|
|
|
|
|
|
| 192 |
|
| 193 |
+
part_nodes = {
|
| 194 |
+
part_name: _word_text_nodes(root)
|
| 195 |
+
for part_name, root in roots.items()
|
| 196 |
+
}
|
| 197 |
+
original_part_texts = {
|
| 198 |
+
part_name: "\n".join(node.text or "" for node in nodes)
|
| 199 |
+
for part_name, nodes in part_nodes.items()
|
| 200 |
+
}
|
| 201 |
+
combined_original_text = "\n".join(
|
| 202 |
+
original_part_texts[part_name]
|
| 203 |
+
for part_name in part_names
|
| 204 |
+
)
|
| 205 |
+
audit_result = _with_document_metadata(
|
| 206 |
+
reinsert_from_scrub_key(combined_original_text, scrub_key),
|
| 207 |
+
"docx",
|
| 208 |
+
)
|
| 209 |
|
| 210 |
if not audit_result.get("validation_issues"):
|
| 211 |
+
for nodes in part_nodes.values():
|
| 212 |
+
for node in nodes:
|
| 213 |
+
node_result = reinsert_from_scrub_key(node.text or "", scrub_key)
|
| 214 |
+
node.text = node_result.get("text", "")
|
| 215 |
+
|
| 216 |
+
restored_part_texts = {
|
| 217 |
+
part_name: "\n".join(node.text or "" for node in part_nodes[part_name])
|
| 218 |
+
for part_name in part_names
|
| 219 |
+
}
|
| 220 |
+
restored_xml_parts = {
|
| 221 |
+
part_name: ET.tostring(
|
| 222 |
+
roots[part_name],
|
| 223 |
+
encoding="utf-8",
|
| 224 |
+
xml_declaration=True,
|
| 225 |
+
)
|
| 226 |
+
for part_name in part_names
|
| 227 |
+
}
|
| 228 |
+
restored_docx = _serialize_package_with_xml_parts(
|
| 229 |
+
original_content,
|
| 230 |
+
restored_xml_parts,
|
| 231 |
+
)
|
| 232 |
+
combined_restored_text = "\n".join(
|
| 233 |
+
restored_part_texts[part_name]
|
| 234 |
+
for part_name in part_names
|
| 235 |
+
)
|
| 236 |
+
|
| 237 |
+
audit_result["text"] = combined_restored_text
|
| 238 |
+
audit_result["content"] = combined_restored_text
|
| 239 |
audit_result["docx_bytes"] = restored_docx
|
| 240 |
audit_result["limitations"] = list(DOCX_LIMITATIONS)
|
| 241 |
audit_result["unsupported_parts"] = list(DOCX_LIMITATIONS)
|
| 242 |
+
audit_result["processed_parts"] = list(part_names)
|
| 243 |
+
audit_result["processed_part_count"] = len(part_names)
|
| 244 |
+
audit_result["part_texts"] = restored_part_texts
|
| 245 |
return audit_result
|
tests/test_mvp_document_fidelity_pr_final_contracts.py
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
ROOT = Path(__file__).resolve().parents[1]
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
def test_temporary_document_fidelity_patch_scripts_are_absent() -> None:
|
| 8 |
+
assert not (ROOT / "scripts" / "patch_docx_reinsert_capability_copy_only.py").exists()
|
| 9 |
+
assert not (ROOT / "scripts" / "finalize_mvp_document_fidelity_pr_cleanup.py").exists()
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def test_document_fidelity_governance_evidence_has_no_duplicate_lines() -> None:
|
| 13 |
+
changelog = (ROOT / "CHANGELOG.md").read_text(encoding="utf-8")
|
| 14 |
+
workpackages = (ROOT / "WORKPACKAGES.md").read_text(encoding="utf-8")
|
| 15 |
+
handover = (
|
| 16 |
+
ROOT
|
| 17 |
+
/ "handover"
|
| 18 |
+
/ "workpackages"
|
| 19 |
+
/ "20260717_2230_mvp_document_hygiene_fidelity_hardening.md"
|
| 20 |
+
).read_text(encoding="utf-8")
|
| 21 |
+
|
| 22 |
+
assert changelog.count("- `tests/test_mvp_document_fidelity_ui_copy.py`") == 1
|
| 23 |
+
assert changelog.count(
|
| 24 |
+
"The DOCX reinsert capability copy now matches the supported body/table/header/footer scope."
|
| 25 |
+
) == 1
|
| 26 |
+
assert workpackages.count(
|
| 27 |
+
"Aligned the existing DOCX reinsert information copy with the supported body/table/header/footer scope without adding controls."
|
| 28 |
+
) == 1
|
| 29 |
+
assert handover.count("- `tests/test_mvp_document_fidelity_ui_copy.py`") == 1
|
| 30 |
+
assert handover.count("Source-level DOCX/PDF capability-copy contract tests.") == 1
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def test_final_claim_preserves_verification_and_product_boundaries() -> None:
|
| 34 |
+
claim = (
|
| 35 |
+
ROOT
|
| 36 |
+
/ "workpackage_claims"
|
| 37 |
+
/ "scrub_wp_mvp_document_hygiene_fidelity_hardening.md"
|
| 38 |
+
).read_text(encoding="utf-8")
|
| 39 |
+
|
| 40 |
+
assert "DOCX header/footer reinsert resolved: true" in claim
|
| 41 |
+
assert "No OCR or restored-PDF implementation" in claim
|
| 42 |
+
assert "App verification required after Actions and Hugging Face sync" in claim
|
tests/test_mvp_document_fidelity_report.py
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
from mvp_document_fidelity_report import (
|
| 7 |
+
build_document_fidelity_report,
|
| 8 |
+
write_document_fidelity_report,
|
| 9 |
+
)
|
| 10 |
+
from mvp_phase6_validation_manifest import load_validation_manifest
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
ROOT = Path(__file__).resolve().parents[1]
|
| 14 |
+
MANIFEST = ROOT / "test_cases" / "mvp_phase6" / "validation_manifest.json"
|
| 15 |
+
BASELINE = ROOT / "output" / "validation" / "mvp_phase6_synthetic_validation_report.json"
|
| 16 |
+
TRIAGE = ROOT / "output" / "validation" / "mvp_phase6_false_negative_gap_triage.json"
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
def _json(path: Path) -> dict:
|
| 20 |
+
return json.loads(path.read_text(encoding="utf-8"))
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def _report() -> dict:
|
| 24 |
+
return build_document_fidelity_report(
|
| 25 |
+
load_validation_manifest(MANIFEST),
|
| 26 |
+
_json(BASELINE),
|
| 27 |
+
_json(TRIAGE),
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def test_hardening_report_closes_docx_header_footer_finding() -> None:
|
| 32 |
+
report = _report()
|
| 33 |
+
|
| 34 |
+
assert report["schema"] == (
|
| 35 |
+
"solidprivacy.mvp_phase6_document_fidelity_hardening_report"
|
| 36 |
+
)
|
| 37 |
+
assert report["schema_version"] == "1.0"
|
| 38 |
+
assert report["docx_header_footer_reinsert_resolved"] is True
|
| 39 |
+
assert report["resolved_finding_count"] == 1
|
| 40 |
+
assert report["remaining_finding_count"] == 1
|
| 41 |
+
|
| 42 |
+
resolved = report["resolved_findings"][0]
|
| 43 |
+
assert resolved["source_category"] == "known_docx_reinsert_limitation"
|
| 44 |
+
assert set(resolved["resolved_placeholders"]) == {
|
| 45 |
+
"[CLIENT_REFERENTIE_01]",
|
| 46 |
+
"[OVERIGE_REFERENTIE_01]",
|
| 47 |
+
}
|
| 48 |
+
assert any(part.startswith("word/header") for part in resolved["processed_parts"])
|
| 49 |
+
assert any(part.startswith("word/footer") for part in resolved["processed_parts"])
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def test_pdf_boundary_remains_explicit_and_unexpanded() -> None:
|
| 53 |
+
report = _report()
|
| 54 |
+
|
| 55 |
+
assert report["pdf_boundary_preserved"] is True
|
| 56 |
+
remaining = report["remaining_findings"][0]
|
| 57 |
+
assert remaining["category"] == "known_pdf_reinsert_limitation"
|
| 58 |
+
assert remaining["status"] == "explicit_product_boundary"
|
| 59 |
+
assert remaining["reinsert_output_type"] == "txt"
|
| 60 |
+
assert remaining["restored_pdf_supported"] is False
|
| 61 |
+
assert remaining["ocr_supported"] is False
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def test_current_matrix_has_no_docx_residual_gap() -> None:
|
| 65 |
+
current = _report()["current_validation_report"]
|
| 66 |
+
docx = next(case for case in current["cases"] if case["document_type"] == "docx")
|
| 67 |
+
|
| 68 |
+
assert current["failing_case_count"] == 0
|
| 69 |
+
assert docx["roundtrip_complete"] is True
|
| 70 |
+
assert docx["residual_placeholders"] == []
|
| 71 |
+
assert docx["header_footer_roundtrip_values_present"] is True
|
| 72 |
+
assert {gap["category"] for gap in current["evidence_gaps"]} == {
|
| 73 |
+
"known_pdf_reinsert_limitation"
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def test_report_retains_privacy_and_claim_boundaries() -> None:
|
| 78 |
+
report = _report()
|
| 79 |
+
|
| 80 |
+
assert report["synthetic_data_only"] is True
|
| 81 |
+
assert report["human_review_required"] is True
|
| 82 |
+
assert report["production_ready"] is False
|
| 83 |
+
assert report["production_readiness_claim"] is False
|
| 84 |
+
assert report["local_only"] is True
|
| 85 |
+
assert report["ai_processing"] is False
|
| 86 |
+
assert report["cloud_processing"] is False
|
| 87 |
+
assert report["ocr_processing"] is False
|
| 88 |
+
assert report["next_recommended_package"] == (
|
| 89 |
+
"SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION"
|
| 90 |
+
)
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def test_report_writer_is_deterministic(tmp_path: Path) -> None:
|
| 94 |
+
output = tmp_path / "hardening.json"
|
| 95 |
+
first = write_document_fidelity_report(
|
| 96 |
+
load_validation_manifest(MANIFEST),
|
| 97 |
+
_json(BASELINE),
|
| 98 |
+
_json(TRIAGE),
|
| 99 |
+
output,
|
| 100 |
+
)
|
| 101 |
+
first_text = output.read_text(encoding="utf-8")
|
| 102 |
+
second = write_document_fidelity_report(
|
| 103 |
+
load_validation_manifest(MANIFEST),
|
| 104 |
+
_json(BASELINE),
|
| 105 |
+
_json(TRIAGE),
|
| 106 |
+
output,
|
| 107 |
+
)
|
| 108 |
+
|
| 109 |
+
assert first == second
|
| 110 |
+
assert first_text == output.read_text(encoding="utf-8")
|
| 111 |
+
assert json.loads(first_text) == first
|
tests/test_mvp_document_fidelity_ui_copy.py
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import ast
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
ROOT = Path(__file__).resolve().parents[1]
|
| 8 |
+
UI = ROOT / "reinsert_mode_ui.py"
|
| 9 |
+
THIS_TEST = ROOT / "tests" / "test_mvp_document_fidelity_ui_copy.py"
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def test_docx_reinsert_copy_matches_supported_parts() -> None:
|
| 13 |
+
text = UI.read_text(encoding="utf-8")
|
| 14 |
+
|
| 15 |
+
assert "normale documenttekst, tabellen en bestaande kop- en voetteksten" in text
|
| 16 |
+
assert "Opmerkingen, bijgehouden wijzigingen, voetnoten/eindnoten, tekstvakken, metadata" in text
|
| 17 |
+
assert "over meerdere tekstfragmenten zijn gesplitst" in text
|
| 18 |
+
assert "Headers, footers" not in text
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def test_pdf_txt_only_and_no_ocr_copy_remains_unchanged() -> None:
|
| 22 |
+
text = UI.read_text(encoding="utf-8")
|
| 23 |
+
|
| 24 |
+
assert "Deze functie maakt geen herstelde PDF" in text
|
| 25 |
+
assert "OCR niet beschikbaar" in text
|
| 26 |
+
assert "PDF-output: Nee" in text
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def test_copy_contract_does_not_import_streamlit_or_product_ui() -> None:
|
| 30 |
+
tree = ast.parse(THIS_TEST.read_text(encoding="utf-8"))
|
| 31 |
+
imported_roots: set[str] = set()
|
| 32 |
+
|
| 33 |
+
for node in ast.walk(tree):
|
| 34 |
+
if isinstance(node, ast.Import):
|
| 35 |
+
imported_roots.update(alias.name.split(".")[0] for alias in node.names)
|
| 36 |
+
elif isinstance(node, ast.ImportFrom) and node.module:
|
| 37 |
+
imported_roots.add(node.module.split(".")[0])
|
| 38 |
+
|
| 39 |
+
assert "streamlit" not in imported_roots
|
| 40 |
+
assert "reinsert_mode_ui" not in imported_roots
|
tests/test_mvp_document_hygiene_fidelity_hardening.py
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
from io import BytesIO
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
from zipfile import ZIP_DEFLATED, ZipFile
|
| 6 |
+
|
| 7 |
+
from docx import Document
|
| 8 |
+
|
| 9 |
+
from document_tools import anonymized_docx_from_original
|
| 10 |
+
from scrub_key import build_scrub_key
|
| 11 |
+
from scrub_key_document_reinsert import (
|
| 12 |
+
DOCX_LIMITATIONS,
|
| 13 |
+
reinsert_docx_bytes,
|
| 14 |
+
)
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
class UploadedBytes:
|
| 18 |
+
def __init__(self, name: str, content: bytes):
|
| 19 |
+
self.name = name
|
| 20 |
+
self._content = bytes(content)
|
| 21 |
+
|
| 22 |
+
def getvalue(self) -> bytes:
|
| 23 |
+
return self._content
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
ROWS = [
|
| 27 |
+
{
|
| 28 |
+
"original_value": "BETROKKENE-TEST-A",
|
| 29 |
+
"placeholder": "[PERSOON_1]",
|
| 30 |
+
"entity_type": "PERSON",
|
| 31 |
+
"type_label": "Naam",
|
| 32 |
+
"source": "detected",
|
| 33 |
+
"review_status": "auto_detected",
|
| 34 |
+
"include": True,
|
| 35 |
+
"timestamp": "2026-07-17T20:30:00Z",
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"original_value": "RECHTBANK TESTDAM",
|
| 39 |
+
"placeholder": "[ORGANISATIE_01]",
|
| 40 |
+
"entity_type": "ORGANIZATION",
|
| 41 |
+
"type_label": "Organisatie",
|
| 42 |
+
"source": "detected",
|
| 43 |
+
"review_status": "auto_detected",
|
| 44 |
+
"include": True,
|
| 45 |
+
"timestamp": "2026-07-17T20:30:01Z",
|
| 46 |
+
},
|
| 47 |
+
{
|
| 48 |
+
"original_value": "ZAAK-TEST-2026-001",
|
| 49 |
+
"placeholder": "[ZAAKNUMMER_1]",
|
| 50 |
+
"entity_type": "LEGAL_REFERENCE",
|
| 51 |
+
"type_label": "Zaaknummer",
|
| 52 |
+
"source": "manual",
|
| 53 |
+
"review_status": "manual",
|
| 54 |
+
"include": True,
|
| 55 |
+
"timestamp": "2026-07-17T20:30:02Z",
|
| 56 |
+
},
|
| 57 |
+
]
|
| 58 |
+
|
| 59 |
+
REPLACEMENTS = {
|
| 60 |
+
row["original_value"]: row["placeholder"]
|
| 61 |
+
for row in ROWS
|
| 62 |
+
}
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def _scrub_key() -> dict:
|
| 66 |
+
return build_scrub_key(ROWS, document_label="Synthetisch fidelitydossier")
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def _source_docx() -> bytes:
|
| 70 |
+
document = Document()
|
| 71 |
+
document.add_paragraph("Body BETROKKENE-TEST-A")
|
| 72 |
+
table = document.add_table(rows=1, cols=1)
|
| 73 |
+
table.cell(0, 0).text = "Tabel ZAAK-TEST-2026-001"
|
| 74 |
+
section = document.sections[0]
|
| 75 |
+
section.header.paragraphs[0].text = "Header RECHTBANK TESTDAM"
|
| 76 |
+
section.footer.paragraphs[0].text = "Footer ZAAK-TEST-2026-001"
|
| 77 |
+
output = BytesIO()
|
| 78 |
+
document.save(output)
|
| 79 |
+
return output.getvalue()
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def _text_surfaces(docx_bytes: bytes) -> dict[str, str]:
|
| 83 |
+
document = Document(BytesIO(docx_bytes))
|
| 84 |
+
section = document.sections[0]
|
| 85 |
+
return {
|
| 86 |
+
"body": "\n".join(paragraph.text for paragraph in document.paragraphs),
|
| 87 |
+
"table": "\n".join(
|
| 88 |
+
paragraph.text
|
| 89 |
+
for table in document.tables
|
| 90 |
+
for row in table.rows
|
| 91 |
+
for cell in row.cells
|
| 92 |
+
for paragraph in cell.paragraphs
|
| 93 |
+
),
|
| 94 |
+
"header": "\n".join(paragraph.text for paragraph in section.header.paragraphs),
|
| 95 |
+
"footer": "\n".join(paragraph.text for paragraph in section.footer.paragraphs),
|
| 96 |
+
}
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def test_scrub_then_reinsert_restores_body_table_header_and_footer() -> None:
|
| 100 |
+
source = _source_docx()
|
| 101 |
+
scrubbed = anonymized_docx_from_original(
|
| 102 |
+
UploadedBytes("synthetic.docx", source),
|
| 103 |
+
REPLACEMENTS,
|
| 104 |
+
)
|
| 105 |
+
scrubbed_surfaces = _text_surfaces(scrubbed)
|
| 106 |
+
|
| 107 |
+
assert "[PERSOON_1]" in scrubbed_surfaces["body"]
|
| 108 |
+
assert "[ZAAKNUMMER_1]" in scrubbed_surfaces["table"]
|
| 109 |
+
assert "[ORGANISATIE_01]" in scrubbed_surfaces["header"]
|
| 110 |
+
assert "[ZAAKNUMMER_1]" in scrubbed_surfaces["footer"]
|
| 111 |
+
|
| 112 |
+
result = reinsert_docx_bytes(scrubbed, _scrub_key())
|
| 113 |
+
restored = _text_surfaces(result["docx_bytes"])
|
| 114 |
+
|
| 115 |
+
assert restored["body"] == "Body BETROKKENE-TEST-A"
|
| 116 |
+
assert restored["table"] == "Tabel ZAAK-TEST-2026-001"
|
| 117 |
+
assert restored["header"] == "Header RECHTBANK TESTDAM"
|
| 118 |
+
assert restored["footer"] == "Footer ZAAK-TEST-2026-001"
|
| 119 |
+
assert result["replacement_count"] == 4
|
| 120 |
+
assert result["placeholders_not_found"] == []
|
| 121 |
+
assert result["unknown_placeholders"] == []
|
| 122 |
+
assert result["validation_issues"] == []
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def test_result_reports_processed_ooxml_parts() -> None:
|
| 126 |
+
scrubbed = anonymized_docx_from_original(
|
| 127 |
+
UploadedBytes("synthetic.docx", _source_docx()),
|
| 128 |
+
REPLACEMENTS,
|
| 129 |
+
)
|
| 130 |
+
result = reinsert_docx_bytes(scrubbed, _scrub_key())
|
| 131 |
+
|
| 132 |
+
assert "word/document.xml" in result["processed_parts"]
|
| 133 |
+
assert any(part.startswith("word/header") for part in result["processed_parts"])
|
| 134 |
+
assert any(part.startswith("word/footer") for part in result["processed_parts"])
|
| 135 |
+
assert result["processed_part_count"] == len(result["processed_parts"])
|
| 136 |
+
assert set(result["processed_parts"]) == set(result["part_texts"])
|
| 137 |
+
assert result["local_only"] is True
|
| 138 |
+
assert result["ai_processing"] is False
|
| 139 |
+
assert result["cloud_processing"] is False
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def test_comments_tracked_changes_metadata_and_split_nodes_remain_unsupported() -> None:
|
| 143 |
+
limitation_text = " ".join(DOCX_LIMITATIONS).lower()
|
| 144 |
+
|
| 145 |
+
assert "header/footer text are supported" in limitation_text
|
| 146 |
+
assert "split" in limitation_text
|
| 147 |
+
assert "comments" in limitation_text
|
| 148 |
+
assert "tracked" in limitation_text
|
| 149 |
+
assert "metadata" in limitation_text
|
| 150 |
+
assert "not processed" in limitation_text
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
def test_unrelated_package_parts_are_preserved_byte_for_byte() -> None:
|
| 154 |
+
marker_path = "customXml/synthetic-marker.xml"
|
| 155 |
+
marker_bytes = b"<synthetic>KEEP-ME</synthetic>"
|
| 156 |
+
scrubbed = anonymized_docx_from_original(
|
| 157 |
+
UploadedBytes("synthetic.docx", _source_docx()),
|
| 158 |
+
REPLACEMENTS,
|
| 159 |
+
)
|
| 160 |
+
enriched = BytesIO()
|
| 161 |
+
with ZipFile(BytesIO(scrubbed), "r") as original, ZipFile(
|
| 162 |
+
enriched,
|
| 163 |
+
"w",
|
| 164 |
+
ZIP_DEFLATED,
|
| 165 |
+
) as output:
|
| 166 |
+
for entry in original.infolist():
|
| 167 |
+
output.writestr(entry, original.read(entry.filename))
|
| 168 |
+
output.writestr(marker_path, marker_bytes)
|
| 169 |
+
|
| 170 |
+
result = reinsert_docx_bytes(enriched.getvalue(), _scrub_key())
|
| 171 |
+
|
| 172 |
+
with ZipFile(BytesIO(result["docx_bytes"]), "r") as restored:
|
| 173 |
+
assert restored.read(marker_path) == marker_bytes
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
def test_invalid_header_xml_returns_validation_issue_without_partial_output() -> None:
|
| 177 |
+
source = _source_docx()
|
| 178 |
+
broken = BytesIO()
|
| 179 |
+
with ZipFile(BytesIO(source), "r") as original, ZipFile(
|
| 180 |
+
broken,
|
| 181 |
+
"w",
|
| 182 |
+
ZIP_DEFLATED,
|
| 183 |
+
) as output:
|
| 184 |
+
for entry in original.infolist():
|
| 185 |
+
data = original.read(entry.filename)
|
| 186 |
+
if entry.filename.startswith("word/header") and entry.filename.endswith(".xml"):
|
| 187 |
+
data = b"<w:hdr>"
|
| 188 |
+
output.writestr(entry, data)
|
| 189 |
+
|
| 190 |
+
result = reinsert_docx_bytes(broken.getvalue(), _scrub_key())
|
| 191 |
+
|
| 192 |
+
assert result["reinserted"] is False
|
| 193 |
+
assert result["replacement_count"] == 0
|
| 194 |
+
assert result["validation_issues"]
|
| 195 |
+
assert "header" in result["validation_issues"][0].lower()
|
| 196 |
+
assert result["docx_bytes"] == broken.getvalue()
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
def test_pdf_ocr_and_restored_pdf_scope_are_not_added() -> None:
|
| 200 |
+
source = Path("MVP_PHASE6_FALSE_NEGATIVE_GAP_TRIAGE.md").read_text(encoding="utf-8")
|
| 201 |
+
|
| 202 |
+
assert "restored-TXT-only" in source
|
| 203 |
+
assert "No OCR or restored-PDF implementation is authorized" in source
|
tests/test_mvp_phase6_e2e_synthetic_validation_matrix.py
CHANGED
|
@@ -117,7 +117,7 @@ def test_txt_detection_evidence_covers_expected_values_and_preserves_roles() ->
|
|
| 117 |
)
|
| 118 |
|
| 119 |
|
| 120 |
-
def
|
| 121 |
case = _case(_report(), "docx_mixed_structure_roundtrip")
|
| 122 |
|
| 123 |
assert case["status"] == "pass_with_known_limitations"
|
|
@@ -125,16 +125,20 @@ def test_docx_case_records_hygiene_and_header_footer_reinsert_limits() -> None:
|
|
| 125 |
assert case["scrubbed_import_type"] == "docx"
|
| 126 |
assert case["restored_import_type"] == "docx"
|
| 127 |
assert case["body_roundtrip_values_present"] is True
|
|
|
|
| 128 |
assert case["hygiene_severity"] == "high"
|
| 129 |
assert {"headers_detected", "footers_detected"}.issubset(
|
| 130 |
set(case["hygiene_findings"])
|
| 131 |
)
|
| 132 |
assert case["audit_expectations_met"] is True
|
| 133 |
assert case["limitation_expectations_met"] is True
|
| 134 |
-
assert case["roundtrip_complete"] is
|
| 135 |
-
assert
|
| 136 |
-
|
|
|
|
| 137 |
)
|
|
|
|
|
|
|
| 138 |
|
| 139 |
|
| 140 |
def test_pdf_case_preserves_text_roundtrip_and_explicit_txt_only_boundary() -> None:
|
|
@@ -171,10 +175,7 @@ def test_report_is_machine_readable_and_never_claims_production_readiness() -> N
|
|
| 171 |
)
|
| 172 |
|
| 173 |
categories = {item["category"] for item in report["evidence_gaps"]}
|
| 174 |
-
assert categories == {
|
| 175 |
-
"known_docx_reinsert_limitation",
|
| 176 |
-
"known_pdf_reinsert_limitation",
|
| 177 |
-
}
|
| 178 |
|
| 179 |
|
| 180 |
def test_report_writer_is_deterministic(tmp_path: Path) -> None:
|
|
|
|
| 117 |
)
|
| 118 |
|
| 119 |
|
| 120 |
+
def test_docx_case_restores_header_footer_and_retains_hygiene_evidence() -> None:
|
| 121 |
case = _case(_report(), "docx_mixed_structure_roundtrip")
|
| 122 |
|
| 123 |
assert case["status"] == "pass_with_known_limitations"
|
|
|
|
| 125 |
assert case["scrubbed_import_type"] == "docx"
|
| 126 |
assert case["restored_import_type"] == "docx"
|
| 127 |
assert case["body_roundtrip_values_present"] is True
|
| 128 |
+
assert case["header_footer_roundtrip_values_present"] is True
|
| 129 |
assert case["hygiene_severity"] == "high"
|
| 130 |
assert {"headers_detected", "footers_detected"}.issubset(
|
| 131 |
set(case["hygiene_findings"])
|
| 132 |
)
|
| 133 |
assert case["audit_expectations_met"] is True
|
| 134 |
assert case["limitation_expectations_met"] is True
|
| 135 |
+
assert case["roundtrip_complete"] is True
|
| 136 |
+
assert case["residual_placeholders"] == []
|
| 137 |
+
assert set(case["expected_residual_placeholders"]) == set(
|
| 138 |
+
case["resolved_header_footer_placeholders"]
|
| 139 |
)
|
| 140 |
+
assert any(part.startswith("word/header") for part in case["processed_parts"])
|
| 141 |
+
assert any(part.startswith("word/footer") for part in case["processed_parts"])
|
| 142 |
|
| 143 |
|
| 144 |
def test_pdf_case_preserves_text_roundtrip_and_explicit_txt_only_boundary() -> None:
|
|
|
|
| 175 |
)
|
| 176 |
|
| 177 |
categories = {item["category"] for item in report["evidence_gaps"]}
|
| 178 |
+
assert categories == {"known_pdf_reinsert_limitation"}
|
|
|
|
|
|
|
|
|
|
| 179 |
|
| 180 |
|
| 181 |
def test_report_writer_is_deterministic(tmp_path: Path) -> None:
|
workpackage_claims/scrub_wp_mvp_document_hygiene_fidelity_hardening.md
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Workpackage claim — SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
|
| 2 |
+
|
| 3 |
+
Repository: solidprivacy-nl/scrub
|
| 4 |
+
|
| 5 |
+
Workpackage title: SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
|
| 6 |
+
|
| 7 |
+
Status: completed / ready for app verification
|
| 8 |
+
|
| 9 |
+
Claimed by: ChatGPT GitHub worker
|
| 10 |
+
|
| 11 |
+
Claimed at: 2026-07-17 22:30 Europe/Amsterdam
|
| 12 |
+
|
| 13 |
+
Branch: scrub-mvp-document-hygiene-fidelity-hardening
|
| 14 |
+
|
| 15 |
+
Dependencies:
|
| 16 |
+
- SCRUB-WP_MVP_E2E_SYNTHETIC_VALIDATION_MATRIX — merged.
|
| 17 |
+
- SCRUB-WP_MVP_FALSE_NEGATIVE_GAP_TRIAGE — merged.
|
| 18 |
+
|
| 19 |
+
Evidence:
|
| 20 |
+
- DOCX body/table reinsert passes, while scrubbed header/footer placeholders remain after reinsert.
|
| 21 |
+
- DOCX hygiene audit already reports header/footer presence.
|
| 22 |
+
- PDF text roundtrip passes only to restored TXT; restored PDF and OCR remain unsupported product boundaries.
|
| 23 |
+
|
| 24 |
+
Scope:
|
| 25 |
+
- Harden deterministic DOCX reinsert fidelity for existing scrubbed header and footer OOXML parts.
|
| 26 |
+
- Add contract and regression tests covering document body, tables, headers and footers.
|
| 27 |
+
- Preserve explicit unsupported boundaries for comments, tracked changes, metadata, split placeholders, OCR and restored PDF.
|
| 28 |
+
- Update the Phase 6 matrix expectation and evidence after the helper change.
|
| 29 |
+
|
| 30 |
+
Boundaries:
|
| 31 |
+
- Helper-level document reinsert plus capability-copy alignment only; no new Streamlit controls or flow.
|
| 32 |
+
- No recognizer, threshold or replacement changes.
|
| 33 |
+
- No export filename, MIME or Scrub Key schema changes.
|
| 34 |
+
- No comments/tracked changes/metadata cleaning guarantee.
|
| 35 |
+
- No OCR or restored-PDF implementation.
|
| 36 |
+
- Synthetic data only; local-only deterministic processing.
|
| 37 |
+
|
| 38 |
+
Next step:
|
| 39 |
+
- Run final PR validation, merge after green Actions, verify Hugging Face sync and request live DOCX reinsert verification.
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
Hardening result:
|
| 43 |
+
- Completed at: 2026-07-17 22:30 Europe/Amsterdam
|
| 44 |
+
- DOCX header/footer reinsert resolved: true
|
| 45 |
+
- Resolved findings: 1
|
| 46 |
+
- Remaining findings: 1
|
| 47 |
+
- Report: `output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json`
|
| 48 |
+
- Handover: `handover/workpackages/20260717_2230_mvp_document_hygiene_fidelity_hardening.md`
|
| 49 |
+
- App verification required after Actions and Hugging Face sync.
|
| 50 |
+
|
| 51 |
+
Final PR cleanup:
|
| 52 |
+
- Temporary patch/finalizer scripts removed and duplicate governance lines normalized before the final Actions run.
|