solidprivacy commited on
Commit
bffab6a
·
1 Parent(s): 870cb34

Rebuild clean MVP document fidelity hardening branch

Browse files
DECISION_LOG.md CHANGED
@@ -4,6 +4,37 @@ This file records accepted strategic, product and architecture decisions.
4
 
5
  ---
6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  ## 2026-07-17 — D029 — Current Phase 6 matrix does not justify a recognizer fix
8
 
9
  Status: accepted evidence-routing decision
 
4
 
5
  ---
6
 
7
+ ## 2026-07-17 — D030 — Restore existing DOCX header and footer text during deterministic reinsert
8
+
9
+ Status: accepted implementation decision
10
+
11
+ Decision:
12
+
13
+ ```text
14
+ Extend the existing local deterministic DOCX reinsert helper to process WordprocessingML text nodes in word/header*.xml and word/footer*.xml in addition to word/document.xml.
15
+ ```
16
+
17
+ Reason:
18
+
19
+ - The scrubbed DOCX export already replaces reviewed values in body, table, header and footer paragraphs.
20
+ - The Phase 6 matrix showed that reinsert restored body/table values but left header/footer placeholders behind.
21
+ - The gap belongs to document fidelity and reinsert scope, not detection or recognizer behavior.
22
+
23
+ Boundaries:
24
+
25
+ - Process only existing body, header and footer WordprocessingML text nodes.
26
+ - Preserve unrelated OOXML package parts byte-for-byte where they are not rewritten.
27
+ - Do not claim support for comments, tracked-change-only parts, footnotes/endnotes, text boxes, metadata or placeholders split across text nodes.
28
+ - Do not add OCR or restored-PDF behavior.
29
+ - Keep processing local, deterministic and Scrub Key driven.
30
+
31
+ Evidence:
32
+
33
+ - `output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json`
34
+ - `output/validation/mvp_phase6_false_negative_gap_triage.json`
35
+
36
+ ---
37
+
38
  ## 2026-07-17 — D029 — Current Phase 6 matrix does not justify a recognizer fix
39
 
40
  Status: accepted evidence-routing decision
RELEASE_NOTES.md CHANGED
@@ -1,3 +1,12 @@
 
 
 
 
 
 
 
 
 
1
  ## 2026-07-16 — Handmatige aanvulling compacter
2
 
3
  - `Gemiste waarde toevoegen` gebruikt bij openen een compactere invoerregel voor waarde, type en vervanging.
 
1
+ ## 2026-07-17 — DOCX-herstel omvat nu kop- en voetteksten
2
+
3
+ - Bij het terugzetten van originele waarden in een DOCX worden nu ook bestaande kop- en voetteksten meegenomen.
4
+ - Hoofdtekst en tabellen blijven ondersteund.
5
+ - Opmerkingen, alleen-in-wijzigingen aanwezige tekst, voetnoten/eindnoten, tekstvakken, metadata en placeholders die over meerdere Word-tekstnodes zijn verdeeld, blijven buiten deze versie.
6
+ - PDF-herstel blijft beperkt tot herstelde TXT; OCR en een hersteld PDF-bestand zijn niet toegevoegd.
7
+
8
+ ---
9
+
10
  ## 2026-07-16 — Handmatige aanvulling compacter
11
 
12
  - `Gemiste waarde toevoegen` gebruikt bij openen een compactere invoerregel voor waarde, type en vervanging.
RISK_REGISTER.md CHANGED
@@ -73,7 +73,7 @@ Risk:
73
  DOCX metadata, comments, tracked changes, headers, footers or hidden content contain sensitive data that is not scrubbed or cleaned.
74
  ```
75
 
76
- DOCX hygiene audit remains report-only. Export grouping keeps audit details available and does not imply a clean-DOCX guarantee. The review debug collapse line explicitly keeps audit details available rather than removing them. The Phase 6 synthetic DOCX case now records header/footer findings and the existing main-document-only reinsert boundary as reproducible evidence for the document-hygiene hardening package. Gap triage classifies this as document fidelity and reinsert scope and routes it to `SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING`.
77
 
78
  ---
79
 
@@ -134,7 +134,7 @@ Risk:
134
  Users may assume PDF support means complete restored PDF reinsert or OCR, while the approved scope is text-based extraction to restored TXT only.
135
  ```
136
 
137
- PDF limitations must remain clear in export/reinsert copy. The Phase 6 text-based PDF case verifies the current restored-TXT-only path and explicitly records that restored PDF and OCR are unsupported. Gap triage retains this as an explicit product boundary and does not authorize OCR or restored-PDF work.
138
 
139
  ---
140
 
 
73
  DOCX metadata, comments, tracked changes, headers, footers or hidden content contain sensitive data that is not scrubbed or cleaned.
74
  ```
75
 
76
+ DOCX hygiene audit remains report-only. Export grouping keeps audit details available and does not imply a clean-DOCX guarantee. The review debug collapse line explicitly keeps audit details available rather than removing them. The Phase 6 synthetic DOCX case now records header/footer findings and the existing main-document-only reinsert boundary as reproducible evidence for the document-hygiene hardening package. Gap triage classifies this as document fidelity and reinsert scope and routes it to `SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING`. The hardening package now restores placeholders in existing DOCX header and footer XML parts while retaining hygiene reporting and explicit unsupported-part warnings.
77
 
78
  ---
79
 
 
134
  Users may assume PDF support means complete restored PDF reinsert or OCR, while the approved scope is text-based extraction to restored TXT only.
135
  ```
136
 
137
+ PDF limitations must remain clear in export/reinsert copy. The Phase 6 text-based PDF case verifies the current restored-TXT-only path and explicitly records that restored PDF and OCR are unsupported. Gap triage retains this as an explicit product boundary and does not authorize OCR or restored-PDF work. Document-fidelity hardening preserves that boundary unchanged.
138
 
139
  ---
140
 
handover/workpackages/20260717_2230_mvp_document_hygiene_fidelity_hardening.md ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Handover — SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
2
+
3
+ Repository worked in: solidprivacy-nl/scrub
4
+
5
+ ## Workpackage title
6
+
7
+ SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
8
+
9
+ ## Status
10
+
11
+ Completed / ready for app verification.
12
+
13
+ ## Files added
14
+
15
+ - `mvp_document_fidelity_report.py`
16
+ - `scripts/run_mvp_document_hygiene_fidelity_report.py`
17
+ - `tests/test_mvp_document_hygiene_fidelity_hardening.py`
18
+ - `tests/test_mvp_document_fidelity_report.py`
19
+ - `tests/test_mvp_document_fidelity_ui_copy.py`
20
+ - `output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json`
21
+ - `handover/workpackages/20260717_2230_mvp_document_hygiene_fidelity_hardening.md`
22
+
23
+ ## Files changed
24
+
25
+ - `scrub_key_document_reinsert.py`
26
+ - `reinsert_mode_ui.py`
27
+ - `mvp_phase6_document_cases.py`
28
+ - `tests/test_mvp_phase6_e2e_synthetic_validation_matrix.py`
29
+ - `CHANGELOG.md`
30
+ - `WORKPACKAGES.md`
31
+ - `RELEASE_NOTES.md`
32
+ - `DECISION_LOG.md`
33
+ - `RISK_REGISTER.md`
34
+ - `workpackage_claims/scrub_wp_mvp_document_hygiene_fidelity_hardening.md`
35
+
36
+ ## Tests
37
+
38
+ - Existing TXT/DOCX Scrub Key document-reinsert suite.
39
+ - New body/table/header/footer end-to-end DOCX tests.
40
+ - OOXML package preservation and malformed-header fail-safe tests.
41
+ - Current Phase 6 synthetic matrix tests.
42
+ - Source-level DOCX/PDF capability-copy contract tests.
43
+ - Fidelity report and PDF-boundary tests.
44
+ - Existing DOCX hygiene and document-tool tests.
45
+ - Python compilation and `git diff --check`.
46
+
47
+ ## Validation status
48
+
49
+ - DOCX header/footer finding resolved: True.
50
+ - Resolved findings: 1.
51
+ - Remaining findings: 1.
52
+ - PDF TXT-only/no-OCR boundary preserved: True.
53
+ - Local-only deterministic processing retained.
54
+ - Human review remains required.
55
+ - Production readiness remains false.
56
+
57
+ ## GitHub Actions status
58
+
59
+ Pending final PR validation.
60
+
61
+ ## Hugging Face sync status
62
+
63
+ Pending after merge.
64
+
65
+ ## App verification status
66
+
67
+ Required after sync because DOCX reinsert behavior changed.
68
+
69
+ ## Remaining risks
70
+
71
+ - Comments, tracked-change-only parts, footnotes/endnotes, text boxes and metadata remain unsupported by reinsert.
72
+ - Placeholders split across Word text nodes remain unsupported.
73
+ - PDF remains restored TXT only; no OCR or restored PDF output.
74
+ - DOCX hygiene audit remains report-only and does not guarantee a clean document.
75
+
76
+ ## Next recommended step
77
+
78
+ After app verification, start `SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION`.
mvp_document_fidelity_report.py ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Evidence report for MVP Phase 6 document hygiene and fidelity hardening."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import json
6
+ from pathlib import Path
7
+ from typing import Any, Mapping
8
+
9
+ from mvp_phase6_validation_report import build_validation_report
10
+
11
+
12
+ REPORT_SCHEMA = "solidprivacy.mvp_phase6_document_fidelity_hardening_report"
13
+ REPORT_SCHEMA_VERSION = "1.0"
14
+ FIXED_TIMESTAMP = "2026-07-17T20:30:00Z"
15
+
16
+
17
+ def build_document_fidelity_report(
18
+ manifest: Mapping[str, Any],
19
+ baseline_report: Mapping[str, Any],
20
+ triage_report: Mapping[str, Any],
21
+ ) -> dict[str, Any]:
22
+ current = build_validation_report(manifest)
23
+ docx_case = next(
24
+ item for item in current["cases"]
25
+ if item["id"] == "docx_mixed_structure_roundtrip"
26
+ )
27
+ pdf_case = next(
28
+ item for item in current["cases"]
29
+ if item["id"] == "pdf_text_based_to_txt_reinsert"
30
+ )
31
+
32
+ docx_resolved = (
33
+ docx_case["roundtrip_complete"] is True
34
+ and docx_case["residual_placeholders"] == []
35
+ and docx_case["header_footer_roundtrip_values_present"] is True
36
+ and docx_case["audit_expectations_met"] is True
37
+ and any(
38
+ part.startswith("word/header")
39
+ for part in docx_case["processed_parts"]
40
+ )
41
+ and any(
42
+ part.startswith("word/footer")
43
+ for part in docx_case["processed_parts"]
44
+ )
45
+ )
46
+ pdf_boundary_preserved = (
47
+ pdf_case["roundtrip_text_equal"] is True
48
+ and pdf_case["reinsert_output_type"] == "txt"
49
+ and pdf_case["restored_pdf_supported"] is False
50
+ and pdf_case["ocr_supported"] is False
51
+ )
52
+
53
+ baseline_docx_gap = next(
54
+ item
55
+ for item in baseline_report["evidence_gaps"]
56
+ if item["category"] == "known_docx_reinsert_limitation"
57
+ )
58
+ triage_docx = next(
59
+ item
60
+ for item in triage_report["classifications"]
61
+ if item["source_category"] == "known_docx_reinsert_limitation"
62
+ )
63
+
64
+ resolved_findings = []
65
+ if docx_resolved:
66
+ resolved_findings.append(
67
+ {
68
+ "source_case_id": baseline_docx_gap["case_id"],
69
+ "source_category": baseline_docx_gap["category"],
70
+ "triage_classification": triage_docx["classification"],
71
+ "resolution": "DOCX body, tables, headers and footers restore deterministically from the existing Scrub Key.",
72
+ "resolved_placeholders": docx_case[
73
+ "resolved_header_footer_placeholders"
74
+ ],
75
+ "processed_parts": docx_case["processed_parts"],
76
+ }
77
+ )
78
+
79
+ remaining_findings = [
80
+ {
81
+ "case_id": pdf_case["id"],
82
+ "category": "known_pdf_reinsert_limitation",
83
+ "status": "explicit_product_boundary",
84
+ "reinsert_output_type": "txt",
85
+ "restored_pdf_supported": False,
86
+ "ocr_supported": False,
87
+ }
88
+ ]
89
+
90
+ return {
91
+ "schema": REPORT_SCHEMA,
92
+ "schema_version": REPORT_SCHEMA_VERSION,
93
+ "generated_at": FIXED_TIMESTAMP,
94
+ "source_baseline_report": (
95
+ "output/validation/mvp_phase6_synthetic_validation_report.json"
96
+ ),
97
+ "source_triage_report": (
98
+ "output/validation/mvp_phase6_false_negative_gap_triage.json"
99
+ ),
100
+ "synthetic_data_only": True,
101
+ "human_review_required": True,
102
+ "production_ready": False,
103
+ "production_readiness_claim": False,
104
+ "local_only": True,
105
+ "ai_processing": False,
106
+ "cloud_processing": False,
107
+ "ocr_processing": False,
108
+ "docx_header_footer_reinsert_resolved": docx_resolved,
109
+ "pdf_boundary_preserved": pdf_boundary_preserved,
110
+ "resolved_finding_count": len(resolved_findings),
111
+ "remaining_finding_count": len(remaining_findings),
112
+ "resolved_findings": resolved_findings,
113
+ "remaining_findings": remaining_findings,
114
+ "current_validation_report": current,
115
+ "next_recommended_package": (
116
+ "SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION"
117
+ ),
118
+ }
119
+
120
+
121
+ def write_document_fidelity_report(
122
+ manifest: Mapping[str, Any],
123
+ baseline_report: Mapping[str, Any],
124
+ triage_report: Mapping[str, Any],
125
+ output_path: str | Path,
126
+ ) -> dict[str, Any]:
127
+ report = build_document_fidelity_report(
128
+ manifest,
129
+ baseline_report,
130
+ triage_report,
131
+ )
132
+ output = Path(output_path)
133
+ output.parent.mkdir(parents=True, exist_ok=True)
134
+ output.write_text(
135
+ json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
136
+ encoding="utf-8",
137
+ )
138
+ return report
mvp_phase6_document_cases.py CHANGED
@@ -49,6 +49,21 @@ def build_docx_case(case: Mapping[str, Any]) -> bytes:
49
  return output.getvalue()
50
 
51
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
52
  def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
53
  source_docx = build_docx_case(case)
54
  source_upload = UploadedBytes(f"{case['id']}.docx", source_docx)
@@ -75,18 +90,31 @@ def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
75
  str(case.get("header_text") or ""),
76
  str(case.get("footer_text") or ""),
77
  ]
78
- expected_residuals = sorted(
79
  placeholder
80
  for original, placeholder in common["replacement_map"].items()
81
  if any(original in container for container in header_footer_text)
82
  )
83
  residuals = detect_placeholders(restored_text)
 
 
 
 
 
84
  body_values = [
85
  original
86
  for original in common["replacement_map"]
87
  if not any(original in container for container in header_footer_text)
88
  ]
89
  body_values_present = all(value in restored_text for value in body_values)
 
 
 
 
 
 
 
 
90
 
91
  expected_findings = set(case.get("expected_hygiene_findings", []) or [])
92
  observed_findings = {
@@ -95,19 +123,21 @@ def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
95
  if finding.get("id")
96
  }
97
  audit_met = expected_findings.issubset(observed_findings)
98
- residuals_documented = set(expected_residuals).issubset(set(residuals))
99
  limitation_copy = " ".join(reinsert_result.get("limitations", [])).lower()
100
  limitation_met = all(
101
- str(term).lower() in limitation_copy
102
  for term in case.get("expected_reinsert_limitations", []) or []
103
  )
 
104
 
105
  status = (
106
  "pass_with_known_limitations"
107
  if body_values_present
 
108
  and audit_met
109
- and residuals_documented
110
  and limitation_met
 
111
  and not common["scrub_key_validation_issues"]
112
  else "fail"
113
  )
@@ -127,14 +157,17 @@ def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
127
  "scrub_key_validation_issues": common["scrub_key_validation_issues"],
128
  "reinsert_replacement_count": reinsert_result.get("replacement_count"),
129
  "body_roundtrip_values_present": body_values_present,
130
- "expected_residual_placeholders": expected_residuals,
 
 
131
  "residual_placeholders": residuals,
 
132
  "hygiene_severity": hygiene_report.get("summary", {}).get("severity"),
133
  "hygiene_findings": sorted(observed_findings),
134
  "audit_expectations_met": audit_met,
135
  "known_limitations": list(reinsert_result.get("limitations", [])),
136
  "limitation_expectations_met": limitation_met,
137
- "roundtrip_complete": not residuals,
138
  "local_only": True,
139
  "ai_processing": False,
140
  "cloud_processing": False,
 
49
  return output.getvalue()
50
 
51
 
52
+ def _expected_scope_is_supported(
53
+ term: str,
54
+ processed_parts: list[str],
55
+ limitation_copy: str,
56
+ ) -> bool:
57
+ normalized = str(term).lower()
58
+ if normalized in limitation_copy:
59
+ return True
60
+ if normalized in {"header", "headers"}:
61
+ return any(part.startswith("word/header") for part in processed_parts)
62
+ if normalized in {"footer", "footers"}:
63
+ return any(part.startswith("word/footer") for part in processed_parts)
64
+ return False
65
+
66
+
67
  def run_docx_case(case: Mapping[str, Any]) -> dict[str, Any]:
68
  source_docx = build_docx_case(case)
69
  source_upload = UploadedBytes(f"{case['id']}.docx", source_docx)
 
90
  str(case.get("header_text") or ""),
91
  str(case.get("footer_text") or ""),
92
  ]
93
+ formerly_expected_residuals = sorted(
94
  placeholder
95
  for original, placeholder in common["replacement_map"].items()
96
  if any(original in container for container in header_footer_text)
97
  )
98
  residuals = detect_placeholders(restored_text)
99
+ resolved_header_footer_placeholders = sorted(
100
+ placeholder
101
+ for placeholder in formerly_expected_residuals
102
+ if placeholder not in residuals
103
+ )
104
  body_values = [
105
  original
106
  for original in common["replacement_map"]
107
  if not any(original in container for container in header_footer_text)
108
  ]
109
  body_values_present = all(value in restored_text for value in body_values)
110
+ header_footer_values = [
111
+ original
112
+ for original in common["replacement_map"]
113
+ if any(original in container for container in header_footer_text)
114
+ ]
115
+ header_footer_values_present = all(
116
+ value in restored_text for value in header_footer_values
117
+ )
118
 
119
  expected_findings = set(case.get("expected_hygiene_findings", []) or [])
120
  observed_findings = {
 
123
  if finding.get("id")
124
  }
125
  audit_met = expected_findings.issubset(observed_findings)
126
+ processed_parts = list(reinsert_result.get("processed_parts", []))
127
  limitation_copy = " ".join(reinsert_result.get("limitations", [])).lower()
128
  limitation_met = all(
129
+ _expected_scope_is_supported(term, processed_parts, limitation_copy)
130
  for term in case.get("expected_reinsert_limitations", []) or []
131
  )
132
+ roundtrip_complete = not residuals
133
 
134
  status = (
135
  "pass_with_known_limitations"
136
  if body_values_present
137
+ and header_footer_values_present
138
  and audit_met
 
139
  and limitation_met
140
+ and roundtrip_complete
141
  and not common["scrub_key_validation_issues"]
142
  else "fail"
143
  )
 
157
  "scrub_key_validation_issues": common["scrub_key_validation_issues"],
158
  "reinsert_replacement_count": reinsert_result.get("replacement_count"),
159
  "body_roundtrip_values_present": body_values_present,
160
+ "header_footer_roundtrip_values_present": header_footer_values_present,
161
+ "expected_residual_placeholders": formerly_expected_residuals,
162
+ "resolved_header_footer_placeholders": resolved_header_footer_placeholders,
163
  "residual_placeholders": residuals,
164
+ "processed_parts": processed_parts,
165
  "hygiene_severity": hygiene_report.get("summary", {}).get("severity"),
166
  "hygiene_findings": sorted(observed_findings),
167
  "audit_expectations_met": audit_met,
168
  "known_limitations": list(reinsert_result.get("limitations", [])),
169
  "limitation_expectations_met": limitation_met,
170
+ "roundtrip_complete": roundtrip_complete,
171
  "local_only": True,
172
  "ai_processing": False,
173
  "cloud_processing": False,
output/validation/mvp_document_fidelity_pr_validation.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "ai_processing": false,
3
+ "branch": "scrub-mvp-document-hygiene-fidelity-hardening",
4
+ "cloud_processing": false,
5
+ "compile_passed": true,
6
+ "diff_check_passed": true,
7
+ "generated_at": "2026-07-17T13:13:02.792165+00:00",
8
+ "head_sha": "4b07e520296f14b7bde2323cad8ce28bbb4d79c5",
9
+ "human_review_required": true,
10
+ "main_synchronized": true,
11
+ "ocr_processing": false,
12
+ "overall_passed": true,
13
+ "production_ready": false,
14
+ "pytest_passed": true,
15
+ "schema": "solidprivacy.mvp_document_fidelity_pr_validation",
16
+ "schema_version": "1.0",
17
+ "synthetic_data_only": true,
18
+ "tests": [
19
+ "tests/test_mvp_document_hygiene_fidelity_hardening.py",
20
+ "tests/test_mvp_document_fidelity_report.py",
21
+ "tests/test_mvp_document_fidelity_ui_copy.py",
22
+ "tests/test_mvp_document_fidelity_pr_final_contracts.py",
23
+ "tests/test_scrub_key_document_reinsert.py",
24
+ "tests/test_mvp_phase6_e2e_synthetic_validation_matrix.py",
25
+ "tests/test_mvp_phase6_false_negative_gap_triage.py",
26
+ "tests/test_docx_hygiene_audit.py"
27
+ ]
28
+ }
output/validation/mvp_document_fidelity_pr_validation.log ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ main_synchronized=true
2
+ ...................................................... [100%]
3
+ 54 passed in 5.56s
4
+
5
+ compile_status=0
6
+ pytest_status=0
7
+ diff_status=0
8
+ overall_status=0
output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json ADDED
@@ -0,0 +1,312 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "ai_processing": false,
3
+ "cloud_processing": false,
4
+ "current_validation_report": {
5
+ "ai_processing": false,
6
+ "case_count": 3,
7
+ "cases": [
8
+ {
9
+ "ai_processing": false,
10
+ "cloud_processing": false,
11
+ "detection": {
12
+ "ai_processing": false,
13
+ "analyzer_result_count": 13,
14
+ "candidate_result_count": 0,
15
+ "cloud_processing": false,
16
+ "context_preservation_met": true,
17
+ "detection_expectations_met": true,
18
+ "expected_values": [
19
+ "DOS-2026-778899",
20
+ "ZK-WOON-55091",
21
+ "CLNT-2026-0042",
22
+ "mila.testerveld@example.com",
23
+ "0612345678",
24
+ "NL91ABNA0417164300"
25
+ ],
26
+ "local_only": true,
27
+ "missing_expected_values": [],
28
+ "preserved_terms": [
29
+ "minderjarige",
30
+ "arts",
31
+ "getuige"
32
+ ],
33
+ "removed_preserved_terms": [],
34
+ "rows": [
35
+ {
36
+ "end": 235,
37
+ "entity_type": "NL_PHONE_NUMBER",
38
+ "score": 1.0,
39
+ "source": "DutchPhoneRecognizer",
40
+ "start": 225,
41
+ "text": "0612345678"
42
+ },
43
+ {
44
+ "end": 168,
45
+ "entity_type": "NL_POSTCODE",
46
+ "score": 0.8,
47
+ "source": "PatternRecognizer",
48
+ "start": 161,
49
+ "text": "1234 AB"
50
+ },
51
+ {
52
+ "end": 261,
53
+ "entity_type": "NL_IBAN",
54
+ "score": 0.9,
55
+ "source": "PatternRecognizer",
56
+ "start": 243,
57
+ "text": "NL91ABNA0417164300"
58
+ },
59
+ {
60
+ "end": 176,
61
+ "entity_type": "NL_ADDRESS",
62
+ "score": 0.66,
63
+ "source": "PatternRecognizer",
64
+ "start": 127,
65
+ "text": "Testerveld woont aan Testlaan 12, 1234 AB Testdam"
66
+ },
67
+ {
68
+ "end": 81,
69
+ "entity_type": "NL_LEGAL_CASE_NUMBER",
70
+ "score": 0.86,
71
+ "source": "NL_LEGAL_CASE_NUMBER_capture_recognizer",
72
+ "start": 68,
73
+ "text": "ZK-WOON-55091"
74
+ },
75
+ {
76
+ "end": 55,
77
+ "entity_type": "NL_DOSSIER_NUMBER",
78
+ "score": 0.83,
79
+ "source": "NL_DOSSIER_NUMBER_capture_recognizer",
80
+ "start": 40,
81
+ "text": "DOS-2026-778899"
82
+ },
83
+ {
84
+ "end": 105,
85
+ "entity_type": "NL_CLIENT_NUMBER",
86
+ "score": 0.86,
87
+ "source": "NL_CLIENT_NUMBER_capture_recognizer",
88
+ "start": 96,
89
+ "text": "CLNT-2026"
90
+ },
91
+ {
92
+ "end": 137,
93
+ "entity_type": "NL_LEGAL_PARTY_NAME",
94
+ "score": 0.79,
95
+ "source": "NL_LEGAL_PARTY_NAME_capture_recognizer",
96
+ "start": 122,
97
+ "text": "Mila Testerveld"
98
+ },
99
+ {
100
+ "end": 24,
101
+ "entity_type": "NL_COURT_OR_AUTHORITY",
102
+ "score": 0.7,
103
+ "source": "PatternRecognizer",
104
+ "start": 7,
105
+ "text": "Rechtbank Testdam"
106
+ },
107
+ {
108
+ "end": 110,
109
+ "entity_type": "NL_CLIENT_REFERENCE",
110
+ "score": 0.9,
111
+ "source": "DutchContextualReferenceRecognizer",
112
+ "start": 96,
113
+ "text": "CLNT-2026-0042"
114
+ },
115
+ {
116
+ "end": 213,
117
+ "entity_type": "EMAIL_ADDRESS",
118
+ "score": 1.0,
119
+ "source": "EmailRecognizer",
120
+ "start": 186,
121
+ "text": "mila.testerveld@example.com"
122
+ }
123
+ ]
124
+ },
125
+ "document_type": "txt",
126
+ "export_import_types": [
127
+ "docx",
128
+ "pdf"
129
+ ],
130
+ "export_placeholders_visible": true,
131
+ "id": "txt_legal_core_roundtrip",
132
+ "known_limitations": [],
133
+ "local_only": true,
134
+ "manual_additions": [
135
+ {
136
+ "is_valid": true,
137
+ "manual_type": "Persoon",
138
+ "message": "",
139
+ "placeholder": "[PERSOON_HANDMATIG_01]",
140
+ "value": "Mila Testerveld"
141
+ },
142
+ {
143
+ "is_valid": true,
144
+ "manual_type": "Anders",
145
+ "message": "",
146
+ "placeholder": "[WAARDE_HANDMATIG_01]",
147
+ "value": "lantaarnbloem"
148
+ }
149
+ ],
150
+ "reinsert_replacement_count": 8,
151
+ "review_row_count": 8,
152
+ "roundtrip_exact": true,
153
+ "scrub_key_item_count": 8,
154
+ "scrub_key_validation_issues": [],
155
+ "scrub_report_requires_manual_review": true,
156
+ "scrubbed_text": "Aan de Rechtbank Testdam.\nDossiernummer [DOSSIERNUMMER_01].\nZaaknummer [ZAAKREFERENTIE_01].\nClientnummer [CLIENTNUMMER_01].\nVerzoeker [PERSOON_HANDMATIG_01] woont aan Testlaan 12, 1234 AB Testdam.\nE-mail: [EMAIL_01].\nTelefoon: [TELEFOON_01].\nIBAN: [IBAN_01].\nDe minderjarige sprak met de arts en de getuige.\nInterne testwaarde [WAARDE_HANDMATIG_01] bleef onopgemerkt.",
157
+ "source_import_type": "txt",
158
+ "status": "pass"
159
+ },
160
+ {
161
+ "ai_processing": false,
162
+ "audit_expectations_met": true,
163
+ "body_roundtrip_values_present": true,
164
+ "cloud_processing": false,
165
+ "document_type": "docx",
166
+ "expected_residual_placeholders": [
167
+ "[CLIENT_REFERENTIE_01]",
168
+ "[OVERIGE_REFERENTIE_01]"
169
+ ],
170
+ "header_footer_roundtrip_values_present": true,
171
+ "hygiene_findings": [
172
+ "footers_detected",
173
+ "headers_detected"
174
+ ],
175
+ "hygiene_severity": "high",
176
+ "id": "docx_mixed_structure_roundtrip",
177
+ "known_limitations": [
178
+ "DOCX helper processes word/document.xml plus word/header*.xml and word/footer*.xml text nodes.",
179
+ "Normal body paragraphs, body tables and header/footer text are supported.",
180
+ "Headers and footers are processed together with the main document body.",
181
+ "Placeholders split across multiple Word runs/text nodes are not restored in this version.",
182
+ "Comments, tracked-change-only parts, footnotes/endnotes, text boxes and metadata are not processed in this version."
183
+ ],
184
+ "limitation_expectations_met": true,
185
+ "local_only": true,
186
+ "processed_parts": [
187
+ "word/document.xml",
188
+ "word/header1.xml",
189
+ "word/header2.xml",
190
+ "word/header3.xml",
191
+ "word/footer1.xml",
192
+ "word/footer2.xml",
193
+ "word/footer3.xml"
194
+ ],
195
+ "reinsert_replacement_count": 6,
196
+ "residual_placeholders": [],
197
+ "resolved_header_footer_placeholders": [
198
+ "[CLIENT_REFERENTIE_01]",
199
+ "[OVERIGE_REFERENTIE_01]"
200
+ ],
201
+ "restored_import_type": "docx",
202
+ "restored_text": "Verzoeker Mila Testerveld behandelt zaak ZK-WOON-55091.\nDe minderjarige blijft als juridische rol leesbaar.\nDossier DOS-2026-778899\nClient CLNT-2026-0042\nVERTROUWELIJK CL-FAM-55201\nPagina TEST-FOOTER-001",
203
+ "review_row_count": 6,
204
+ "roundtrip_complete": true,
205
+ "scrub_key_item_count": 6,
206
+ "scrub_key_validation_issues": [],
207
+ "scrubbed_import_type": "docx",
208
+ "scrubbed_text": "Verzoeker [PERSOON_01] behandelt zaak [ZAAKREFERENTIE_01].\nDe minderjarige blijft als juridische rol leesbaar.\nDossier [DOSSIERNUMMER_01]\nClient [CLIENTNUMMER_01]\nVERTROUWELIJK [CLIENT_REFERENTIE_01]\nPagina [OVERIGE_REFERENTIE_01]",
209
+ "source_import_type": "docx",
210
+ "source_text": "Verzoeker Mila Testerveld behandelt zaak ZK-WOON-55091.\nDe minderjarige blijft als juridische rol leesbaar.\nDossier DOS-2026-778899\nClient CLNT-2026-0042\nVERTROUWELIJK CL-FAM-55201\nPagina TEST-FOOTER-001",
211
+ "status": "pass_with_known_limitations"
212
+ },
213
+ {
214
+ "ai_processing": false,
215
+ "cloud_processing": false,
216
+ "document_type": "pdf",
217
+ "id": "pdf_text_based_to_txt_reinsert",
218
+ "known_limitations": [
219
+ "PDF reinsert produces restored TXT only.",
220
+ "No restored PDF output is supported.",
221
+ "OCR and scanned/image-only PDFs are unsupported."
222
+ ],
223
+ "limitation_contract_met": true,
224
+ "local_only": true,
225
+ "ocr_supported": false,
226
+ "reinsert_output_type": "txt",
227
+ "reinsert_replacement_count": 3,
228
+ "restored_pdf_supported": false,
229
+ "restored_text": "Zaak ZK-WOON-55091 betreft Mila Testerveld. Dossier DOS-2026-778899.",
230
+ "review_row_count": 3,
231
+ "roundtrip_text_equal": true,
232
+ "scrub_key_item_count": 3,
233
+ "scrub_key_validation_issues": [],
234
+ "scrubbed_import_type": "pdf",
235
+ "scrubbed_text": "Zaak [ZAAKREFERENTIE_01] betreft [PERSOON_01]. Dossier [DOSSIERNUMMER_01].",
236
+ "source_import_type": "pdf",
237
+ "source_text": "Zaak ZK-WOON-55091 betreft Mila Testerveld. Dossier DOS-2026-778899.",
238
+ "status": "pass_with_known_limitations"
239
+ }
240
+ ],
241
+ "cloud_processing": false,
242
+ "evidence_gap_count": 1,
243
+ "evidence_gaps": [
244
+ {
245
+ "case_id": "pdf_text_based_to_txt_reinsert",
246
+ "category": "known_pdf_reinsert_limitation",
247
+ "value": "restored TXT only; no OCR or restored PDF"
248
+ }
249
+ ],
250
+ "failing_case_count": 0,
251
+ "failing_cases": [],
252
+ "generated_at": "2026-07-17T18:20:00Z",
253
+ "human_review_required": true,
254
+ "local_only": true,
255
+ "manifest_schema": "solidprivacy.mvp_phase6_validation_matrix",
256
+ "manifest_schema_version": "1.0",
257
+ "next_recommended_package": "SCRUB-WP_MVP_FALSE_NEGATIVE_GAP_TRIAGE",
258
+ "ocr_processing": false,
259
+ "production_readiness_claim": false,
260
+ "production_ready": false,
261
+ "schema": "solidprivacy.mvp_phase6_validation_report",
262
+ "schema_version": "1.0",
263
+ "synthetic_data_only": true
264
+ },
265
+ "docx_header_footer_reinsert_resolved": true,
266
+ "generated_at": "2026-07-17T20:30:00Z",
267
+ "human_review_required": true,
268
+ "local_only": true,
269
+ "next_recommended_package": "SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION",
270
+ "ocr_processing": false,
271
+ "pdf_boundary_preserved": true,
272
+ "production_readiness_claim": false,
273
+ "production_ready": false,
274
+ "remaining_finding_count": 1,
275
+ "remaining_findings": [
276
+ {
277
+ "case_id": "pdf_text_based_to_txt_reinsert",
278
+ "category": "known_pdf_reinsert_limitation",
279
+ "ocr_supported": false,
280
+ "reinsert_output_type": "txt",
281
+ "restored_pdf_supported": false,
282
+ "status": "explicit_product_boundary"
283
+ }
284
+ ],
285
+ "resolved_finding_count": 1,
286
+ "resolved_findings": [
287
+ {
288
+ "processed_parts": [
289
+ "word/document.xml",
290
+ "word/header1.xml",
291
+ "word/header2.xml",
292
+ "word/header3.xml",
293
+ "word/footer1.xml",
294
+ "word/footer2.xml",
295
+ "word/footer3.xml"
296
+ ],
297
+ "resolution": "DOCX body, tables, headers and footers restore deterministically from the existing Scrub Key.",
298
+ "resolved_placeholders": [
299
+ "[CLIENT_REFERENTIE_01]",
300
+ "[OVERIGE_REFERENTIE_01]"
301
+ ],
302
+ "source_case_id": "docx_mixed_structure_roundtrip",
303
+ "source_category": "known_docx_reinsert_limitation",
304
+ "triage_classification": "document_fidelity_and_reinsert_scope"
305
+ }
306
+ ],
307
+ "schema": "solidprivacy.mvp_phase6_document_fidelity_hardening_report",
308
+ "schema_version": "1.0",
309
+ "source_baseline_report": "output/validation/mvp_phase6_synthetic_validation_report.json",
310
+ "source_triage_report": "output/validation/mvp_phase6_false_negative_gap_triage.json",
311
+ "synthetic_data_only": true
312
+ }
reinsert_mode_ui.py CHANGED
@@ -209,9 +209,9 @@ def render_reinsert_mode() -> None:
209
  st.warning(CONFIDENTIAL_OUTPUT_WARNING)
210
  st.caption("Upload een DOCX-bestand met placeholders. Er wordt geen AI- of cloudverwerking gebruikt.")
211
  st.info(
212
- "Let op: DOCX-terugzetten ondersteunt in deze versie normale documenttekst en tabellen. "
213
- "Headers, footers, opmerkingen, bijgehouden wijzigingen en placeholders die door Word "
214
- "over meerdere tekstfragmenten zijn gesplitst worden nog niet volledig ondersteund."
215
  )
216
  docx_reinsert_file = st.file_uploader(
217
  "Upload een DOCX-bestand met placeholders",
 
209
  st.warning(CONFIDENTIAL_OUTPUT_WARNING)
210
  st.caption("Upload een DOCX-bestand met placeholders. Er wordt geen AI- of cloudverwerking gebruikt.")
211
  st.info(
212
+ "DOCX-terugzetten ondersteunt normale documenttekst, tabellen en bestaande kop- en voetteksten. "
213
+ "Opmerkingen, bijgehouden wijzigingen, voetnoten/eindnoten, tekstvakken, metadata en placeholders "
214
+ "die door Word over meerdere tekstfragmenten zijn gesplitst worden nog niet volledig ondersteund."
215
  )
216
  docx_reinsert_file = st.file_uploader(
217
  "Upload een DOCX-bestand met placeholders",
scripts/run_mvp_document_hygiene_fidelity_report.py ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import json
4
+ from pathlib import Path
5
+
6
+ from mvp_document_fidelity_report import write_document_fidelity_report
7
+ from mvp_phase6_validation_manifest import load_validation_manifest
8
+
9
+
10
+ MANIFEST = Path("test_cases/mvp_phase6/validation_manifest.json")
11
+ BASELINE = Path("output/validation/mvp_phase6_synthetic_validation_report.json")
12
+ TRIAGE = Path("output/validation/mvp_phase6_false_negative_gap_triage.json")
13
+ OUTPUT = Path(
14
+ "output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json"
15
+ )
16
+
17
+
18
+ def _json(path: Path) -> dict:
19
+ return json.loads(path.read_text(encoding="utf-8"))
20
+
21
+
22
+ def main() -> None:
23
+ report = write_document_fidelity_report(
24
+ load_validation_manifest(MANIFEST),
25
+ _json(BASELINE),
26
+ _json(TRIAGE),
27
+ OUTPUT,
28
+ )
29
+ print(
30
+ f"Wrote {OUTPUT}: resolved={report['resolved_finding_count']}, "
31
+ f"remaining={report['remaining_finding_count']}."
32
+ )
33
+
34
+
35
+ if __name__ == "__main__":
36
+ main()
scrub_key_document_reinsert.py CHANGED
@@ -5,11 +5,15 @@ for TXT and DOCX inputs. They deliberately avoid Streamlit UI integration,
5
  PDF handling, AI calls, remote processing, file-system persistence and export
6
  semantic changes.
7
 
8
- DOCX support is intentionally limited in this foundation version:
9
- - only `word/document.xml` text nodes are processed;
10
- - normal body paragraphs and tables in `word/document.xml` are covered;
11
- - placeholders split across multiple Word runs/text nodes are not restored;
12
- - headers, footers, comments, tracked changes and metadata are not processed.
 
 
 
 
13
  """
14
 
15
  from __future__ import annotations
@@ -17,19 +21,23 @@ from __future__ import annotations
17
  from io import BytesIO
18
  from typing import Any
19
  from zipfile import ZIP_DEFLATED, BadZipFile, ZipFile
 
20
  import xml.etree.ElementTree as ET
21
 
22
  from scrub_key_reinsert import reinsert_from_scrub_key
23
 
24
  DOCX_MAIN_DOCUMENT = "word/document.xml"
 
 
25
  WORDPROCESSINGML_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
26
  XML_NS = "http://www.w3.org/XML/1998/namespace"
27
 
28
  DOCX_LIMITATIONS = [
29
- "DOCX helper processes only word/document.xml text nodes in this foundation version.",
30
- "Normal body paragraphs and tables in word/document.xml are supported.",
 
31
  "Placeholders split across multiple Word runs/text nodes are not restored in this version.",
32
- "Headers, footers, comments, tracked changes and metadata are not processed in this version.",
33
  ]
34
 
35
  ET.register_namespace("w", WORDPROCESSINGML_NS)
@@ -48,11 +56,7 @@ def _with_document_metadata(result: dict[str, Any], document_type: str) -> dict[
48
 
49
 
50
  def reinsert_text_document(text: str, scrub_key: dict[str, Any]) -> dict[str, Any]:
51
- """Reinsert placeholders in plain text and return restored text plus audit.
52
-
53
- This is a document-level wrapper around ``reinsert_from_scrub_key`` for plain
54
- text/TXT use cases. It does not mutate the supplied Scrub Key.
55
- """
56
  result = _with_document_metadata(reinsert_from_scrub_key(text, scrub_key), "txt")
57
  result["content"] = result.get("text", "")
58
  result["limitations"] = []
@@ -64,11 +68,7 @@ def reinsert_txt_bytes(
64
  scrub_key: dict[str, Any],
65
  encoding: str = "utf-8",
66
  ) -> dict[str, Any]:
67
- """Decode TXT bytes, reinsert placeholders and return restored text/bytes.
68
-
69
- Decoding is strict by default so invalid input is reported instead of being
70
- silently changed. No file-system writes are performed.
71
- """
72
  if not isinstance(content, (bytes, bytearray)):
73
  result = reinsert_text_document("", scrub_key)
74
  result["validation_issues"] = ["TXT content must be bytes."]
@@ -111,6 +111,9 @@ def _docx_validation_result(content: bytes, message: str) -> dict[str, Any]:
111
  "cloud_processing": False,
112
  "limitations": list(DOCX_LIMITATIONS),
113
  "unsupported_parts": list(DOCX_LIMITATIONS),
 
 
 
114
  }
115
 
116
 
@@ -118,23 +121,50 @@ def _word_text_nodes(root: ET.Element) -> list[ET.Element]:
118
  return list(root.iter(f"{{{WORDPROCESSINGML_NS}}}t"))
119
 
120
 
121
- def _serialize_package_with_document_xml(content: bytes, document_xml: bytes) -> bytes:
 
 
 
 
 
 
 
 
 
 
122
  output = BytesIO()
123
- with ZipFile(BytesIO(content), "r") as source_package, ZipFile(output, "w", ZIP_DEFLATED) as target_package:
 
 
 
 
124
  for entry in source_package.infolist():
125
- data = document_xml if entry.filename == DOCX_MAIN_DOCUMENT else source_package.read(entry.filename)
 
 
126
  target_package.writestr(entry, data)
127
  return output.getvalue()
128
 
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  def reinsert_docx_bytes(content: bytes, scrub_key: dict[str, Any]) -> dict[str, Any]:
131
- """Reinsert placeholders in the main body of a DOCX document.
132
 
133
- This foundation helper processes ``word/document.xml`` text nodes only. That
134
- covers normal paragraphs and tables in the main document body, while leaving
135
- headers, footers, comments, tracked changes and metadata untouched. It returns
136
- restored DOCX bytes and an audit summary. The input bytes and Scrub Key are
137
- not mutated.
138
  """
139
  if not isinstance(content, (bytes, bytearray)):
140
  return _docx_validation_result(b"", "DOCX content must be bytes.")
@@ -149,30 +179,67 @@ def reinsert_docx_bytes(content: bytes, scrub_key: dict[str, Any]) -> dict[str,
149
  original_content,
150
  "DOCX package is missing word/document.xml.",
151
  )
152
- document_xml = docx_package.read(DOCX_MAIN_DOCUMENT)
 
153
  except BadZipFile:
154
- return _docx_validation_result(original_content, "DOCX content is not a valid OOXML package.")
 
 
 
155
 
156
- try:
157
- root = ET.fromstring(document_xml)
158
- except ET.ParseError as exc:
159
- return _docx_validation_result(original_content, f"DOCX document.xml could not be parsed: {exc}")
160
 
161
- text_nodes = _word_text_nodes(root)
162
- original_text = "\n".join(node.text or "" for node in text_nodes)
163
- audit_result = _with_document_metadata(reinsert_from_scrub_key(original_text, scrub_key), "docx")
 
 
 
 
 
 
 
 
 
 
 
 
 
164
 
165
  if not audit_result.get("validation_issues"):
166
- for node in text_nodes:
167
- node_result = reinsert_from_scrub_key(node.text or "", scrub_key)
168
- node.text = node_result.get("text", "")
169
-
170
- restored_document_xml = ET.tostring(root, encoding="utf-8", xml_declaration=True)
171
- restored_docx = _serialize_package_with_document_xml(original_content, restored_document_xml)
172
-
173
- audit_result["text"] = "\n".join(node.text or "" for node in text_nodes)
174
- audit_result["content"] = audit_result["text"]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
175
  audit_result["docx_bytes"] = restored_docx
176
  audit_result["limitations"] = list(DOCX_LIMITATIONS)
177
  audit_result["unsupported_parts"] = list(DOCX_LIMITATIONS)
 
 
 
178
  return audit_result
 
5
  PDF handling, AI calls, remote processing, file-system persistence and export
6
  semantic changes.
7
 
8
+ DOCX support covers text nodes in:
9
+ - ``word/document.xml``;
10
+ - ``word/header*.xml``;
11
+ - ``word/footer*.xml``.
12
+
13
+ Normal body paragraphs, body tables and header/footer text are restored. The
14
+ helper still does not restore placeholders split across multiple Word text
15
+ nodes, comments, tracked-change-only parts, footnotes/endnotes, text boxes or
16
+ metadata.
17
  """
18
 
19
  from __future__ import annotations
 
21
  from io import BytesIO
22
  from typing import Any
23
  from zipfile import ZIP_DEFLATED, BadZipFile, ZipFile
24
+ import re
25
  import xml.etree.ElementTree as ET
26
 
27
  from scrub_key_reinsert import reinsert_from_scrub_key
28
 
29
  DOCX_MAIN_DOCUMENT = "word/document.xml"
30
+ DOCX_HEADER_RE = re.compile(r"^word/header[^/]*\.xml$")
31
+ DOCX_FOOTER_RE = re.compile(r"^word/footer[^/]*\.xml$")
32
  WORDPROCESSINGML_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
33
  XML_NS = "http://www.w3.org/XML/1998/namespace"
34
 
35
  DOCX_LIMITATIONS = [
36
+ "DOCX helper processes word/document.xml plus word/header*.xml and word/footer*.xml text nodes.",
37
+ "Normal body paragraphs, body tables and header/footer text are supported.",
38
+ "Headers and footers are processed together with the main document body.",
39
  "Placeholders split across multiple Word runs/text nodes are not restored in this version.",
40
+ "Comments, tracked-change-only parts, footnotes/endnotes, text boxes and metadata are not processed in this version.",
41
  ]
42
 
43
  ET.register_namespace("w", WORDPROCESSINGML_NS)
 
56
 
57
 
58
  def reinsert_text_document(text: str, scrub_key: dict[str, Any]) -> dict[str, Any]:
59
+ """Reinsert placeholders in plain text and return restored text plus audit."""
 
 
 
 
60
  result = _with_document_metadata(reinsert_from_scrub_key(text, scrub_key), "txt")
61
  result["content"] = result.get("text", "")
62
  result["limitations"] = []
 
68
  scrub_key: dict[str, Any],
69
  encoding: str = "utf-8",
70
  ) -> dict[str, Any]:
71
+ """Decode TXT bytes, reinsert placeholders and return restored text/bytes."""
 
 
 
 
72
  if not isinstance(content, (bytes, bytearray)):
73
  result = reinsert_text_document("", scrub_key)
74
  result["validation_issues"] = ["TXT content must be bytes."]
 
111
  "cloud_processing": False,
112
  "limitations": list(DOCX_LIMITATIONS),
113
  "unsupported_parts": list(DOCX_LIMITATIONS),
114
+ "processed_parts": [],
115
+ "processed_part_count": 0,
116
+ "part_texts": {},
117
  }
118
 
119
 
 
121
  return list(root.iter(f"{{{WORDPROCESSINGML_NS}}}t"))
122
 
123
 
124
+ def _supported_docx_xml_parts(names: set[str]) -> list[str]:
125
+ parts = [DOCX_MAIN_DOCUMENT]
126
+ parts.extend(sorted(name for name in names if DOCX_HEADER_RE.match(name)))
127
+ parts.extend(sorted(name for name in names if DOCX_FOOTER_RE.match(name)))
128
+ return parts
129
+
130
+
131
+ def _serialize_package_with_xml_parts(
132
+ content: bytes,
133
+ replacement_parts: dict[str, bytes],
134
+ ) -> bytes:
135
  output = BytesIO()
136
+ with ZipFile(BytesIO(content), "r") as source_package, ZipFile(
137
+ output,
138
+ "w",
139
+ ZIP_DEFLATED,
140
+ ) as target_package:
141
  for entry in source_package.infolist():
142
+ data = replacement_parts.get(entry.filename)
143
+ if data is None:
144
+ data = source_package.read(entry.filename)
145
  target_package.writestr(entry, data)
146
  return output.getvalue()
147
 
148
 
149
+ def _parse_supported_parts(
150
+ package: ZipFile,
151
+ part_names: list[str],
152
+ ) -> tuple[dict[str, ET.Element], str | None]:
153
+ roots: dict[str, ET.Element] = {}
154
+ for part_name in part_names:
155
+ try:
156
+ roots[part_name] = ET.fromstring(package.read(part_name))
157
+ except ET.ParseError as exc:
158
+ return {}, f"DOCX {part_name} could not be parsed: {exc}"
159
+ return roots, None
160
+
161
+
162
  def reinsert_docx_bytes(content: bytes, scrub_key: dict[str, Any]) -> dict[str, Any]:
163
+ """Reinsert placeholders in supported DOCX body/header/footer text nodes.
164
 
165
+ The input bytes and Scrub Key are not mutated. All supported OOXML parts are
166
+ validated before any output package is produced. Audit counts and unresolved
167
+ placeholders cover the combined supported text surface.
 
 
168
  """
169
  if not isinstance(content, (bytes, bytearray)):
170
  return _docx_validation_result(b"", "DOCX content must be bytes.")
 
179
  original_content,
180
  "DOCX package is missing word/document.xml.",
181
  )
182
+ part_names = _supported_docx_xml_parts(names)
183
+ roots, parse_issue = _parse_supported_parts(docx_package, part_names)
184
  except BadZipFile:
185
+ return _docx_validation_result(
186
+ original_content,
187
+ "DOCX content is not a valid OOXML package.",
188
+ )
189
 
190
+ if parse_issue:
191
+ return _docx_validation_result(original_content, parse_issue)
 
 
192
 
193
+ part_nodes = {
194
+ part_name: _word_text_nodes(root)
195
+ for part_name, root in roots.items()
196
+ }
197
+ original_part_texts = {
198
+ part_name: "\n".join(node.text or "" for node in nodes)
199
+ for part_name, nodes in part_nodes.items()
200
+ }
201
+ combined_original_text = "\n".join(
202
+ original_part_texts[part_name]
203
+ for part_name in part_names
204
+ )
205
+ audit_result = _with_document_metadata(
206
+ reinsert_from_scrub_key(combined_original_text, scrub_key),
207
+ "docx",
208
+ )
209
 
210
  if not audit_result.get("validation_issues"):
211
+ for nodes in part_nodes.values():
212
+ for node in nodes:
213
+ node_result = reinsert_from_scrub_key(node.text or "", scrub_key)
214
+ node.text = node_result.get("text", "")
215
+
216
+ restored_part_texts = {
217
+ part_name: "\n".join(node.text or "" for node in part_nodes[part_name])
218
+ for part_name in part_names
219
+ }
220
+ restored_xml_parts = {
221
+ part_name: ET.tostring(
222
+ roots[part_name],
223
+ encoding="utf-8",
224
+ xml_declaration=True,
225
+ )
226
+ for part_name in part_names
227
+ }
228
+ restored_docx = _serialize_package_with_xml_parts(
229
+ original_content,
230
+ restored_xml_parts,
231
+ )
232
+ combined_restored_text = "\n".join(
233
+ restored_part_texts[part_name]
234
+ for part_name in part_names
235
+ )
236
+
237
+ audit_result["text"] = combined_restored_text
238
+ audit_result["content"] = combined_restored_text
239
  audit_result["docx_bytes"] = restored_docx
240
  audit_result["limitations"] = list(DOCX_LIMITATIONS)
241
  audit_result["unsupported_parts"] = list(DOCX_LIMITATIONS)
242
+ audit_result["processed_parts"] = list(part_names)
243
+ audit_result["processed_part_count"] = len(part_names)
244
+ audit_result["part_texts"] = restored_part_texts
245
  return audit_result
tests/test_mvp_document_fidelity_pr_final_contracts.py ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+
3
+
4
+ ROOT = Path(__file__).resolve().parents[1]
5
+
6
+
7
+ def test_temporary_document_fidelity_patch_scripts_are_absent() -> None:
8
+ assert not (ROOT / "scripts" / "patch_docx_reinsert_capability_copy_only.py").exists()
9
+ assert not (ROOT / "scripts" / "finalize_mvp_document_fidelity_pr_cleanup.py").exists()
10
+
11
+
12
+ def test_document_fidelity_governance_evidence_has_no_duplicate_lines() -> None:
13
+ changelog = (ROOT / "CHANGELOG.md").read_text(encoding="utf-8")
14
+ workpackages = (ROOT / "WORKPACKAGES.md").read_text(encoding="utf-8")
15
+ handover = (
16
+ ROOT
17
+ / "handover"
18
+ / "workpackages"
19
+ / "20260717_2230_mvp_document_hygiene_fidelity_hardening.md"
20
+ ).read_text(encoding="utf-8")
21
+
22
+ assert changelog.count("- `tests/test_mvp_document_fidelity_ui_copy.py`") == 1
23
+ assert changelog.count(
24
+ "The DOCX reinsert capability copy now matches the supported body/table/header/footer scope."
25
+ ) == 1
26
+ assert workpackages.count(
27
+ "Aligned the existing DOCX reinsert information copy with the supported body/table/header/footer scope without adding controls."
28
+ ) == 1
29
+ assert handover.count("- `tests/test_mvp_document_fidelity_ui_copy.py`") == 1
30
+ assert handover.count("Source-level DOCX/PDF capability-copy contract tests.") == 1
31
+
32
+
33
+ def test_final_claim_preserves_verification_and_product_boundaries() -> None:
34
+ claim = (
35
+ ROOT
36
+ / "workpackage_claims"
37
+ / "scrub_wp_mvp_document_hygiene_fidelity_hardening.md"
38
+ ).read_text(encoding="utf-8")
39
+
40
+ assert "DOCX header/footer reinsert resolved: true" in claim
41
+ assert "No OCR or restored-PDF implementation" in claim
42
+ assert "App verification required after Actions and Hugging Face sync" in claim
tests/test_mvp_document_fidelity_report.py ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import json
4
+ from pathlib import Path
5
+
6
+ from mvp_document_fidelity_report import (
7
+ build_document_fidelity_report,
8
+ write_document_fidelity_report,
9
+ )
10
+ from mvp_phase6_validation_manifest import load_validation_manifest
11
+
12
+
13
+ ROOT = Path(__file__).resolve().parents[1]
14
+ MANIFEST = ROOT / "test_cases" / "mvp_phase6" / "validation_manifest.json"
15
+ BASELINE = ROOT / "output" / "validation" / "mvp_phase6_synthetic_validation_report.json"
16
+ TRIAGE = ROOT / "output" / "validation" / "mvp_phase6_false_negative_gap_triage.json"
17
+
18
+
19
+ def _json(path: Path) -> dict:
20
+ return json.loads(path.read_text(encoding="utf-8"))
21
+
22
+
23
+ def _report() -> dict:
24
+ return build_document_fidelity_report(
25
+ load_validation_manifest(MANIFEST),
26
+ _json(BASELINE),
27
+ _json(TRIAGE),
28
+ )
29
+
30
+
31
+ def test_hardening_report_closes_docx_header_footer_finding() -> None:
32
+ report = _report()
33
+
34
+ assert report["schema"] == (
35
+ "solidprivacy.mvp_phase6_document_fidelity_hardening_report"
36
+ )
37
+ assert report["schema_version"] == "1.0"
38
+ assert report["docx_header_footer_reinsert_resolved"] is True
39
+ assert report["resolved_finding_count"] == 1
40
+ assert report["remaining_finding_count"] == 1
41
+
42
+ resolved = report["resolved_findings"][0]
43
+ assert resolved["source_category"] == "known_docx_reinsert_limitation"
44
+ assert set(resolved["resolved_placeholders"]) == {
45
+ "[CLIENT_REFERENTIE_01]",
46
+ "[OVERIGE_REFERENTIE_01]",
47
+ }
48
+ assert any(part.startswith("word/header") for part in resolved["processed_parts"])
49
+ assert any(part.startswith("word/footer") for part in resolved["processed_parts"])
50
+
51
+
52
+ def test_pdf_boundary_remains_explicit_and_unexpanded() -> None:
53
+ report = _report()
54
+
55
+ assert report["pdf_boundary_preserved"] is True
56
+ remaining = report["remaining_findings"][0]
57
+ assert remaining["category"] == "known_pdf_reinsert_limitation"
58
+ assert remaining["status"] == "explicit_product_boundary"
59
+ assert remaining["reinsert_output_type"] == "txt"
60
+ assert remaining["restored_pdf_supported"] is False
61
+ assert remaining["ocr_supported"] is False
62
+
63
+
64
+ def test_current_matrix_has_no_docx_residual_gap() -> None:
65
+ current = _report()["current_validation_report"]
66
+ docx = next(case for case in current["cases"] if case["document_type"] == "docx")
67
+
68
+ assert current["failing_case_count"] == 0
69
+ assert docx["roundtrip_complete"] is True
70
+ assert docx["residual_placeholders"] == []
71
+ assert docx["header_footer_roundtrip_values_present"] is True
72
+ assert {gap["category"] for gap in current["evidence_gaps"]} == {
73
+ "known_pdf_reinsert_limitation"
74
+ }
75
+
76
+
77
+ def test_report_retains_privacy_and_claim_boundaries() -> None:
78
+ report = _report()
79
+
80
+ assert report["synthetic_data_only"] is True
81
+ assert report["human_review_required"] is True
82
+ assert report["production_ready"] is False
83
+ assert report["production_readiness_claim"] is False
84
+ assert report["local_only"] is True
85
+ assert report["ai_processing"] is False
86
+ assert report["cloud_processing"] is False
87
+ assert report["ocr_processing"] is False
88
+ assert report["next_recommended_package"] == (
89
+ "SCRUB-WP_MVP_SCRUB_KEY_ROUNDTRIP_VALIDATION"
90
+ )
91
+
92
+
93
+ def test_report_writer_is_deterministic(tmp_path: Path) -> None:
94
+ output = tmp_path / "hardening.json"
95
+ first = write_document_fidelity_report(
96
+ load_validation_manifest(MANIFEST),
97
+ _json(BASELINE),
98
+ _json(TRIAGE),
99
+ output,
100
+ )
101
+ first_text = output.read_text(encoding="utf-8")
102
+ second = write_document_fidelity_report(
103
+ load_validation_manifest(MANIFEST),
104
+ _json(BASELINE),
105
+ _json(TRIAGE),
106
+ output,
107
+ )
108
+
109
+ assert first == second
110
+ assert first_text == output.read_text(encoding="utf-8")
111
+ assert json.loads(first_text) == first
tests/test_mvp_document_fidelity_ui_copy.py ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import ast
4
+ from pathlib import Path
5
+
6
+
7
+ ROOT = Path(__file__).resolve().parents[1]
8
+ UI = ROOT / "reinsert_mode_ui.py"
9
+ THIS_TEST = ROOT / "tests" / "test_mvp_document_fidelity_ui_copy.py"
10
+
11
+
12
+ def test_docx_reinsert_copy_matches_supported_parts() -> None:
13
+ text = UI.read_text(encoding="utf-8")
14
+
15
+ assert "normale documenttekst, tabellen en bestaande kop- en voetteksten" in text
16
+ assert "Opmerkingen, bijgehouden wijzigingen, voetnoten/eindnoten, tekstvakken, metadata" in text
17
+ assert "over meerdere tekstfragmenten zijn gesplitst" in text
18
+ assert "Headers, footers" not in text
19
+
20
+
21
+ def test_pdf_txt_only_and_no_ocr_copy_remains_unchanged() -> None:
22
+ text = UI.read_text(encoding="utf-8")
23
+
24
+ assert "Deze functie maakt geen herstelde PDF" in text
25
+ assert "OCR niet beschikbaar" in text
26
+ assert "PDF-output: Nee" in text
27
+
28
+
29
+ def test_copy_contract_does_not_import_streamlit_or_product_ui() -> None:
30
+ tree = ast.parse(THIS_TEST.read_text(encoding="utf-8"))
31
+ imported_roots: set[str] = set()
32
+
33
+ for node in ast.walk(tree):
34
+ if isinstance(node, ast.Import):
35
+ imported_roots.update(alias.name.split(".")[0] for alias in node.names)
36
+ elif isinstance(node, ast.ImportFrom) and node.module:
37
+ imported_roots.add(node.module.split(".")[0])
38
+
39
+ assert "streamlit" not in imported_roots
40
+ assert "reinsert_mode_ui" not in imported_roots
tests/test_mvp_document_hygiene_fidelity_hardening.py ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ from io import BytesIO
4
+ from pathlib import Path
5
+ from zipfile import ZIP_DEFLATED, ZipFile
6
+
7
+ from docx import Document
8
+
9
+ from document_tools import anonymized_docx_from_original
10
+ from scrub_key import build_scrub_key
11
+ from scrub_key_document_reinsert import (
12
+ DOCX_LIMITATIONS,
13
+ reinsert_docx_bytes,
14
+ )
15
+
16
+
17
+ class UploadedBytes:
18
+ def __init__(self, name: str, content: bytes):
19
+ self.name = name
20
+ self._content = bytes(content)
21
+
22
+ def getvalue(self) -> bytes:
23
+ return self._content
24
+
25
+
26
+ ROWS = [
27
+ {
28
+ "original_value": "BETROKKENE-TEST-A",
29
+ "placeholder": "[PERSOON_1]",
30
+ "entity_type": "PERSON",
31
+ "type_label": "Naam",
32
+ "source": "detected",
33
+ "review_status": "auto_detected",
34
+ "include": True,
35
+ "timestamp": "2026-07-17T20:30:00Z",
36
+ },
37
+ {
38
+ "original_value": "RECHTBANK TESTDAM",
39
+ "placeholder": "[ORGANISATIE_01]",
40
+ "entity_type": "ORGANIZATION",
41
+ "type_label": "Organisatie",
42
+ "source": "detected",
43
+ "review_status": "auto_detected",
44
+ "include": True,
45
+ "timestamp": "2026-07-17T20:30:01Z",
46
+ },
47
+ {
48
+ "original_value": "ZAAK-TEST-2026-001",
49
+ "placeholder": "[ZAAKNUMMER_1]",
50
+ "entity_type": "LEGAL_REFERENCE",
51
+ "type_label": "Zaaknummer",
52
+ "source": "manual",
53
+ "review_status": "manual",
54
+ "include": True,
55
+ "timestamp": "2026-07-17T20:30:02Z",
56
+ },
57
+ ]
58
+
59
+ REPLACEMENTS = {
60
+ row["original_value"]: row["placeholder"]
61
+ for row in ROWS
62
+ }
63
+
64
+
65
+ def _scrub_key() -> dict:
66
+ return build_scrub_key(ROWS, document_label="Synthetisch fidelitydossier")
67
+
68
+
69
+ def _source_docx() -> bytes:
70
+ document = Document()
71
+ document.add_paragraph("Body BETROKKENE-TEST-A")
72
+ table = document.add_table(rows=1, cols=1)
73
+ table.cell(0, 0).text = "Tabel ZAAK-TEST-2026-001"
74
+ section = document.sections[0]
75
+ section.header.paragraphs[0].text = "Header RECHTBANK TESTDAM"
76
+ section.footer.paragraphs[0].text = "Footer ZAAK-TEST-2026-001"
77
+ output = BytesIO()
78
+ document.save(output)
79
+ return output.getvalue()
80
+
81
+
82
+ def _text_surfaces(docx_bytes: bytes) -> dict[str, str]:
83
+ document = Document(BytesIO(docx_bytes))
84
+ section = document.sections[0]
85
+ return {
86
+ "body": "\n".join(paragraph.text for paragraph in document.paragraphs),
87
+ "table": "\n".join(
88
+ paragraph.text
89
+ for table in document.tables
90
+ for row in table.rows
91
+ for cell in row.cells
92
+ for paragraph in cell.paragraphs
93
+ ),
94
+ "header": "\n".join(paragraph.text for paragraph in section.header.paragraphs),
95
+ "footer": "\n".join(paragraph.text for paragraph in section.footer.paragraphs),
96
+ }
97
+
98
+
99
+ def test_scrub_then_reinsert_restores_body_table_header_and_footer() -> None:
100
+ source = _source_docx()
101
+ scrubbed = anonymized_docx_from_original(
102
+ UploadedBytes("synthetic.docx", source),
103
+ REPLACEMENTS,
104
+ )
105
+ scrubbed_surfaces = _text_surfaces(scrubbed)
106
+
107
+ assert "[PERSOON_1]" in scrubbed_surfaces["body"]
108
+ assert "[ZAAKNUMMER_1]" in scrubbed_surfaces["table"]
109
+ assert "[ORGANISATIE_01]" in scrubbed_surfaces["header"]
110
+ assert "[ZAAKNUMMER_1]" in scrubbed_surfaces["footer"]
111
+
112
+ result = reinsert_docx_bytes(scrubbed, _scrub_key())
113
+ restored = _text_surfaces(result["docx_bytes"])
114
+
115
+ assert restored["body"] == "Body BETROKKENE-TEST-A"
116
+ assert restored["table"] == "Tabel ZAAK-TEST-2026-001"
117
+ assert restored["header"] == "Header RECHTBANK TESTDAM"
118
+ assert restored["footer"] == "Footer ZAAK-TEST-2026-001"
119
+ assert result["replacement_count"] == 4
120
+ assert result["placeholders_not_found"] == []
121
+ assert result["unknown_placeholders"] == []
122
+ assert result["validation_issues"] == []
123
+
124
+
125
+ def test_result_reports_processed_ooxml_parts() -> None:
126
+ scrubbed = anonymized_docx_from_original(
127
+ UploadedBytes("synthetic.docx", _source_docx()),
128
+ REPLACEMENTS,
129
+ )
130
+ result = reinsert_docx_bytes(scrubbed, _scrub_key())
131
+
132
+ assert "word/document.xml" in result["processed_parts"]
133
+ assert any(part.startswith("word/header") for part in result["processed_parts"])
134
+ assert any(part.startswith("word/footer") for part in result["processed_parts"])
135
+ assert result["processed_part_count"] == len(result["processed_parts"])
136
+ assert set(result["processed_parts"]) == set(result["part_texts"])
137
+ assert result["local_only"] is True
138
+ assert result["ai_processing"] is False
139
+ assert result["cloud_processing"] is False
140
+
141
+
142
+ def test_comments_tracked_changes_metadata_and_split_nodes_remain_unsupported() -> None:
143
+ limitation_text = " ".join(DOCX_LIMITATIONS).lower()
144
+
145
+ assert "header/footer text are supported" in limitation_text
146
+ assert "split" in limitation_text
147
+ assert "comments" in limitation_text
148
+ assert "tracked" in limitation_text
149
+ assert "metadata" in limitation_text
150
+ assert "not processed" in limitation_text
151
+
152
+
153
+ def test_unrelated_package_parts_are_preserved_byte_for_byte() -> None:
154
+ marker_path = "customXml/synthetic-marker.xml"
155
+ marker_bytes = b"<synthetic>KEEP-ME</synthetic>"
156
+ scrubbed = anonymized_docx_from_original(
157
+ UploadedBytes("synthetic.docx", _source_docx()),
158
+ REPLACEMENTS,
159
+ )
160
+ enriched = BytesIO()
161
+ with ZipFile(BytesIO(scrubbed), "r") as original, ZipFile(
162
+ enriched,
163
+ "w",
164
+ ZIP_DEFLATED,
165
+ ) as output:
166
+ for entry in original.infolist():
167
+ output.writestr(entry, original.read(entry.filename))
168
+ output.writestr(marker_path, marker_bytes)
169
+
170
+ result = reinsert_docx_bytes(enriched.getvalue(), _scrub_key())
171
+
172
+ with ZipFile(BytesIO(result["docx_bytes"]), "r") as restored:
173
+ assert restored.read(marker_path) == marker_bytes
174
+
175
+
176
+ def test_invalid_header_xml_returns_validation_issue_without_partial_output() -> None:
177
+ source = _source_docx()
178
+ broken = BytesIO()
179
+ with ZipFile(BytesIO(source), "r") as original, ZipFile(
180
+ broken,
181
+ "w",
182
+ ZIP_DEFLATED,
183
+ ) as output:
184
+ for entry in original.infolist():
185
+ data = original.read(entry.filename)
186
+ if entry.filename.startswith("word/header") and entry.filename.endswith(".xml"):
187
+ data = b"<w:hdr>"
188
+ output.writestr(entry, data)
189
+
190
+ result = reinsert_docx_bytes(broken.getvalue(), _scrub_key())
191
+
192
+ assert result["reinserted"] is False
193
+ assert result["replacement_count"] == 0
194
+ assert result["validation_issues"]
195
+ assert "header" in result["validation_issues"][0].lower()
196
+ assert result["docx_bytes"] == broken.getvalue()
197
+
198
+
199
+ def test_pdf_ocr_and_restored_pdf_scope_are_not_added() -> None:
200
+ source = Path("MVP_PHASE6_FALSE_NEGATIVE_GAP_TRIAGE.md").read_text(encoding="utf-8")
201
+
202
+ assert "restored-TXT-only" in source
203
+ assert "No OCR or restored-PDF implementation is authorized" in source
tests/test_mvp_phase6_e2e_synthetic_validation_matrix.py CHANGED
@@ -117,7 +117,7 @@ def test_txt_detection_evidence_covers_expected_values_and_preserves_roles() ->
117
  )
118
 
119
 
120
- def test_docx_case_records_hygiene_and_header_footer_reinsert_limits() -> None:
121
  case = _case(_report(), "docx_mixed_structure_roundtrip")
122
 
123
  assert case["status"] == "pass_with_known_limitations"
@@ -125,16 +125,20 @@ def test_docx_case_records_hygiene_and_header_footer_reinsert_limits() -> None:
125
  assert case["scrubbed_import_type"] == "docx"
126
  assert case["restored_import_type"] == "docx"
127
  assert case["body_roundtrip_values_present"] is True
 
128
  assert case["hygiene_severity"] == "high"
129
  assert {"headers_detected", "footers_detected"}.issubset(
130
  set(case["hygiene_findings"])
131
  )
132
  assert case["audit_expectations_met"] is True
133
  assert case["limitation_expectations_met"] is True
134
- assert case["roundtrip_complete"] is False
135
- assert set(case["expected_residual_placeholders"]).issubset(
136
- set(case["residual_placeholders"])
 
137
  )
 
 
138
 
139
 
140
  def test_pdf_case_preserves_text_roundtrip_and_explicit_txt_only_boundary() -> None:
@@ -171,10 +175,7 @@ def test_report_is_machine_readable_and_never_claims_production_readiness() -> N
171
  )
172
 
173
  categories = {item["category"] for item in report["evidence_gaps"]}
174
- assert categories == {
175
- "known_docx_reinsert_limitation",
176
- "known_pdf_reinsert_limitation",
177
- }
178
 
179
 
180
  def test_report_writer_is_deterministic(tmp_path: Path) -> None:
 
117
  )
118
 
119
 
120
+ def test_docx_case_restores_header_footer_and_retains_hygiene_evidence() -> None:
121
  case = _case(_report(), "docx_mixed_structure_roundtrip")
122
 
123
  assert case["status"] == "pass_with_known_limitations"
 
125
  assert case["scrubbed_import_type"] == "docx"
126
  assert case["restored_import_type"] == "docx"
127
  assert case["body_roundtrip_values_present"] is True
128
+ assert case["header_footer_roundtrip_values_present"] is True
129
  assert case["hygiene_severity"] == "high"
130
  assert {"headers_detected", "footers_detected"}.issubset(
131
  set(case["hygiene_findings"])
132
  )
133
  assert case["audit_expectations_met"] is True
134
  assert case["limitation_expectations_met"] is True
135
+ assert case["roundtrip_complete"] is True
136
+ assert case["residual_placeholders"] == []
137
+ assert set(case["expected_residual_placeholders"]) == set(
138
+ case["resolved_header_footer_placeholders"]
139
  )
140
+ assert any(part.startswith("word/header") for part in case["processed_parts"])
141
+ assert any(part.startswith("word/footer") for part in case["processed_parts"])
142
 
143
 
144
  def test_pdf_case_preserves_text_roundtrip_and_explicit_txt_only_boundary() -> None:
 
175
  )
176
 
177
  categories = {item["category"] for item in report["evidence_gaps"]}
178
+ assert categories == {"known_pdf_reinsert_limitation"}
 
 
 
179
 
180
 
181
  def test_report_writer_is_deterministic(tmp_path: Path) -> None:
workpackage_claims/scrub_wp_mvp_document_hygiene_fidelity_hardening.md ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Workpackage claim — SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
2
+
3
+ Repository: solidprivacy-nl/scrub
4
+
5
+ Workpackage title: SCRUB-WP_MVP_DOCUMENT_HYGIENE_FIDELITY_HARDENING
6
+
7
+ Status: completed / ready for app verification
8
+
9
+ Claimed by: ChatGPT GitHub worker
10
+
11
+ Claimed at: 2026-07-17 22:30 Europe/Amsterdam
12
+
13
+ Branch: scrub-mvp-document-hygiene-fidelity-hardening
14
+
15
+ Dependencies:
16
+ - SCRUB-WP_MVP_E2E_SYNTHETIC_VALIDATION_MATRIX — merged.
17
+ - SCRUB-WP_MVP_FALSE_NEGATIVE_GAP_TRIAGE — merged.
18
+
19
+ Evidence:
20
+ - DOCX body/table reinsert passes, while scrubbed header/footer placeholders remain after reinsert.
21
+ - DOCX hygiene audit already reports header/footer presence.
22
+ - PDF text roundtrip passes only to restored TXT; restored PDF and OCR remain unsupported product boundaries.
23
+
24
+ Scope:
25
+ - Harden deterministic DOCX reinsert fidelity for existing scrubbed header and footer OOXML parts.
26
+ - Add contract and regression tests covering document body, tables, headers and footers.
27
+ - Preserve explicit unsupported boundaries for comments, tracked changes, metadata, split placeholders, OCR and restored PDF.
28
+ - Update the Phase 6 matrix expectation and evidence after the helper change.
29
+
30
+ Boundaries:
31
+ - Helper-level document reinsert plus capability-copy alignment only; no new Streamlit controls or flow.
32
+ - No recognizer, threshold or replacement changes.
33
+ - No export filename, MIME or Scrub Key schema changes.
34
+ - No comments/tracked changes/metadata cleaning guarantee.
35
+ - No OCR or restored-PDF implementation.
36
+ - Synthetic data only; local-only deterministic processing.
37
+
38
+ Next step:
39
+ - Run final PR validation, merge after green Actions, verify Hugging Face sync and request live DOCX reinsert verification.
40
+
41
+
42
+ Hardening result:
43
+ - Completed at: 2026-07-17 22:30 Europe/Amsterdam
44
+ - DOCX header/footer reinsert resolved: true
45
+ - Resolved findings: 1
46
+ - Remaining findings: 1
47
+ - Report: `output/validation/mvp_phase6_document_hygiene_fidelity_hardening_report.json`
48
+ - Handover: `handover/workpackages/20260717_2230_mvp_document_hygiene_fidelity_hardening.md`
49
+ - App verification required after Actions and Hugging Face sync.
50
+
51
+ Final PR cleanup:
52
+ - Temporary patch/finalizer scripts removed and duplicate governance lines normalized before the final Actions run.