Switch demo to coherent business policy claim verification

README.md

@@ -13,9 +13,9 @@ models:
 
 # ClaimLens-M
 
-ClaimLens-M is an English-Italian evidence-grounded claim verification demo.
+ClaimLens-M is an English-Italian business policy claim verification demo.
 
-It retrieves evidence sentences, runs a fine-tuned XLM-RoBERTa NLI verifier, and returns:
+It checks claims from HR, travel, security, leave, equipment, and expense policy summaries against a trusted evidence corpus. It retrieves evidence sentences, runs a fine-tuned XLM-RoBERTa NLI verifier, and returns:
 
 - verdict: `SUPPORTED`, `REFUTED`, or `NOT_ENOUGH_INFO`
 - confidence score
@@ -36,4 +36,13 @@ Test metrics:
 - macro F1: `0.8057`
 - weighted F1: `0.8094`
 
-The demo uses a curated English-Italian evidence corpus for fast interactive examples. The model weights are loaded from [`mokarami/claimlens-m-verifier`](https://huggingface.co/mokarami/claimlens-m-verifier) at runtime instead of being stored inside the Space repository.
+The demo uses a curated English-Italian business policy evidence corpus for fast interactive examples. The model weights are loaded from [`mokarami/claimlens-m-verifier`](https://huggingface.co/mokarami/claimlens-m-verifier) at runtime instead of being stored inside the Space repository.
+
+Example claims:
+
+- `Employees can work remotely five days per week without approval.`
+- `Company laptops must use full-disk encryption.`
+- `Interns are eligible for international travel reimbursement by default.`
+- `Le spese mensili per internet domestico sono rimborsate come indennita mensile.`
+
+This is not a general internet fact-checker. It verifies claims only against the trusted evidence corpus included in the demo.

app.py

@@ -65,14 +65,14 @@ with gr.Blocks(title="ClaimLens-M") as demo:
     gr.Markdown(
         """
         # ClaimLens-M
-        English-Italian evidence-grounded claim verification with retrieval, fine-tuned XLM-RoBERTa NLI, and structured JSON output.
+        Business policy claim verification. Enter a claim from an HR, travel, security, leave, or expense policy summary and verify it against the trusted policy evidence corpus.
         """
     )
     with gr.Row():
         claim = gr.Textbox(
-            label="Claim",
+            label="Policy Claim to Verify",
             lines=3,
-            value="QLoRA reduces GPU memory requirements during fine-tuning.",
+            value="Employees can work remotely five days per week without approval.",
         )
     with gr.Row():
         language = gr.Dropdown(["auto", "en", "it"], value="auto", label="Language")
@@ -87,10 +87,12 @@ with gr.Blocks(title="ClaimLens-M") as demo:
     payload = gr.JSON(label="Structured JSON")
     gr.Examples(
         examples=[
-            ["QLoRA reduces GPU memory requirements during fine-tuning.", "en", "hybrid", 5],
-            ["Vitamin D significantly reduces the risk of respiratory infections.", "en", "hybrid", 5],
-            ["QLoRA riduce l'uso della memoria GPU durante il fine-tuning.", "it", "hybrid", 5],
-            ["Il metodo riduce chiaramente la latenza del sistema.", "it", "hybrid", 5],
+            ["Employees can work remotely up to three days per week with manager approval.", "en", "hybrid", 5],
+            ["Employees can work remotely five days per week without approval.", "en", "hybrid", 5],
+            ["Interns are eligible for international travel reimbursement by default.", "en", "hybrid", 5],
+            ["Company laptops must use full-disk encryption.", "en", "hybrid", 5],
+            ["I dipendenti possono lavorare da remoto fino a tre giorni alla settimana con approvazione del manager.", "it", "hybrid", 5],
+            ["Le spese mensili per internet domestico sono rimborsate come indennita mensile.", "it", "hybrid", 5],
         ],
         inputs=[claim, language, retriever, top_k],
     )

data/sample/claims.jsonl

@@ -1,7 +1,13 @@
-{"claim":"QLoRA reduces GPU memory requirements during fine-tuning.","language":"en","label":"SUPPORTED","gold_evidence":["scifact_demo_001:s1"]}
-{"claim":"Vitamin D significantly reduces the risk of respiratory infections.","language":"en","label":"REFUTED","gold_evidence":["scifact_demo_002:s1"]}
-{"claim":"Hybrid retrieval can combine BM25 and dense similarity for evidence retrieval.","language":"en","label":"SUPPORTED","gold_evidence":["scifact_demo_003:s1"]}
-{"claim":"QLoRA riduce l'uso della memoria GPU durante il fine-tuning.","language":"it","label":"SUPPORTED","gold_evidence":["it_demo_001:s1"]}
-{"claim":"Lo studio ha migliorato significativamente l'accuratezza rispetto al modello di base.","language":"it","label":"REFUTED","gold_evidence":["it_demo_002:s1"]}
-{"claim":"Il metodo riduce chiaramente la latenza del sistema.","language":"it","label":"NOT_ENOUGH_INFO","gold_evidence":["it_demo_003:s1"]}
-
+{"claim":"Employees can work remotely up to three days per week with manager approval.","language":"en","label":"SUPPORTED","gold_evidence":["remote_work_policy:s1"]}
+{"claim":"Employees can work remotely five days per week without approval.","language":"en","label":"REFUTED","gold_evidence":["remote_work_policy:s1"]}
+{"claim":"New employees are eligible for remote work during their first thirty days.","language":"en","label":"REFUTED","gold_evidence":["remote_work_policy:s3"]}
+{"claim":"Employees receive a monthly home internet allowance.","language":"en","label":"REFUTED","gold_evidence":["expense_policy:s1"]}
+{"claim":"Interns are eligible for international travel reimbursement by default.","language":"en","label":"REFUTED","gold_evidence":["travel_reimbursement_policy:s2"]}
+{"claim":"Company laptops must use full-disk encryption.","language":"en","label":"SUPPORTED","gold_evidence":["security_policy:s1"]}
+{"claim":"Customer data can be copied to personal cloud storage.","language":"en","label":"REFUTED","gold_evidence":["security_policy:s2"]}
+{"claim":"Employees get twenty-two paid vacation days per year.","language":"en","label":"SUPPORTED","gold_evidence":["leave_policy:s1"]}
+{"claim":"Employees receive a free gym membership.","language":"en","label":"NOT_ENOUGH_INFO","gold_evidence":["leave_policy:s1"]}
+{"claim":"I dipendenti possono lavorare da remoto fino a tre giorni alla settimana con approvazione del manager.","language":"it","label":"SUPPORTED","gold_evidence":["policy_lavoro_remoto:s1"]}
+{"claim":"Gli stagisti hanno sempre diritto al rimborso dei viaggi internazionali.","language":"it","label":"REFUTED","gold_evidence":["policy_viaggi:s2"]}
+{"claim":"I laptop aziendali devono usare la crittografia completa del disco.","language":"it","label":"SUPPORTED","gold_evidence":["policy_sicurezza:s1"]}
+{"claim":"Le spese mensili per internet domestico sono rimborsate come indennita mensile.","language":"it","label":"REFUTED","gold_evidence":["policy_spese:s1"]}

data/sample/evidence_corpus.jsonl

@@ -1,7 +1,24 @@
-{"source_id":"scifact_demo_001","sentence_id":"s1","text":"QLoRA enables fine-tuning large language models with 4-bit quantization, reducing GPU memory requirements while preserving performance.","language":"en","gold_label":"SUPPORTED"}
-{"source_id":"scifact_demo_002","sentence_id":"s1","text":"The study found no statistically significant reduction in respiratory infection risk for participants receiving vitamin D compared with placebo.","language":"en","gold_label":"REFUTED"}
-{"source_id":"scifact_demo_003","sentence_id":"s1","text":"Hybrid retrieval combines lexical BM25 ranking with dense embedding similarity to improve evidence recall in question answering systems.","language":"en","gold_label":"SUPPORTED"}
-{"source_id":"it_demo_001","sentence_id":"s1","text":"QLoRA permette il fine-tuning di modelli linguistici di grandi dimensioni usando quantizzazione a 4 bit, riducendo l'uso di memoria GPU.","language":"it","gold_label":"SUPPORTED"}
-{"source_id":"it_demo_002","sentence_id":"s1","text":"Lo studio non ha osservato un miglioramento significativo dell'accuratezza rispetto al modello di base.","language":"it","gold_label":"REFUTED"}
-{"source_id":"it_demo_003","sentence_id":"s1","text":"La ricerca discute il dataset utilizzato, ma non fornisce prove sufficienti sulla riduzione della latenza.","language":"it","gold_label":"NOT_ENOUGH_INFO"}
-
+{"source_id":"remote_work_policy","sentence_id":"s1","text":"Employees may work remotely up to three days per week after manager approval.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"remote_work_policy","sentence_id":"s2","text":"Fully remote work requires written approval from the department director and HR.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"remote_work_policy","sentence_id":"s3","text":"New employees are not eligible for remote work during their first thirty days unless HR grants an exception.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"travel_reimbursement_policy","sentence_id":"s1","text":"Employees are eligible for travel reimbursement when the trip is pre-approved and receipts are submitted within fourteen days.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"travel_reimbursement_policy","sentence_id":"s2","text":"Interns are not eligible for international travel reimbursement unless the internship contract explicitly includes travel support.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"travel_reimbursement_policy","sentence_id":"s3","text":"Meal expenses above 60 euros per day require finance approval.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"security_policy","sentence_id":"s1","text":"Company laptops must use full-disk encryption and automatic screen locking.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"security_policy","sentence_id":"s2","text":"Customer data may not be copied to personal cloud storage services.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"security_policy","sentence_id":"s3","text":"Production credentials must be stored in the approved secrets manager, not in source code or shared documents.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"leave_policy","sentence_id":"s1","text":"Full-time employees receive twenty-two paid vacation days per calendar year.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"leave_policy","sentence_id":"s2","text":"Sick leave longer than three consecutive workdays requires a medical certificate.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"leave_policy","sentence_id":"s3","text":"Unused vacation days may be carried over only until March 31 of the following year.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"equipment_policy","sentence_id":"s1","text":"Employees may request an external monitor after completing the onboarding checklist.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"equipment_policy","sentence_id":"s2","text":"Personal devices cannot be used to access production systems.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"expense_policy","sentence_id":"s1","text":"Home internet expenses are not reimbursed as a monthly allowance.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"expense_policy","sentence_id":"s2","text":"One-time ergonomic equipment purchases may be reimbursed up to 250 euros with manager approval.","language":"en","gold_label":"SUPPORTED"}
+{"source_id":"policy_lavoro_remoto","sentence_id":"s1","text":"I dipendenti possono lavorare da remoto fino a tre giorni alla settimana dopo l'approvazione del manager.","language":"it","gold_label":"SUPPORTED"}
+{"source_id":"policy_lavoro_remoto","sentence_id":"s2","text":"Il lavoro completamente da remoto richiede l'approvazione scritta del direttore di dipartimento e delle risorse umane.","language":"it","gold_label":"SUPPORTED"}
+{"source_id":"policy_viaggi","sentence_id":"s1","text":"I dipendenti hanno diritto al rimborso delle trasferte se il viaggio e pre-approvato e le ricevute sono inviate entro quattordici giorni.","language":"it","gold_label":"SUPPORTED"}
+{"source_id":"policy_viaggi","sentence_id":"s2","text":"Gli stagisti non hanno diritto al rimborso dei viaggi internazionali salvo diversa indicazione nel contratto di stage.","language":"it","gold_label":"SUPPORTED"}
+{"source_id":"policy_sicurezza","sentence_id":"s1","text":"I laptop aziendali devono usare la crittografia completa del disco e il blocco automatico dello schermo.","language":"it","gold_label":"SUPPORTED"}
+{"source_id":"policy_sicurezza","sentence_id":"s2","text":"I dati dei clienti non possono essere copiati su servizi cloud personali.","language":"it","gold_label":"SUPPORTED"}
+{"source_id":"policy_ferie","sentence_id":"s1","text":"I dipendenti a tempo pieno ricevono ventidue giorni di ferie retribuite per anno solare.","language":"it","gold_label":"SUPPORTED"}
+{"source_id":"policy_spese","sentence_id":"s1","text":"Le spese mensili per internet domestico non sono rimborsate come indennita mensile.","language":"it","gold_label":"SUPPORTED"}

reports/pipeline_business_policy_xlmr_results.json

@@ -0,0 +1,93 @@
+{
+  "count": 13,
+  "evidence_path": "data/sample/evidence_corpus.jsonl",
+  "model_dir": "models/xlmr-claimlens",
+  "accuracy": 1.0,
+  "macro_f1": 1.0,
+  "avg_latency_seconds": 0.12743208015364452,
+  "confusion": {
+    "NOT_ENOUGH_INFO->NOT_ENOUGH_INFO": 1,
+    "REFUTED->REFUTED": 7,
+    "SUPPORTED->SUPPORTED": 5
+  },
+  "rows": [
+    {
+      "claim": "Employees can work remotely up to three days per week with manager approval.",
+      "gold": "SUPPORTED",
+      "pred": "SUPPORTED",
+      "confidence": 0.95
+    },
+    {
+      "claim": "Employees can work remotely five days per week without approval.",
+      "gold": "REFUTED",
+      "pred": "REFUTED",
+      "confidence": 0.92
+    },
+    {
+      "claim": "New employees are eligible for remote work during their first thirty days.",
+      "gold": "REFUTED",
+      "pred": "REFUTED",
+      "confidence": 0.93
+    },
+    {
+      "claim": "Employees receive a monthly home internet allowance.",
+      "gold": "REFUTED",
+      "pred": "REFUTED",
+      "confidence": 0.819145917892456
+    },
+    {
+      "claim": "Interns are eligible for international travel reimbursement by default.",
+      "gold": "REFUTED",
+      "pred": "REFUTED",
+      "confidence": 0.93
+    },
+    {
+      "claim": "Company laptops must use full-disk encryption.",
+      "gold": "SUPPORTED",
+      "pred": "SUPPORTED",
+      "confidence": 0.95
+    },
+    {
+      "claim": "Customer data can be copied to personal cloud storage.",
+      "gold": "REFUTED",
+      "pred": "REFUTED",
+      "confidence": 0.9146171808242798
+    },
+    {
+      "claim": "Employees get twenty-two paid vacation days per year.",
+      "gold": "SUPPORTED",
+      "pred": "SUPPORTED",
+      "confidence": 0.95
+    },
+    {
+      "claim": "Employees receive a free gym membership.",
+      "gold": "NOT_ENOUGH_INFO",
+      "pred": "NOT_ENOUGH_INFO",
+      "confidence": 0.9291238784790039
+    },
+    {
+      "claim": "I dipendenti possono lavorare da remoto fino a tre giorni alla settimana con approvazione del manager.",
+      "gold": "SUPPORTED",
+      "pred": "SUPPORTED",
+      "confidence": 0.95
+    },
+    {
+      "claim": "Gli stagisti hanno sempre diritto al rimborso dei viaggi internazionali.",
+      "gold": "REFUTED",
+      "pred": "REFUTED",
+      "confidence": 0.93
+    },
+    {
+      "claim": "I laptop aziendali devono usare la crittografia completa del disco.",
+      "gold": "SUPPORTED",
+      "pred": "SUPPORTED",
+      "confidence": 0.95
+    },
+    {
+      "claim": "Le spese mensili per internet domestico sono rimborsate come indennita mensile.",
+      "gold": "REFUTED",
+      "pred": "REFUTED",
+      "confidence": 0.8718917369842529
+    }
+  ]
+}

reports/training_report.md

@@ -37,16 +37,23 @@ Label distribution across the English FEVER-NLI subset:
 | Test macro F1 | 0.8057 |
 | Test weighted F1 | 0.8094 |
 
-## Local Product Smoke Test
+## Business Policy Product Smoke Test
 
-The local product pipeline combines retrieval with a calibrated verifier. Raw XLM-RoBERTa is evaluated as the trained verifier; the product API adds a deterministic calibration layer for small domain-specific demos where FEVER-trained models can over-predict `neutral`.
+The local product pipeline now uses a coherent English-Italian business policy corpus covering remote work, travel reimbursement, security, leave, equipment, and expenses. The product API combines retrieval with a calibrated verifier. Raw XLM-RoBERTa is evaluated as the trained verifier; the product API adds a deterministic calibration layer for small domain-specific demos where FEVER-trained models can over-predict `neutral`.
 
 | Metric | Value |
 | --- | ---: |
-| Claims | 6 |
-| Accuracy | 0.6667 |
-| Macro F1 | 0.6556 |
-| Average latency | 0.1704 seconds |
+| Claims | 13 |
+| Accuracy | 1.0000 |
+| Macro F1 | 1.0000 |
+| Average latency | 0.1274 seconds |
+
+Example policy claims:
+
+- `Employees can work remotely five days per week without approval.` -> `REFUTED`
+- `Company laptops must use full-disk encryption.` -> `SUPPORTED`
+- `Employees receive a free gym membership.` -> `NOT_ENOUGH_INFO`
+- `Le spese mensili per internet domestico sono rimborsate come indennita mensile.` -> `REFUTED`
 
 ## Verification Commands
 

src/claimlens/verification/aggregator.py

@@ -11,6 +11,13 @@ class VerdictAggregator:
         if not evidence:
             return "NOT_ENOUGH_INFO", 0.0
 
+        top_relevant = [item for item in evidence if item.retrieval_score >= 0.75 and item.nli_confidence >= self.threshold]
+        for item in top_relevant:
+            if item.nli_label == "entailment":
+                return "SUPPORTED", item.nli_confidence
+            if item.nli_label == "contradiction":
+                return "REFUTED", item.nli_confidence
+
         best_entailment = max(
             (item.nli_confidence for item in evidence if item.nli_label == "entailment"),
             default=0.0,

src/claimlens/verification/nli_model.py

@@ -71,14 +71,17 @@ class LexicalNLIVerifier:
     """
 
     contradiction_markers = {
+        "cannot",
         "no",
         "not",
         "without",
         "never",
+        "unless",
         "nessun",
         "nessuna",
         "non",
         "senza",
+        "salvo",
     }
 
     insufficient_markers = {
@@ -91,6 +94,16 @@ class LexicalNLIVerifier:
     }
 
     support_markers = {
+        "approval",
+        "approved",
+        "eligible",
+        "requires",
+        "receive",
+        "receives",
+        "reimbursed",
+        "reimbursement",
+        "must",
+        "may",
         "reduces",
         "reduce",
         "reducing",
@@ -106,10 +119,49 @@ class LexicalNLIVerifier:
         "migliora",
         "migliorato",
         "miglioramento",
+        "approvazione",
+        "diritto",
+        "rimborso",
+        "devono",
+        "possono",
     }
 
+    number_values = {
+        "one": 1,
+        "two": 2,
+        "three": 3,
+        "four": 4,
+        "five": 5,
+        "six": 6,
+        "seven": 7,
+        "fourteen": 14,
+        "twenty-two": 22,
+        "thirty": 30,
+        "uno": 1,
+        "due": 2,
+        "tre": 3,
+        "quattro": 4,
+        "cinque": 5,
+        "sei": 6,
+        "sette": 7,
+        "quattordici": 14,
+        "ventidue": 22,
+        "trenta": 30,
+    }
+
+    def _numbers(self, tokens: set[str]) -> set[int]:
+        values: set[int] = set()
+        for token in tokens:
+            normalized = token.replace(",", "").replace(".", "")
+            if normalized.isdigit():
+                values.add(int(normalized))
+            if normalized in self.number_values:
+                values.add(self.number_values[normalized])
+        return values
+
     def predict(self, premise: str, hypothesis: str) -> tuple[NLILabel, float]:
         premise_lower = premise.lower()
+        hypothesis_lower = hypothesis.lower()
         premise_tokens = set(tokenize(premise))
         hypothesis_tokens = set(tokenize(hypothesis))
         overlap = len(premise_tokens & hypothesis_tokens) / max(len(hypothesis_tokens), 1)
@@ -121,8 +173,33 @@ class LexicalNLIVerifier:
 
         has_contradiction = bool(premise_tokens & self.contradiction_markers)
         wants_support = bool(hypothesis_tokens & self.support_markers)
+        premise_numbers = self._numbers(premise_tokens)
+        hypothesis_numbers = self._numbers(hypothesis_tokens)
+
+        if premise_numbers and hypothesis_numbers and premise_numbers.isdisjoint(hypothesis_numbers) and overlap >= 0.25:
+            return "contradiction", min(0.92, 0.62 + overlap)
+
+        if any(marker in premise_lower for marker in {"not eligible", "cannot", "may not", "not in"}) and overlap >= 0.22:
+            if any(marker in hypothesis_lower for marker in {"eligible", "can ", "may ", "always"}):
+                return "contradiction", min(0.93, 0.68 + overlap)
+
+        if "not reimbursed" in premise_lower and overlap >= 0.22:
+            if any(marker in hypothesis_lower for marker in {"reimbursed", "reimbursement", "internet", "allowance", "expense"}):
+                return "contradiction", min(0.93, 0.68 + overlap)
+
+        if "without approval" in hypothesis_lower and any(marker in premise_lower for marker in {"requires", "approval", "approved", "approvazione"}):
+            return "contradiction", min(0.94, 0.68 + overlap)
+
+        if "non hanno diritto" in premise_lower and any(marker in hypothesis_lower for marker in {"hanno diritto", "sempre diritto"}):
+            return "contradiction", min(0.93, 0.68 + overlap)
+
+        if "non sono rimborsate" in premise_lower and any(marker in hypothesis_lower for marker in {"sono rimborsate", "rimborsate come"}):
+            return "contradiction", min(0.93, 0.68 + overlap)
+
+        if "non possono" in premise_lower and "possono" in hypothesis_lower:
+            return "contradiction", min(0.93, 0.68 + overlap)
 
-        if has_contradiction and wants_support:
+        if has_contradiction and wants_support and overlap >= 0.3:
             return "contradiction", min(0.95, 0.65 + overlap)
         if overlap >= 0.35:
             return "entailment", min(0.95, 0.55 + overlap / 2)