| import gradio as gr |
|
|
| |
| |
| |
|
|
| CLAIMS = [ |
| {"claim": "Nigeria's EIA process is governed by the EIA Act No. 86 of 1992.", |
| "country": "Nigeria", "actual": "TRUE", "predicted": "TRUE", "correct": True, |
| "note": "Correct. Direct restatement of source text, high keyword overlap correctly read as TRUE."}, |
| {"claim": "Ghana requires EIA approval before mining operations can begin.", |
| "country": "Ghana", "actual": "TRUE", "predicted": "TRUE", "correct": True, |
| "note": "Correct. Matches Ghana Minerals and Mining Act 703 of 2006 source content."}, |
| {"claim": "Kenya's EIA process is regulated by the Environmental Management and Coordination Act (EMCA) 1999.", |
| "country": "Kenya", "actual": "TRUE", "predicted": "TRUE", "correct": True, |
| "note": "Correct. Matches Kenya EMCA 1999 source content."}, |
| {"claim": "Rwanda has banned single-use plastics since 2008.", |
| "country": "Rwanda", "actual": "TRUE", "predicted": "TRUE", "correct": True, |
| "note": "Correct. Matches Rwanda REMA source content."}, |
| {"claim": "The African Union's Maputo Convention addresses conservation of nature and natural resources.", |
| "country": "Pan-Africa", "actual": "TRUE", "predicted": "TRUE", "correct": True, |
| "note": "Correct. Matches AU Maputo Convention 2003 source content."}, |
| {"claim": "Rwanda has no EIA requirements for development projects.", |
| "country": "Rwanda", "actual": "FALSE", "predicted": "TRUE", "correct": False, |
| "note": "WRONG. This is the negation-blindness failure. The claim shares nearly all vocabulary with the true statement (Rwanda, EIA, requirements, development projects) but the word 'no' inverts the meaning entirely. Keyword overlap cannot detect this."}, |
| {"claim": "Nigeria allows unrestricted importation of hazardous waste as long as it is for industrial reuse.", |
| "country": "Nigeria", "actual": "FALSE", "predicted": "UNVERIFIABLE", "correct": False, |
| "note": "WRONG. Source text uses different phrasing (prohibits importation, transit, deposit) so overlap was low, landing in UNVERIFIABLE territory by accident rather than correctly flagging this as a direct contradiction."}, |
| {"claim": "South Africa's EIA Basic Assessment process has no fixed legal timeframe for a decision.", |
| "country": "South Africa", "actual": "FALSE", "predicted": "TRUE", "correct": False, |
| "note": "WRONG. Same negation-blindness pattern. Shares vocabulary with the true 107-day timeframe statement, but 'no fixed timeframe' is the opposite claim."}, |
| {"claim": "Senegal's environmental assessment process has no public participation requirement.", |
| "country": "Senegal", "actual": "FALSE", "predicted": "TRUE", "correct": False, |
| "note": "WRONG. Again, negation inverts a true statement (public participation IS mandatory for Category 1) into a false one, undetected by keyword overlap."}, |
| {"claim": "ECOWAS has no shared environmental policy and leaves all environmental regulation entirely to individual member states.", |
| "country": "ECOWAS Region", "actual": "FALSE", "predicted": "TRUE", "correct": False, |
| "note": "WRONG. Same pattern. High vocabulary overlap with the true ECOWAS Environmental Policy 2008 statement, but negated into a false claim."}, |
| {"claim": "Nigeria's NESREA has approved over 5,000 EIA certificates in the last year.", |
| "country": "Nigeria", "actual": "UNVERIFIABLE", "predicted": "TRUE", "correct": False, |
| "note": "WRONG. A specific operational statistic not in the source dataset. Shares enough vocabulary (NESREA, EIA, certificates) with general true statements about NESREA's role to be wrongly read as TRUE."}, |
| {"claim": "Ethiopia is planning to repeal its EIA Proclamation 299/2002 next year.", |
| "country": "Ethiopia", "actual": "UNVERIFIABLE", "predicted": "TRUE", "correct": False, |
| "note": "WRONG. A forward-looking legislative prediction, not covered by a dataset documenting the current framework. High overlap on the law's name wrongly suggests confirmation."}, |
| {"claim": "Ghana's EPA processes EIA applications faster than South Africa's DFFE.", |
| "country": "Ghana / South Africa", "actual": "UNVERIFIABLE", "predicted": "UNVERIFIABLE", "correct": True, |
| "note": "Correct. A cross-country comparative speed claim with no shared source content at all, correctly flagged as unverifiable."}, |
| ] |
|
|
| def get_summary(): |
| total = len(CLAIMS) |
| correct = sum(1 for c in CLAIMS if c["correct"]) |
| true_claims = [c for c in CLAIMS if c["actual"] == "TRUE"] |
| false_claims = [c for c in CLAIMS if c["actual"] == "FALSE"] |
| unverif_claims = [c for c in CLAIMS if c["actual"] == "UNVERIFIABLE"] |
|
|
| true_correct = sum(1 for c in true_claims if c["correct"]) |
| false_correct = sum(1 for c in false_claims if c["correct"]) |
| unverif_correct = sum(1 for c in unverif_claims if c["correct"]) |
|
|
| return f""" |
| ## 📊 Overall Result: {correct}/{total} correct ({correct/total*100:.1f}%) |
| |
| | Claim Type | Accuracy | |
| |------------|----------| |
| | TRUE claims | {true_correct}/{len(true_claims)} ({true_correct/len(true_claims)*100:.0f}%) | |
| | FALSE claims | {false_correct}/{len(false_claims)} ({false_correct/len(false_claims)*100:.0f}%) | |
| | UNVERIFIABLE claims | {unverif_correct}/{len(unverif_claims)} ({unverif_correct/len(unverif_claims)*100:.0f}%) | |
| |
| ### 🔍 The Headline Finding: Negation Blindness |
| |
| A simple keyword-overlap verifier got **every single TRUE claim right** but **every single FALSE claim wrong**, always defaulting to TRUE. |
| |
| The reason is structural, not random. Claims like *"Rwanda has no EIA requirements"* share almost all the same vocabulary as the true statement *"Rwanda has EIA requirements"*, same country, same law, same nouns, just inverted by one negation word. Surface-level keyword matching cannot detect that inversion. |
| |
| This means a verification layer built only on lexical overlap is **systematically blind to the most dangerous category of misinformation**, a confidently worded false claim that uses all the right vocabulary. |
| """ |
|
|
| def show_claim(index): |
| c = CLAIMS[int(index)] |
| status_emoji = "✅" if c["correct"] else "❌" |
| return f""" |
| ### {status_emoji} Claim {int(index)+1} of {len(CLAIMS)} |
| |
| **Claim:** {c['claim']} |
| |
| **Country:** {c['country']} |
| |
| **Actual verdict:** {c['actual']} |
| **Predicted verdict:** {c['predicted']} |
| **Verifier was:** {"Correct ✅" if c['correct'] else "Wrong ❌"} |
| |
| **Analysis:** {c['note']} |
| """ |
|
|
| with gr.Blocks(title="Africa Law Claim Verifier Eval", theme=gr.themes.Soft()) as demo: |
| gr.Markdown(""" |
| # ⚖️ Africa Environmental Law Claim Verifier |
| ## Standalone Eval — AutoScientist Challenge 2026 |
| |
| **Author:** Hussein Adeiza (mabera) — Licensed Environmental Health Officer, Abuja Nigeria |
| **Built on:** Africa Environmental Law Model (Legal Category submission) |
| |
| This is a complementary analysis, not the official challenge metric. It stress-tests |
| a claim verification approach against 13 real claims about African environmental law, |
| 5 true, 5 false, 3 genuinely unverifiable, all grounded in the structured legal dataset |
| from this challenge's Legal category submission. |
| |
| Inspired by the community's honest-finding eval Space pattern. Reporting the result as |
| it actually came out, including where it failed. |
| """) |
|
|
| gr.Markdown(get_summary()) |
|
|
| gr.Markdown("---\n### Browse individual claims") |
| with gr.Row(): |
| with gr.Column(): |
| claim_slider = gr.Slider(0, len(CLAIMS)-1, value=0, step=1, label="Claim Index") |
| with gr.Column(): |
| claim_output = gr.Markdown() |
| claim_slider.change(show_claim, inputs=claim_slider, outputs=claim_output) |
| demo.load(lambda: show_claim(0), outputs=claim_output) |
|
|
| gr.Markdown(""" |
| --- |
| ### Why This Matters |
| NGOs, investors and businesses increasingly rely on AI for quick answers about African |
| regulatory environments. A model that can fluently restate true facts but cannot detect |
| a negated false claim is a real risk, it sounds equally confident either way. This finding |
| suggests verification layers need negation-aware reasoning, not just topical relevance |
| matching, before being trusted for compliance-adjacent use cases. |
| |
| 🤗 [Legal Model](https://huggingface.co/mabera/africa-environmental-law-model) | |
| 📊 [Source Dataset](https://huggingface.co/datasets/mabera/africa-environmental-law-dataset) | |
| 🚀 [Main Demo](https://huggingface.co/spaces/mabera/nigeria-health-ai-demo) |
| |
| Powered by Adaptive Data — Adaption Labs |
| """) |
|
|
| demo.launch() |
