evaluation/constrained_selector_joint_eval_report.md

Constrained Selector Joint Verifier Evaluation

• Model repo: rishhh/verisci-claim-verifier-constrained-selector-joint-seed123
• Selector base: microsoft/deberta-v3-small
• Verifier base: rishhh/verisci-claim-verifier-retrieval-adapted-seed123
• Retriever: rishhh/verisci-scifact-e5-retriever

SciFact Validation

• Raw original-label accuracy: 0.700
• Raw original-label macro F1: 0.661
• Thresholded original-label accuracy: 0.700
• Thresholded original-label macro F1: 0.661
• Raw retrieved-evidence-label accuracy: 0.736
• Raw retrieved-evidence-label macro F1: 0.709
• Thresholded retrieved-evidence-label accuracy: 0.736
• Thresholded retrieved-evidence-label macro F1: 0.709
• Retrieval oracle accuracy: 0.929
• Promotion gate passed: False
• Promotion gate reason: candidate_accuracy_below_target: 0.700 < 0.910

Decision Policy

{
  "strategy": "argmax_fallback_no_policy_met_nei_recall_floor",
  "min_nei_recall": 0.74
}

Selection Protocol

• Scope: SciFact train only; train/dev split by claim id
• Validation used for model selection: False

Limitations

• Selector supervision uses SciFact evidence sentence IDs when the evidence document is retrieved.
• SciFact validation is public and is not a hidden benchmark.
• This is still limited by retrieval recall and by the small SciFact train set.