Spaces:

---

mayafree

/

typed-decision-leaderboard

Like 44

Running

Hi! Could you add WaterSheep to the open-weight division? It's an Apache 2.0 ModernBERT-base fine-tune that answers typed questions with a probability for every option and never generates tokens. Answer checking was only one part of its training mix, so I'd like to see where it actually lands.

Maker's code for your setup, one forward pass per item:

# pip install git+https://github.com/SamratDuttaOfficial/WaterSheep
from watersheep import WaterSheep

ws = WaterSheep.load("samratduttaofficial/WaterSheep")
r = ws.ask({
    "state": {"question": question, "answer": answer},
    "questions": {"correct": {"type": "noul",
                  "instructions": "Is `answer` factually correct for `question`?"}},
})
p_correct = r["answers"]["correct"]["noul"]

Weights: https://huggingface.co/samratduttaofficial/WaterSheep
Code: https://github.com/SamratDuttaOfficial/WaterSheep

English only, and long inputs get truncated. I'm the author, so ping me if anything doesn't run.