PhysicalEYE
Collection
2 items • Updated • 1
physical ai
masterset-ai
AI & ML interests
https://www.masterset.ai
Recent Activity
updated a collection 33 minutes ago
PhysicalEYE upvoted a collection 33 minutes ago
PhysicalEYE liked a model 37 minutes ago
masterset-ai/PhysicalEye-Decide-35BOrganizations
None yet
updated a
collection 33 minutes ago
upvoted a collection 33 minutes ago
liked a
model 37 minutes ago
liked a
Space 37 minutes ago
updated a
Space about 1 hour ago
published a
model about 2 hours ago
updated a
model about 2 hours ago
published a
Space about 8 hours ago
liked a
Space about 8 hours ago
reacted to SeaWolf-AI's post with ❤️🤗 about 12 hours ago
Post
1625
🏆 Darwin-27B-ZTC-v2 just took #1 on the System One Mosaic Benchmark (S1MB).
S1MB compares 102 models across 137 specialized benchmarks, in three task types: Noul (assess a condition), Choice (select an option), Score (rate on a scale). Ranking is by overall Borda score.
📊 Top of the board
🥇 Darwin ZTC v2 (FINAL-Bench) 89.58
🥈 OpenJev-27B 87.50
🥉 AutoJev-27B 87.07
4️⃣ Eikos 27B 85.43
5️⃣ Jev 1.13 85.05
🔎 Ranks 2 to 5 are all the JEV family (TypeSafe AI's System One model, from ex-OpenAI researchers). S1MB exists to compare these System One judges, so leading it is the headline.
⚙️ Why a zero-token judge wins here
🔹 It does not generate. It reads the input and typed questions and returns a calibrated distribution in a single forward pass.
🔹 Zero generated tokens, no decoding loop, so latency and cost stay low.
🔹 Holds up out of distribution too: General Noul 96.00, General Choice 99.34.
It is also #1 on the typed-decisions leaderboard (0.743, zero-shot). Same message from both: a deterministic, calibrated judge at one forward pass per call.
🔗 Model: FINAL-Bench/Darwin-27B-ZTC-v2
🔗 Leaderboard: hotchpotch/S1MB-leaderboard
Standings move as new models are added. Numbers reflect the board at the time of writing. 🙌
S1MB compares 102 models across 137 specialized benchmarks, in three task types: Noul (assess a condition), Choice (select an option), Score (rate on a scale). Ranking is by overall Borda score.
📊 Top of the board
🥇 Darwin ZTC v2 (FINAL-Bench) 89.58
🥈 OpenJev-27B 87.50
🥉 AutoJev-27B 87.07
4️⃣ Eikos 27B 85.43
5️⃣ Jev 1.13 85.05
🔎 Ranks 2 to 5 are all the JEV family (TypeSafe AI's System One model, from ex-OpenAI researchers). S1MB exists to compare these System One judges, so leading it is the headline.
⚙️ Why a zero-token judge wins here
🔹 It does not generate. It reads the input and typed questions and returns a calibrated distribution in a single forward pass.
🔹 Zero generated tokens, no decoding loop, so latency and cost stay low.
🔹 Holds up out of distribution too: General Noul 96.00, General Choice 99.34.
It is also #1 on the typed-decisions leaderboard (0.743, zero-shot). Same message from both: a deterministic, calibrated judge at one forward pass per call.
🔗 Model: FINAL-Bench/Darwin-27B-ZTC-v2
🔗 Leaderboard: hotchpotch/S1MB-leaderboard
Standings move as new models are added. Numbers reflect the board at the time of writing. 🙌
upvoted an article about 20 hours ago
Article
Leading the System One Mosaic Benchmark: What Darwin-27B-ZTC-v2's #1 Means
FINAL-Bench
• • 6 liked 4
models about 20 hours ago
reacted to SeaWolf-AI's post with 🚀🔥 about 23 hours ago
Post
1625
🏆 Darwin-27B-ZTC-v2 just took #1 on the System One Mosaic Benchmark (S1MB).
S1MB compares 102 models across 137 specialized benchmarks, in three task types: Noul (assess a condition), Choice (select an option), Score (rate on a scale). Ranking is by overall Borda score.
📊 Top of the board
🥇 Darwin ZTC v2 (FINAL-Bench) 89.58
🥈 OpenJev-27B 87.50
🥉 AutoJev-27B 87.07
4️⃣ Eikos 27B 85.43
5️⃣ Jev 1.13 85.05
🔎 Ranks 2 to 5 are all the JEV family (TypeSafe AI's System One model, from ex-OpenAI researchers). S1MB exists to compare these System One judges, so leading it is the headline.
⚙️ Why a zero-token judge wins here
🔹 It does not generate. It reads the input and typed questions and returns a calibrated distribution in a single forward pass.
🔹 Zero generated tokens, no decoding loop, so latency and cost stay low.
🔹 Holds up out of distribution too: General Noul 96.00, General Choice 99.34.
It is also #1 on the typed-decisions leaderboard (0.743, zero-shot). Same message from both: a deterministic, calibrated judge at one forward pass per call.
🔗 Model: FINAL-Bench/Darwin-27B-ZTC-v2
🔗 Leaderboard: hotchpotch/S1MB-leaderboard
Standings move as new models are added. Numbers reflect the board at the time of writing. 🙌
S1MB compares 102 models across 137 specialized benchmarks, in three task types: Noul (assess a condition), Choice (select an option), Score (rate on a scale). Ranking is by overall Borda score.
📊 Top of the board
🥇 Darwin ZTC v2 (FINAL-Bench) 89.58
🥈 OpenJev-27B 87.50
🥉 AutoJev-27B 87.07
4️⃣ Eikos 27B 85.43
5️⃣ Jev 1.13 85.05
🔎 Ranks 2 to 5 are all the JEV family (TypeSafe AI's System One model, from ex-OpenAI researchers). S1MB exists to compare these System One judges, so leading it is the headline.
⚙️ Why a zero-token judge wins here
🔹 It does not generate. It reads the input and typed questions and returns a calibrated distribution in a single forward pass.
🔹 Zero generated tokens, no decoding loop, so latency and cost stay low.
🔹 Holds up out of distribution too: General Noul 96.00, General Choice 99.34.
It is also #1 on the typed-decisions leaderboard (0.743, zero-shot). Same message from both: a deterministic, calibrated judge at one forward pass per call.
🔗 Model: FINAL-Bench/Darwin-27B-ZTC-v2
🔗 Leaderboard: hotchpotch/S1MB-leaderboard
Standings move as new models are added. Numbers reflect the board at the time of writing. 🙌
liked a
model 1 day ago
reacted to SeaWolf-AI's post with 🚀 1 day ago
Post
3834
🧠 We just released Darwin-27B-ZTC, a judgment engine that reaches a verdict without generating anything.
Most LLMs answer by generating, decoding one token at a time. Darwin-27B-ZTC takes a different route.
⚙️ How it works
🔹 It makes its call in a single forward pass.
🔹 Zero generated tokens, and no decoding loop.
🔹 That keeps latency and cost far below what a generative model needs.
🎯 What it judges
🔹 It handles several question types: free-form correctness (noul), multiple choice (choice), and scoring (score).
🔹 For each one it hands back a calibrated confidence, not just an answer.
📊 How well calibrated (measured)
🔹 KL 0.204, Brier 0.097, so the confidence it reports lines up with what actually happens.
🔹 0.743 accuracy (zero-shot, general split), across 2,000 judgments with zero errors.
🔹 By type: noul 0.847, choice 0.723, score 0.675.
🔹 None of the benchmark's train split went into it. It is pure zero-shot.
🚀 Where it fits
🔹 Grading at scale, model routing, safety gating, anywhere you want a fast decision without paying for generation.
🏆 It currently sits at #1 on the official typed-decisions leaderboard on Hugging Face (0.743 accuracy, zero-shot).
🔗 Links
Model: FINAL-Bench/Darwin-27B-ZTC
Leaderboard: LocalLLaMA/typed-decisions
Curious to hear what you make of the single-pass, no-generation approach. 🙌
Most LLMs answer by generating, decoding one token at a time. Darwin-27B-ZTC takes a different route.
⚙️ How it works
🔹 It makes its call in a single forward pass.
🔹 Zero generated tokens, and no decoding loop.
🔹 That keeps latency and cost far below what a generative model needs.
🎯 What it judges
🔹 It handles several question types: free-form correctness (noul), multiple choice (choice), and scoring (score).
🔹 For each one it hands back a calibrated confidence, not just an answer.
📊 How well calibrated (measured)
🔹 KL 0.204, Brier 0.097, so the confidence it reports lines up with what actually happens.
🔹 0.743 accuracy (zero-shot, general split), across 2,000 judgments with zero errors.
🔹 By type: noul 0.847, choice 0.723, score 0.675.
🔹 None of the benchmark's train split went into it. It is pure zero-shot.
🚀 Where it fits
🔹 Grading at scale, model routing, safety gating, anywhere you want a fast decision without paying for generation.
🏆 It currently sits at #1 on the official typed-decisions leaderboard on Hugging Face (0.743 accuracy, zero-shot).
🔗 Links
Model: FINAL-Bench/Darwin-27B-ZTC
Leaderboard: LocalLLaMA/typed-decisions
Curious to hear what you make of the single-pass, no-generation approach. 🙌
- 2 replies