| <svg xmlns="http://www.w3.org/2000/svg" width="1200" height="868" viewBox="0 0 1200 868" role="img" aria-labelledby="dataset-subsets-title dataset-subsets-desc"><title id="dataset-subsets-title">Frozen80 evaluation subset composition</title><desc id="dataset-subsets-desc">Five subsets each contain 16 questions; Browser, HelpSteer3, BoolQ and MNLI each have 16 parent groups, while Score has 15. Total 80 questions and 79 groups.</desc><rect x="0" y="0" width="1200" height="868" rx="20" fill="#f7f9f8"/><g font-family="Arial, Helvetica, sans-serif"><text x="40" y="52" font-size="30" font-weight="700" fill="#19333f" text-anchor="start">Frozen80: five decision capabilities</text><text x="40" y="84" font-size="18" font-weight="400" fill="#536b77" text-anchor="start">80 questions / 79 parent groups / the same frozen IDs for every model</text><rect x="40" y="112" width="216" height="60" rx="8" fill="#007c72"/><text x="148" y="138" font-size="17" font-weight="700" fill="#ffffff" text-anchor="middle">Browser</text><text x="148" y="158" font-size="14" font-weight="400" fill="#ffffff" text-anchor="middle">16 questions 路 20%</text><rect x="264" y="112" width="216" height="60" rx="8" fill="#176d69"/><text x="372" y="138" font-size="17" font-weight="700" fill="#ffffff" text-anchor="middle">HelpSteer3</text><text x="372" y="158" font-size="14" font-weight="400" fill="#ffffff" text-anchor="middle">16 questions 路 20%</text><rect x="488" y="112" width="216" height="60" rx="8" fill="#007c72"/><text x="596" y="138" font-size="17" font-weight="700" fill="#ffffff" text-anchor="middle">BoolQ</text><text x="596" y="158" font-size="14" font-weight="400" fill="#ffffff" text-anchor="middle">16 questions 路 20%</text><rect x="712" y="112" width="216" height="60" rx="8" fill="#176d69"/><text x="820" y="138" font-size="17" font-weight="700" fill="#ffffff" text-anchor="middle">MNLI</text><text x="820" y="158" font-size="14" font-weight="400" fill="#ffffff" text-anchor="middle">16 questions 路 20%</text><rect x="936" y="112" width="216" height="60" rx="8" fill="#007c72"/><text x="1044" y="138" font-size="17" font-weight="700" fill="#ffffff" text-anchor="middle">Score</text><text x="1044" y="158" font-size="14" font-weight="400" fill="#ffffff" text-anchor="middle">16 questions 路 20%</text><rect x="40" y="204" width="1120" height="526" rx="12" fill="#ffffff" stroke="#d6e1e1"/><rect x="41" y="205" width="1118" height="62" rx="10" fill="#e8f2ef"/><text x="64" y="244" font-size="15" font-weight="700" fill="#536b77" text-anchor="start">SUBSET & SOURCE</text><text x="400" y="244" font-size="15" font-weight="700" fill="#536b77" text-anchor="start">TARGET CAPABILITY</text><text x="1032" y="244" font-size="14" font-weight="700" fill="#536b77" text-anchor="middle">QUESTIONS / GROUPS</text><text x="64" y="302" font-size="24" font-weight="700" fill="#19333f" text-anchor="start">Browser</text><text x="64" y="333" font-size="16" font-weight="400" fill="#536b77" text-anchor="start">OSU NLP / Mind2Web</text><text x="400" y="302" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">Select the next browser action</text><text x="400" y="329" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">from page context and a goal.</text><text x="1032" y="314" font-size="22" font-weight="700" fill="#19333f" text-anchor="middle">16 / 16</text><path d="M40 360 H1160" stroke="#d6e1e1"/><text x="64" y="394" font-size="24" font-weight="700" fill="#19333f" text-anchor="start">HelpSteer3</text><text x="64" y="425" font-size="16" font-weight="400" fill="#536b77" text-anchor="start">NVIDIA / principle-based judgments</text><text x="400" y="394" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">Judge content against a supplied</text><text x="400" y="421" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">natural-language principle.</text><text x="1032" y="406" font-size="22" font-weight="700" fill="#19333f" text-anchor="middle">16 / 16</text><path d="M40 452 H1160" stroke="#d6e1e1"/><text x="64" y="486" font-size="24" font-weight="700" fill="#19333f" text-anchor="start">BoolQ</text><text x="64" y="517" font-size="16" font-weight="400" fill="#536b77" text-anchor="start">Boolean Questions</text><text x="400" y="486" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">Answer yes/no questions</text><text x="400" y="513" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">using supporting context.</text><text x="1032" y="498" font-size="22" font-weight="700" fill="#19333f" text-anchor="middle">16 / 16</text><path d="M40 544 H1160" stroke="#d6e1e1"/><text x="64" y="578" font-size="24" font-weight="700" fill="#19333f" text-anchor="start">MNLI</text><text x="64" y="609" font-size="16" font-weight="400" fill="#536b77" text-anchor="start">Multi-Genre NLI</text><text x="400" y="578" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">Identify entailment, contradiction</text><text x="400" y="605" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">or a neutral relationship.</text><text x="1032" y="590" font-size="22" font-weight="700" fill="#19333f" text-anchor="middle">16 / 16</text><path d="M40 636 H1160" stroke="#d6e1e1"/><text x="64" y="670" font-size="24" font-weight="700" fill="#19333f" text-anchor="start">Score</text><text x="64" y="701" font-size="16" font-weight="400" fill="#536b77" text-anchor="start">Google / GoEmotions</text><text x="400" y="670" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">Judge whether a specified</text><text x="400" y="697" font-size="19" font-weight="400" fill="#19333f" text-anchor="start">emotion attribute is present.</text><text x="1032" y="682" font-size="22" font-weight="700" fill="#19333f" text-anchor="middle">16 / 15</text><text x="40" y="772" font-size="17" font-weight="400" fill="#536b77" text-anchor="start">Scope: static browser actions, principle judgments, contextual reasoning and attribute decisions.</text><text x="40" y="802" font-size="17" font-weight="400" fill="#536b77" text-anchor="start">Score has 16 No labels and no Yes labels. This is a development regression panel, not a final test.</text><text x="40" y="840" font-size="14" font-weight="400" fill="#536b77" text-anchor="start">Source: subset-chart-data.json 路 Equal task sizes do not imply equal task difficulty.</text></g></svg> | |