Hugging Face's logo

vllm-sr
/
Decision-1.0-Sol-2B

Zero-Shot Classification
Safetensors
PyTorch
decision1
multilingual
decision-model
classification
qwen3_5
custom_code
Model card Files
xet
 Community
2
Decision-1.0-Sol-2B / evaluation /TASKS.md
Xunzhuo's picture
Xunzhuo
Publish clean Decision model repository
53ecba4
preview code
|
Raw History Blame
10.4 kB

All 54 tasks

Accuracy (%) on the same requested rows. Bold marks a Decision-family result strictly above every external open or untuned reference; Jev and the other Decision models do not set that threshold. Ties are not bold. Each task uses its full requested denominator. These task rows are diagnostic; the headline retains its declared within-panel weights.

Decisions · 10 tasks
Task n Lux-9B Nox-4B Kev-9B Kev-4B Qwen3.5-9B Decider Qwen3.5-4B Sol-2B Eos-0.8B Kev-0.8B Qwen3.5-2B Kai-0.6B Laya · English Laya · Multilingual Jev
News classification 128 89.06 85.16 87.50 87.50 85.94 86.72 84.38 83.59 82.03 86.72 80.47 76.56 91.41 89.84 85.16
Boolean constraints 64 100.00 93.75 95.31 71.88 98.44 71.88 62.50 50.00 82.81 81.25 37.50 53.12 43.75 39.06 100.00
Entity classification 112 96.43 96.43 98.21 98.21 96.43 98.21 97.32 95.54 91.96 96.43 92.86 84.82 83.93 57.14 96.43
Intent routing 64 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 96.88 89.06 81.25 81.25 100.00
Evidence placement 96 72.92 100.00 50.00 41.67 79.17 8.33 72.92 98.96 19.79 33.33 9.38 100.00 95.83 54.17 30.21
Ordered rubric 64 100.00 89.06 96.88 100.00 84.38 84.38 90.62 65.62 84.38 59.38 81.25 6.25 18.75 12.50 100.00
Relation composition 96 60.42 52.08 51.04 62.50 36.46 54.17 51.04 37.50 64.58 44.79 48.96 40.62 25.00 35.42 56.25
Scoped evidence 96 96.88 78.12 65.62 47.92 55.21 47.92 51.04 79.17 25.00 10.42 36.46 25.00 37.50 36.46 89.58
State tracking 96 28.12 35.42 38.54 34.38 31.25 29.17 28.12 27.08 38.54 31.25 25.00 29.17 23.96 29.17 33.33
In / out of menu 64 100.00 100.00 84.38 75.00 71.88 59.38 60.94 100.00 70.31 57.81 62.50 75.00 64.06 37.50 100.00
Composition · 10 tasks
Task n Lux-9B Nox-4B Kev-9B Kev-4B Qwen3.5-9B Decider Qwen3.5-4B Sol-2B Eos-0.8B Kev-0.8B Qwen3.5-2B Kai-0.6B Laya · English Laya · Multilingual Jev
Record identity 80 61.25 53.75 45.00 50.00 57.50 56.25 50.00 50.00 60.00 50.00 46.25 51.25 47.50 50.00 76.25
Capacity assignment 80 73.75 46.25 50.00 50.00 50.00 51.25 50.00 47.50 50.00 50.00 50.00 50.00 63.75 50.00 68.75
Constraint assignment 80 32.50 41.25 27.50 32.50 25.00 23.75 23.75 26.25 25.00 20.00 21.25 26.25 22.50 27.50 55.00
Intent routing · EN 120 90.00 91.67 86.67 87.50 88.33 90.00 91.67 90.83 91.67 83.33 70.83 81.67 74.17 70.83 91.67
Intent routing · ZH 120 87.50 87.50 85.83 86.67 86.67 88.33 86.67 87.50 85.00 85.83 74.17 79.17 44.17 73.33 88.33
Multiset reconciliation 80 28.75 28.75 37.50 28.75 20.00 23.75 27.50 28.75 26.25 25.00 27.50 25.00 26.25 32.50 58.75
Ordered service loss 80 26.25 30.00 25.00 20.00 20.00 22.50 21.25 25.00 20.00 27.50 20.00 20.00 20.00 17.50 42.50
Conflicting rule closure 80 36.25 33.75 27.50 33.75 27.50 26.25 25.00 25.00 27.50 27.50 27.50 25.00 18.75 20.00 66.25
Temporal exclusion 80 37.50 42.50 28.75 45.00 36.25 42.50 31.25 41.25 32.50 21.25 28.75 17.50 12.50 28.75 41.25
Transaction recovery 80 53.75 62.50 43.75 51.25 35.00 41.25 26.25 38.75 42.50 32.50 23.75 32.50 23.75 18.75 75.00
Reading · 3 tasks
Task n Lux-9B Nox-4B Kev-9B Kev-4B Qwen3.5-9B Decider Qwen3.5-4B Sol-2B Eos-0.8B Kev-0.8B Qwen3.5-2B Kai-0.6B Laya · English Laya · Multilingual Jev
Yes / no reading 160 89.38 86.25 91.25 88.75 84.38 91.88 83.75 83.12 75.62 74.38 65.00 74.38 69.38 69.38 92.50
Reading · EN 160 92.50 73.12 83.12 75.62 98.75 93.12 93.12 70.00 68.12 63.75 83.75 38.12 38.12 36.25 96.88
Reading · ZH 160 89.38 70.62 81.25 74.38 91.88 91.25 91.25 70.00 61.88 58.75 81.25 31.87 28.75 28.12 96.25
Inference · 4 tasks
Task n Lux-9B Nox-4B Kev-9B Kev-4B Qwen3.5-9B Decider Qwen3.5-4B Sol-2B Eos-0.8B Kev-0.8B Qwen3.5-2B Kai-0.6B Laya · English Laya · Multilingual Jev
Contextual reasoning 120 82.50 70.83 68.33 70.83 64.17 66.67 60.83 69.17 70.00 42.50 56.67 46.67 30.00 20.00 86.67
Answerability 120 93.33 88.33 78.33 81.67 74.17 84.17 81.67 83.33 81.67 66.67 82.50 55.83 57.50 55.00 90.83
Textual entailment 120 92.50 89.17 90.83 89.17 81.67 90.83 80.00 88.33 84.17 77.50 64.17 78.33 72.50 72.50 82.50
Scientific inference 120 97.50 96.67 96.67 96.67 98.33 95.83 96.67 95.83 90.83 88.33 85.83 98.33 95.00 81.67 99.17
Transfer · 27 tasks
Task n Lux-9B Nox-4B Kev-9B Kev-4B Qwen3.5-9B Decider Qwen3.5-4B Sol-2B Eos-0.8B Kev-0.8B Qwen3.5-2B Kai-0.6B Laya · English Laya · Multilingual Jev
Buried emotion 20 45.00 25.00 65.00 55.00 40.00 85.00 45.00 20.00 40.00 40.00 55.00 45.00 60.00 35.00 70.00
Buried paraphrase 20 80.00 75.00 80.00 75.00 80.00 75.00 75.00 70.00 70.00 60.00 70.00 35.00 55.00 70.00 95.00
Buried entailment 20 90.00 90.00 95.00 90.00 95.00 90.00 75.00 75.00 85.00 85.00 65.00 55.00 60.00 55.00 95.00
Buried offensive-language detection 20 40.00 45.00 80.00 65.00 45.00 80.00 40.00 35.00 15.00 60.00 35.00 75.00 70.00 80.00 55.00
Combined policy conditions 32 96.88 75.00 68.75 78.12 53.12 53.12 59.38 78.12 37.50 40.62 62.50 62.50 62.50 43.75 96.88
Policy exceptions 32 90.62 96.88 87.50 96.88 59.38 56.25 75.00 71.88 53.12 65.62 40.62 53.12 43.75 37.50 100.00
Policy negation 32 96.88 87.50 90.62 87.50 62.50 62.50 53.12 71.88 68.75 71.88 46.88 56.25 53.12 50.00 90.62
Authorization contrast 40 100.00 97.50 100.00 100.00 100.00 100.00 97.50 50.00 50.00 97.50 62.50 50.00 67.50 50.00 100.00
Deadline contrast 40 90.00 70.00 87.50 75.00 77.50 30.00 52.50 40.00 50.00 45.00 25.00 50.00 35.00 22.50 92.50
Emotion 80 60.00 60.00 67.50 65.00 52.50 86.25 65.00 22.50 45.00 60.00 67.50 53.75 62.50 56.25 67.50
MMLU 80 75.00 61.25 75.00 68.75 73.75 62.50 66.25 52.50 47.50 51.25 53.75 37.50 22.50 27.50 88.75
MMLU-Pro 200 53.00 37.00 53.00 45.50 53.50 37.50 45.00 24.50 17.00 22.50 28.00 12.50 11.00 11.50 84.00
Paraphrase 80 91.25 85.00 81.25 81.25 88.75 77.50 83.75 80.00 71.25 58.75 70.00 47.50 86.25 75.00 87.50
Question entailment 80 91.25 91.25 95.00 91.25 90.00 87.50 87.50 83.75 81.25 81.25 63.75 71.25 80.00 73.75 91.25
Science questions 80 100.00 98.75 100.00 100.00 100.00 98.75 98.75 97.50 97.50 96.25 96.25 90.00 90.00 72.50 100.00
Offensive-language detection 80 75.00 75.00 86.25 85.00 83.75 88.75 77.50 75.00 52.50 72.50 83.75 78.75 81.25 82.50 76.25
Evidence control · age eligibility 10 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 50.00 90.00 50.00 100.00
Evidence control · authorization 10 100.00 100.00 100.00 100.00 100.00 80.00 100.00 50.00 50.00 100.00 50.00 50.00 70.00 50.00 100.00
Evidence control · deadline 10 80.00 60.00 70.00 60.00 80.00 30.00 50.00 30.00 40.00 40.00 30.00 50.00 30.00 30.00 100.00
Evidence control · late fee 10 100.00 90.00 100.00 100.00 90.00 70.00 80.00 80.00 60.00 90.00 70.00 30.00 40.00 40.00 100.00
Evidence control · quantity limit 10 100.00 80.00 100.00 100.00 70.00 80.00 70.00 70.00 50.00 100.00 20.00 20.00 60.00 30.00 100.00
Evidence control · return window 10 60.00 40.00 70.00 70.00 60.00 60.00 40.00 40.00 50.00 80.00 50.00 50.00 60.00 40.00 50.00
Evidence control · shipping delay 10 90.00 40.00 90.00 80.00 80.00 80.00 60.00 80.00 20.00 100.00 30.00 20.00 30.00 30.00 90.00
Evidence control · sla response 10 70.00 40.00 100.00 100.00 50.00 30.00 50.00 40.00 30.00 100.00 50.00 30.00 50.00 30.00 100.00
Evidence control · spend threshold 10 100.00 100.00 100.00 100.00 100.00 100.00 100.00 90.00 70.00 100.00 100.00 50.00 50.00 50.00 100.00
Evidence control · volume discount 10 100.00 80.00 100.00 100.00 100.00 90.00 100.00 90.00 90.00 100.00 20.00 40.00 20.00 30.00 100.00
Evidence control · warranty claim 10 50.00 70.00 80.00 100.00 60.00 40.00 50.00 50.00 50.00 50.00 50.00 20.00 40.00 30.00 90.00