oktayd commited on
Commit
9c72efb
Β·
verified Β·
1 Parent(s): 682a3ae

Explain MoE and inherited stages; add per-test benchmark dropdowns and laptop capability roadmap

Browse files
Files changed (4) hide show
  1. BENCHMARK-TEST-COUNTS.json +1362 -0
  2. LAPTOP-CAPABILITY-PLAN.md +51 -0
  3. README.md +241 -1
  4. SHA256SUMS +3 -1
BENCHMARK-TEST-COUNTS.json ADDED
@@ -0,0 +1,1362 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": "q36.public-strict-counts.v1",
3
+ "source_sha256": "9aef69af26b03d3b26a9250a1f6c21fdd4972a63522806d7724454b66b820b17",
4
+ "scoring": "Original strict evaluator, not semantic regrade. Ungraded and incomplete may overlap.",
5
+ "runs": [
6
+ {
7
+ "run": "q36-final-q4_k_m-llamacpp-h200",
8
+ "model": "Q36",
9
+ "device": "H200",
10
+ "benchmarks": [
11
+ {
12
+ "benchmark": "ARC-Challenge",
13
+ "planned": 10,
14
+ "attempted": 10,
15
+ "completed": 8,
16
+ "scored": 8,
17
+ "passed": 8,
18
+ "ungraded": 2,
19
+ "not_run": 0,
20
+ "incomplete": 2
21
+ },
22
+ {
23
+ "benchmark": "BBH",
24
+ "planned": 23,
25
+ "attempted": 23,
26
+ "completed": 20,
27
+ "scored": 20,
28
+ "passed": 3,
29
+ "ungraded": 3,
30
+ "not_run": 0,
31
+ "incomplete": 3
32
+ },
33
+ {
34
+ "benchmark": "ChartQA",
35
+ "planned": 5,
36
+ "attempted": 5,
37
+ "completed": 5,
38
+ "scored": 5,
39
+ "passed": 4,
40
+ "ungraded": 0,
41
+ "not_run": 0,
42
+ "incomplete": 0
43
+ },
44
+ {
45
+ "benchmark": "GPQA-Diamond",
46
+ "planned": 5,
47
+ "attempted": 5,
48
+ "completed": 5,
49
+ "scored": 5,
50
+ "passed": 2,
51
+ "ungraded": 0,
52
+ "not_run": 0,
53
+ "incomplete": 0
54
+ },
55
+ {
56
+ "benchmark": "GSM8K",
57
+ "planned": 10,
58
+ "attempted": 10,
59
+ "completed": 9,
60
+ "scored": 9,
61
+ "passed": 5,
62
+ "ungraded": 1,
63
+ "not_run": 0,
64
+ "incomplete": 1
65
+ },
66
+ {
67
+ "benchmark": "HumanEval-Plus",
68
+ "planned": 5,
69
+ "attempted": 5,
70
+ "completed": 5,
71
+ "scored": 0,
72
+ "passed": 0,
73
+ "ungraded": 5,
74
+ "not_run": 0,
75
+ "incomplete": 0
76
+ },
77
+ {
78
+ "benchmark": "IFEval",
79
+ "planned": 10,
80
+ "attempted": 10,
81
+ "completed": 6,
82
+ "scored": 0,
83
+ "passed": 0,
84
+ "ungraded": 10,
85
+ "not_run": 0,
86
+ "incomplete": 4
87
+ },
88
+ {
89
+ "benchmark": "LiveCodeBench",
90
+ "planned": 9,
91
+ "attempted": 9,
92
+ "completed": 9,
93
+ "scored": 0,
94
+ "passed": 0,
95
+ "ungraded": 9,
96
+ "not_run": 0,
97
+ "incomplete": 0
98
+ },
99
+ {
100
+ "benchmark": "MATH-Level-5",
101
+ "planned": 10,
102
+ "attempted": 10,
103
+ "completed": 8,
104
+ "scored": 0,
105
+ "passed": 0,
106
+ "ungraded": 10,
107
+ "not_run": 0,
108
+ "incomplete": 2
109
+ },
110
+ {
111
+ "benchmark": "MBPP-Plus",
112
+ "planned": 5,
113
+ "attempted": 5,
114
+ "completed": 5,
115
+ "scored": 0,
116
+ "passed": 0,
117
+ "ungraded": 5,
118
+ "not_run": 0,
119
+ "incomplete": 0
120
+ },
121
+ {
122
+ "benchmark": "MMLU-Pro",
123
+ "planned": 1,
124
+ "attempted": 1,
125
+ "completed": 1,
126
+ "scored": 1,
127
+ "passed": 0,
128
+ "ungraded": 0,
129
+ "not_run": 0,
130
+ "incomplete": 0
131
+ },
132
+ {
133
+ "benchmark": "MMMU",
134
+ "planned": 6,
135
+ "attempted": 6,
136
+ "completed": 6,
137
+ "scored": 6,
138
+ "passed": 4,
139
+ "ungraded": 0,
140
+ "not_run": 0,
141
+ "incomplete": 0
142
+ },
143
+ {
144
+ "benchmark": "MathVista",
145
+ "planned": 5,
146
+ "attempted": 5,
147
+ "completed": 5,
148
+ "scored": 5,
149
+ "passed": 2,
150
+ "ungraded": 0,
151
+ "not_run": 0,
152
+ "incomplete": 0
153
+ },
154
+ {
155
+ "benchmark": "Q36-Agent-Function-Calling",
156
+ "planned": 5,
157
+ "attempted": 5,
158
+ "completed": 5,
159
+ "scored": 5,
160
+ "passed": 0,
161
+ "ungraded": 0,
162
+ "not_run": 0,
163
+ "incomplete": 0
164
+ },
165
+ {
166
+ "benchmark": "Q36-Answer-Termination-No-Looping",
167
+ "planned": 10,
168
+ "attempted": 10,
169
+ "completed": 10,
170
+ "scored": 10,
171
+ "passed": 7,
172
+ "ungraded": 0,
173
+ "not_run": 0,
174
+ "incomplete": 0
175
+ },
176
+ {
177
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
178
+ "planned": 20,
179
+ "attempted": 20,
180
+ "completed": 18,
181
+ "scored": 0,
182
+ "passed": 0,
183
+ "ungraded": 20,
184
+ "not_run": 0,
185
+ "incomplete": 2
186
+ },
187
+ {
188
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
189
+ "planned": 10,
190
+ "attempted": 10,
191
+ "completed": 10,
192
+ "scored": 10,
193
+ "passed": 10,
194
+ "ungraded": 0,
195
+ "not_run": 0,
196
+ "incomplete": 0
197
+ },
198
+ {
199
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
200
+ "planned": 10,
201
+ "attempted": 10,
202
+ "completed": 8,
203
+ "scored": 0,
204
+ "passed": 0,
205
+ "ungraded": 10,
206
+ "not_run": 0,
207
+ "incomplete": 2
208
+ },
209
+ {
210
+ "benchmark": "Q36-Direct-Style-and-Personality",
211
+ "planned": 10,
212
+ "attempted": 10,
213
+ "completed": 9,
214
+ "scored": 0,
215
+ "passed": 0,
216
+ "ungraded": 10,
217
+ "not_run": 0,
218
+ "incomplete": 1
219
+ },
220
+ {
221
+ "benchmark": "Q36-Hermes-Tool-Format",
222
+ "planned": 5,
223
+ "attempted": 5,
224
+ "completed": 5,
225
+ "scored": 5,
226
+ "passed": 5,
227
+ "ungraded": 0,
228
+ "not_run": 0,
229
+ "incomplete": 0
230
+ },
231
+ {
232
+ "benchmark": "Q36-JSON-Schema",
233
+ "planned": 5,
234
+ "attempted": 5,
235
+ "completed": 5,
236
+ "scored": 5,
237
+ "passed": 0,
238
+ "ungraded": 0,
239
+ "not_run": 0,
240
+ "incomplete": 0
241
+ },
242
+ {
243
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
244
+ "planned": 10,
245
+ "attempted": 10,
246
+ "completed": 10,
247
+ "scored": 0,
248
+ "passed": 0,
249
+ "ungraded": 10,
250
+ "not_run": 0,
251
+ "incomplete": 0
252
+ },
253
+ {
254
+ "benchmark": "Q36-Output-Integrity",
255
+ "planned": 5,
256
+ "attempted": 5,
257
+ "completed": 5,
258
+ "scored": 5,
259
+ "passed": 5,
260
+ "ungraded": 0,
261
+ "not_run": 0,
262
+ "incomplete": 0
263
+ },
264
+ {
265
+ "benchmark": "TruthfulQA",
266
+ "planned": 10,
267
+ "attempted": 10,
268
+ "completed": 9,
269
+ "scored": 9,
270
+ "passed": 6,
271
+ "ungraded": 1,
272
+ "not_run": 0,
273
+ "incomplete": 1
274
+ }
275
+ ]
276
+ },
277
+ {
278
+ "run": "q36-final-q4_k_m-llamacpp-rtx5090",
279
+ "model": "Q36",
280
+ "device": "RTX 5090",
281
+ "benchmarks": [
282
+ {
283
+ "benchmark": "ARC-Challenge",
284
+ "planned": 10,
285
+ "attempted": 10,
286
+ "completed": 8,
287
+ "scored": 8,
288
+ "passed": 8,
289
+ "ungraded": 2,
290
+ "not_run": 0,
291
+ "incomplete": 2
292
+ },
293
+ {
294
+ "benchmark": "BBH",
295
+ "planned": 23,
296
+ "attempted": 23,
297
+ "completed": 20,
298
+ "scored": 20,
299
+ "passed": 3,
300
+ "ungraded": 3,
301
+ "not_run": 0,
302
+ "incomplete": 3
303
+ },
304
+ {
305
+ "benchmark": "ChartQA",
306
+ "planned": 5,
307
+ "attempted": 5,
308
+ "completed": 5,
309
+ "scored": 5,
310
+ "passed": 4,
311
+ "ungraded": 0,
312
+ "not_run": 0,
313
+ "incomplete": 0
314
+ },
315
+ {
316
+ "benchmark": "GPQA-Diamond",
317
+ "planned": 5,
318
+ "attempted": 5,
319
+ "completed": 5,
320
+ "scored": 5,
321
+ "passed": 1,
322
+ "ungraded": 0,
323
+ "not_run": 0,
324
+ "incomplete": 0
325
+ },
326
+ {
327
+ "benchmark": "GSM8K",
328
+ "planned": 10,
329
+ "attempted": 10,
330
+ "completed": 10,
331
+ "scored": 10,
332
+ "passed": 7,
333
+ "ungraded": 0,
334
+ "not_run": 0,
335
+ "incomplete": 0
336
+ },
337
+ {
338
+ "benchmark": "HumanEval-Plus",
339
+ "planned": 5,
340
+ "attempted": 5,
341
+ "completed": 4,
342
+ "scored": 0,
343
+ "passed": 0,
344
+ "ungraded": 5,
345
+ "not_run": 0,
346
+ "incomplete": 1
347
+ },
348
+ {
349
+ "benchmark": "IFEval",
350
+ "planned": 10,
351
+ "attempted": 10,
352
+ "completed": 8,
353
+ "scored": 0,
354
+ "passed": 0,
355
+ "ungraded": 10,
356
+ "not_run": 0,
357
+ "incomplete": 2
358
+ },
359
+ {
360
+ "benchmark": "LiveCodeBench",
361
+ "planned": 9,
362
+ "attempted": 9,
363
+ "completed": 9,
364
+ "scored": 0,
365
+ "passed": 0,
366
+ "ungraded": 9,
367
+ "not_run": 0,
368
+ "incomplete": 0
369
+ },
370
+ {
371
+ "benchmark": "MATH-Level-5",
372
+ "planned": 10,
373
+ "attempted": 10,
374
+ "completed": 9,
375
+ "scored": 0,
376
+ "passed": 0,
377
+ "ungraded": 10,
378
+ "not_run": 0,
379
+ "incomplete": 1
380
+ },
381
+ {
382
+ "benchmark": "MBPP-Plus",
383
+ "planned": 5,
384
+ "attempted": 5,
385
+ "completed": 5,
386
+ "scored": 0,
387
+ "passed": 0,
388
+ "ungraded": 5,
389
+ "not_run": 0,
390
+ "incomplete": 0
391
+ },
392
+ {
393
+ "benchmark": "MMLU-Pro",
394
+ "planned": 1,
395
+ "attempted": 1,
396
+ "completed": 1,
397
+ "scored": 1,
398
+ "passed": 0,
399
+ "ungraded": 0,
400
+ "not_run": 0,
401
+ "incomplete": 0
402
+ },
403
+ {
404
+ "benchmark": "MMMU",
405
+ "planned": 6,
406
+ "attempted": 6,
407
+ "completed": 6,
408
+ "scored": 6,
409
+ "passed": 4,
410
+ "ungraded": 0,
411
+ "not_run": 0,
412
+ "incomplete": 0
413
+ },
414
+ {
415
+ "benchmark": "MathVista",
416
+ "planned": 5,
417
+ "attempted": 5,
418
+ "completed": 5,
419
+ "scored": 5,
420
+ "passed": 2,
421
+ "ungraded": 0,
422
+ "not_run": 0,
423
+ "incomplete": 0
424
+ },
425
+ {
426
+ "benchmark": "Q36-Agent-Function-Calling",
427
+ "planned": 5,
428
+ "attempted": 5,
429
+ "completed": 5,
430
+ "scored": 5,
431
+ "passed": 0,
432
+ "ungraded": 0,
433
+ "not_run": 0,
434
+ "incomplete": 0
435
+ },
436
+ {
437
+ "benchmark": "Q36-Answer-Termination-No-Looping",
438
+ "planned": 10,
439
+ "attempted": 10,
440
+ "completed": 10,
441
+ "scored": 10,
442
+ "passed": 9,
443
+ "ungraded": 0,
444
+ "not_run": 0,
445
+ "incomplete": 0
446
+ },
447
+ {
448
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
449
+ "planned": 20,
450
+ "attempted": 20,
451
+ "completed": 18,
452
+ "scored": 0,
453
+ "passed": 0,
454
+ "ungraded": 20,
455
+ "not_run": 0,
456
+ "incomplete": 2
457
+ },
458
+ {
459
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
460
+ "planned": 10,
461
+ "attempted": 10,
462
+ "completed": 10,
463
+ "scored": 10,
464
+ "passed": 10,
465
+ "ungraded": 0,
466
+ "not_run": 0,
467
+ "incomplete": 0
468
+ },
469
+ {
470
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
471
+ "planned": 10,
472
+ "attempted": 10,
473
+ "completed": 8,
474
+ "scored": 0,
475
+ "passed": 0,
476
+ "ungraded": 10,
477
+ "not_run": 0,
478
+ "incomplete": 2
479
+ },
480
+ {
481
+ "benchmark": "Q36-Direct-Style-and-Personality",
482
+ "planned": 10,
483
+ "attempted": 10,
484
+ "completed": 10,
485
+ "scored": 0,
486
+ "passed": 0,
487
+ "ungraded": 10,
488
+ "not_run": 0,
489
+ "incomplete": 0
490
+ },
491
+ {
492
+ "benchmark": "Q36-Hermes-Tool-Format",
493
+ "planned": 5,
494
+ "attempted": 5,
495
+ "completed": 5,
496
+ "scored": 5,
497
+ "passed": 3,
498
+ "ungraded": 0,
499
+ "not_run": 0,
500
+ "incomplete": 0
501
+ },
502
+ {
503
+ "benchmark": "Q36-JSON-Schema",
504
+ "planned": 5,
505
+ "attempted": 5,
506
+ "completed": 5,
507
+ "scored": 5,
508
+ "passed": 0,
509
+ "ungraded": 0,
510
+ "not_run": 0,
511
+ "incomplete": 0
512
+ },
513
+ {
514
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
515
+ "planned": 10,
516
+ "attempted": 10,
517
+ "completed": 10,
518
+ "scored": 0,
519
+ "passed": 0,
520
+ "ungraded": 10,
521
+ "not_run": 0,
522
+ "incomplete": 0
523
+ },
524
+ {
525
+ "benchmark": "Q36-Output-Integrity",
526
+ "planned": 5,
527
+ "attempted": 5,
528
+ "completed": 5,
529
+ "scored": 5,
530
+ "passed": 5,
531
+ "ungraded": 0,
532
+ "not_run": 0,
533
+ "incomplete": 0
534
+ },
535
+ {
536
+ "benchmark": "TruthfulQA",
537
+ "planned": 10,
538
+ "attempted": 10,
539
+ "completed": 9,
540
+ "scored": 9,
541
+ "passed": 6,
542
+ "ungraded": 1,
543
+ "not_run": 0,
544
+ "incomplete": 1
545
+ }
546
+ ]
547
+ },
548
+ {
549
+ "run": "huihui-original-q4_k_m-llamacpp-rtx5090",
550
+ "model": "Huihui",
551
+ "device": "RTX 5090",
552
+ "benchmarks": [
553
+ {
554
+ "benchmark": "ARC-Challenge",
555
+ "planned": 10,
556
+ "attempted": 10,
557
+ "completed": 10,
558
+ "scored": 10,
559
+ "passed": 10,
560
+ "ungraded": 0,
561
+ "not_run": 0,
562
+ "incomplete": 0
563
+ },
564
+ {
565
+ "benchmark": "BBH",
566
+ "planned": 23,
567
+ "attempted": 23,
568
+ "completed": 23,
569
+ "scored": 23,
570
+ "passed": 16,
571
+ "ungraded": 0,
572
+ "not_run": 0,
573
+ "incomplete": 0
574
+ },
575
+ {
576
+ "benchmark": "ChartQA",
577
+ "planned": 5,
578
+ "attempted": 5,
579
+ "completed": 5,
580
+ "scored": 5,
581
+ "passed": 5,
582
+ "ungraded": 0,
583
+ "not_run": 0,
584
+ "incomplete": 0
585
+ },
586
+ {
587
+ "benchmark": "GPQA-Diamond",
588
+ "planned": 5,
589
+ "attempted": 5,
590
+ "completed": 5,
591
+ "scored": 5,
592
+ "passed": 5,
593
+ "ungraded": 0,
594
+ "not_run": 0,
595
+ "incomplete": 0
596
+ },
597
+ {
598
+ "benchmark": "GSM8K",
599
+ "planned": 10,
600
+ "attempted": 10,
601
+ "completed": 10,
602
+ "scored": 10,
603
+ "passed": 7,
604
+ "ungraded": 0,
605
+ "not_run": 0,
606
+ "incomplete": 0
607
+ },
608
+ {
609
+ "benchmark": "HumanEval-Plus",
610
+ "planned": 5,
611
+ "attempted": 5,
612
+ "completed": 5,
613
+ "scored": 0,
614
+ "passed": 0,
615
+ "ungraded": 5,
616
+ "not_run": 0,
617
+ "incomplete": 0
618
+ },
619
+ {
620
+ "benchmark": "IFEval",
621
+ "planned": 10,
622
+ "attempted": 10,
623
+ "completed": 10,
624
+ "scored": 0,
625
+ "passed": 0,
626
+ "ungraded": 10,
627
+ "not_run": 0,
628
+ "incomplete": 0
629
+ },
630
+ {
631
+ "benchmark": "LiveCodeBench",
632
+ "planned": 9,
633
+ "attempted": 9,
634
+ "completed": 4,
635
+ "scored": 0,
636
+ "passed": 0,
637
+ "ungraded": 9,
638
+ "not_run": 0,
639
+ "incomplete": 5
640
+ },
641
+ {
642
+ "benchmark": "MATH-Level-5",
643
+ "planned": 10,
644
+ "attempted": 10,
645
+ "completed": 8,
646
+ "scored": 0,
647
+ "passed": 0,
648
+ "ungraded": 10,
649
+ "not_run": 0,
650
+ "incomplete": 2
651
+ },
652
+ {
653
+ "benchmark": "MBPP-Plus",
654
+ "planned": 5,
655
+ "attempted": 5,
656
+ "completed": 5,
657
+ "scored": 0,
658
+ "passed": 0,
659
+ "ungraded": 5,
660
+ "not_run": 0,
661
+ "incomplete": 0
662
+ },
663
+ {
664
+ "benchmark": "MMLU-Pro",
665
+ "planned": 1,
666
+ "attempted": 1,
667
+ "completed": 1,
668
+ "scored": 1,
669
+ "passed": 0,
670
+ "ungraded": 0,
671
+ "not_run": 0,
672
+ "incomplete": 0
673
+ },
674
+ {
675
+ "benchmark": "MMMU",
676
+ "planned": 6,
677
+ "attempted": 6,
678
+ "completed": 6,
679
+ "scored": 6,
680
+ "passed": 4,
681
+ "ungraded": 0,
682
+ "not_run": 0,
683
+ "incomplete": 0
684
+ },
685
+ {
686
+ "benchmark": "MathVista",
687
+ "planned": 5,
688
+ "attempted": 5,
689
+ "completed": 5,
690
+ "scored": 5,
691
+ "passed": 3,
692
+ "ungraded": 0,
693
+ "not_run": 0,
694
+ "incomplete": 0
695
+ },
696
+ {
697
+ "benchmark": "Q36-Agent-Function-Calling",
698
+ "planned": 5,
699
+ "attempted": 5,
700
+ "completed": 5,
701
+ "scored": 5,
702
+ "passed": 0,
703
+ "ungraded": 0,
704
+ "not_run": 0,
705
+ "incomplete": 0
706
+ },
707
+ {
708
+ "benchmark": "Q36-Answer-Termination-No-Looping",
709
+ "planned": 10,
710
+ "attempted": 10,
711
+ "completed": 10,
712
+ "scored": 10,
713
+ "passed": 10,
714
+ "ungraded": 0,
715
+ "not_run": 0,
716
+ "incomplete": 0
717
+ },
718
+ {
719
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
720
+ "planned": 20,
721
+ "attempted": 20,
722
+ "completed": 20,
723
+ "scored": 0,
724
+ "passed": 0,
725
+ "ungraded": 20,
726
+ "not_run": 0,
727
+ "incomplete": 0
728
+ },
729
+ {
730
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
731
+ "planned": 10,
732
+ "attempted": 10,
733
+ "completed": 10,
734
+ "scored": 10,
735
+ "passed": 10,
736
+ "ungraded": 0,
737
+ "not_run": 0,
738
+ "incomplete": 0
739
+ },
740
+ {
741
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
742
+ "planned": 10,
743
+ "attempted": 10,
744
+ "completed": 10,
745
+ "scored": 0,
746
+ "passed": 0,
747
+ "ungraded": 10,
748
+ "not_run": 0,
749
+ "incomplete": 0
750
+ },
751
+ {
752
+ "benchmark": "Q36-Direct-Style-and-Personality",
753
+ "planned": 10,
754
+ "attempted": 10,
755
+ "completed": 10,
756
+ "scored": 0,
757
+ "passed": 0,
758
+ "ungraded": 10,
759
+ "not_run": 0,
760
+ "incomplete": 0
761
+ },
762
+ {
763
+ "benchmark": "Q36-Hermes-Tool-Format",
764
+ "planned": 5,
765
+ "attempted": 5,
766
+ "completed": 5,
767
+ "scored": 5,
768
+ "passed": 4,
769
+ "ungraded": 0,
770
+ "not_run": 0,
771
+ "incomplete": 0
772
+ },
773
+ {
774
+ "benchmark": "Q36-JSON-Schema",
775
+ "planned": 5,
776
+ "attempted": 5,
777
+ "completed": 5,
778
+ "scored": 5,
779
+ "passed": 2,
780
+ "ungraded": 0,
781
+ "not_run": 0,
782
+ "incomplete": 0
783
+ },
784
+ {
785
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
786
+ "planned": 10,
787
+ "attempted": 10,
788
+ "completed": 10,
789
+ "scored": 0,
790
+ "passed": 0,
791
+ "ungraded": 10,
792
+ "not_run": 0,
793
+ "incomplete": 0
794
+ },
795
+ {
796
+ "benchmark": "Q36-Output-Integrity",
797
+ "planned": 5,
798
+ "attempted": 5,
799
+ "completed": 5,
800
+ "scored": 5,
801
+ "passed": 5,
802
+ "ungraded": 0,
803
+ "not_run": 0,
804
+ "incomplete": 0
805
+ },
806
+ {
807
+ "benchmark": "TruthfulQA",
808
+ "planned": 10,
809
+ "attempted": 10,
810
+ "completed": 9,
811
+ "scored": 9,
812
+ "passed": 6,
813
+ "ungraded": 1,
814
+ "not_run": 0,
815
+ "incomplete": 1
816
+ }
817
+ ]
818
+ },
819
+ {
820
+ "run": "q36-final-q4_k_m-llamacpp-rtx2000ada-mixed24",
821
+ "model": "Q36",
822
+ "device": "RTX 2000 Ada (CPU+GPU)",
823
+ "benchmarks": [
824
+ {
825
+ "benchmark": "ARC-Challenge",
826
+ "planned": 10,
827
+ "attempted": 10,
828
+ "completed": 7,
829
+ "scored": 7,
830
+ "passed": 7,
831
+ "ungraded": 3,
832
+ "not_run": 0,
833
+ "incomplete": 3
834
+ },
835
+ {
836
+ "benchmark": "BBH",
837
+ "planned": 23,
838
+ "attempted": 23,
839
+ "completed": 20,
840
+ "scored": 20,
841
+ "passed": 3,
842
+ "ungraded": 3,
843
+ "not_run": 0,
844
+ "incomplete": 3
845
+ },
846
+ {
847
+ "benchmark": "ChartQA",
848
+ "planned": 5,
849
+ "attempted": 5,
850
+ "completed": 5,
851
+ "scored": 5,
852
+ "passed": 4,
853
+ "ungraded": 0,
854
+ "not_run": 0,
855
+ "incomplete": 0
856
+ },
857
+ {
858
+ "benchmark": "GPQA-Diamond",
859
+ "planned": 5,
860
+ "attempted": 5,
861
+ "completed": 5,
862
+ "scored": 5,
863
+ "passed": 2,
864
+ "ungraded": 0,
865
+ "not_run": 0,
866
+ "incomplete": 0
867
+ },
868
+ {
869
+ "benchmark": "GSM8K",
870
+ "planned": 10,
871
+ "attempted": 10,
872
+ "completed": 10,
873
+ "scored": 10,
874
+ "passed": 8,
875
+ "ungraded": 0,
876
+ "not_run": 0,
877
+ "incomplete": 0
878
+ },
879
+ {
880
+ "benchmark": "HumanEval-Plus",
881
+ "planned": 5,
882
+ "attempted": 5,
883
+ "completed": 5,
884
+ "scored": 0,
885
+ "passed": 0,
886
+ "ungraded": 5,
887
+ "not_run": 0,
888
+ "incomplete": 0
889
+ },
890
+ {
891
+ "benchmark": "IFEval",
892
+ "planned": 10,
893
+ "attempted": 10,
894
+ "completed": 9,
895
+ "scored": 0,
896
+ "passed": 0,
897
+ "ungraded": 10,
898
+ "not_run": 0,
899
+ "incomplete": 1
900
+ },
901
+ {
902
+ "benchmark": "LiveCodeBench",
903
+ "planned": 9,
904
+ "attempted": 9,
905
+ "completed": 9,
906
+ "scored": 0,
907
+ "passed": 0,
908
+ "ungraded": 9,
909
+ "not_run": 0,
910
+ "incomplete": 0
911
+ },
912
+ {
913
+ "benchmark": "MATH-Level-5",
914
+ "planned": 10,
915
+ "attempted": 10,
916
+ "completed": 9,
917
+ "scored": 0,
918
+ "passed": 0,
919
+ "ungraded": 10,
920
+ "not_run": 0,
921
+ "incomplete": 1
922
+ },
923
+ {
924
+ "benchmark": "MBPP-Plus",
925
+ "planned": 5,
926
+ "attempted": 5,
927
+ "completed": 5,
928
+ "scored": 0,
929
+ "passed": 0,
930
+ "ungraded": 5,
931
+ "not_run": 0,
932
+ "incomplete": 0
933
+ },
934
+ {
935
+ "benchmark": "MMLU-Pro",
936
+ "planned": 1,
937
+ "attempted": 1,
938
+ "completed": 1,
939
+ "scored": 1,
940
+ "passed": 0,
941
+ "ungraded": 0,
942
+ "not_run": 0,
943
+ "incomplete": 0
944
+ },
945
+ {
946
+ "benchmark": "MMMU",
947
+ "planned": 6,
948
+ "attempted": 6,
949
+ "completed": 6,
950
+ "scored": 6,
951
+ "passed": 1,
952
+ "ungraded": 0,
953
+ "not_run": 0,
954
+ "incomplete": 0
955
+ },
956
+ {
957
+ "benchmark": "MathVista",
958
+ "planned": 5,
959
+ "attempted": 5,
960
+ "completed": 5,
961
+ "scored": 5,
962
+ "passed": 3,
963
+ "ungraded": 0,
964
+ "not_run": 0,
965
+ "incomplete": 0
966
+ },
967
+ {
968
+ "benchmark": "Q36-Agent-Function-Calling",
969
+ "planned": 5,
970
+ "attempted": 5,
971
+ "completed": 5,
972
+ "scored": 5,
973
+ "passed": 0,
974
+ "ungraded": 0,
975
+ "not_run": 0,
976
+ "incomplete": 0
977
+ },
978
+ {
979
+ "benchmark": "Q36-Answer-Termination-No-Looping",
980
+ "planned": 10,
981
+ "attempted": 10,
982
+ "completed": 10,
983
+ "scored": 10,
984
+ "passed": 8,
985
+ "ungraded": 0,
986
+ "not_run": 0,
987
+ "incomplete": 0
988
+ },
989
+ {
990
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
991
+ "planned": 20,
992
+ "attempted": 20,
993
+ "completed": 19,
994
+ "scored": 0,
995
+ "passed": 0,
996
+ "ungraded": 20,
997
+ "not_run": 0,
998
+ "incomplete": 1
999
+ },
1000
+ {
1001
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
1002
+ "planned": 10,
1003
+ "attempted": 10,
1004
+ "completed": 10,
1005
+ "scored": 10,
1006
+ "passed": 9,
1007
+ "ungraded": 0,
1008
+ "not_run": 0,
1009
+ "incomplete": 0
1010
+ },
1011
+ {
1012
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
1013
+ "planned": 10,
1014
+ "attempted": 10,
1015
+ "completed": 9,
1016
+ "scored": 0,
1017
+ "passed": 0,
1018
+ "ungraded": 10,
1019
+ "not_run": 0,
1020
+ "incomplete": 1
1021
+ },
1022
+ {
1023
+ "benchmark": "Q36-Direct-Style-and-Personality",
1024
+ "planned": 10,
1025
+ "attempted": 10,
1026
+ "completed": 10,
1027
+ "scored": 0,
1028
+ "passed": 0,
1029
+ "ungraded": 10,
1030
+ "not_run": 0,
1031
+ "incomplete": 0
1032
+ },
1033
+ {
1034
+ "benchmark": "Q36-Hermes-Tool-Format",
1035
+ "planned": 5,
1036
+ "attempted": 5,
1037
+ "completed": 5,
1038
+ "scored": 5,
1039
+ "passed": 5,
1040
+ "ungraded": 0,
1041
+ "not_run": 0,
1042
+ "incomplete": 0
1043
+ },
1044
+ {
1045
+ "benchmark": "Q36-JSON-Schema",
1046
+ "planned": 5,
1047
+ "attempted": 5,
1048
+ "completed": 5,
1049
+ "scored": 5,
1050
+ "passed": 0,
1051
+ "ungraded": 0,
1052
+ "not_run": 0,
1053
+ "incomplete": 0
1054
+ },
1055
+ {
1056
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
1057
+ "planned": 10,
1058
+ "attempted": 10,
1059
+ "completed": 10,
1060
+ "scored": 0,
1061
+ "passed": 0,
1062
+ "ungraded": 10,
1063
+ "not_run": 0,
1064
+ "incomplete": 0
1065
+ },
1066
+ {
1067
+ "benchmark": "Q36-Output-Integrity",
1068
+ "planned": 5,
1069
+ "attempted": 5,
1070
+ "completed": 5,
1071
+ "scored": 5,
1072
+ "passed": 5,
1073
+ "ungraded": 0,
1074
+ "not_run": 0,
1075
+ "incomplete": 0
1076
+ },
1077
+ {
1078
+ "benchmark": "TruthfulQA",
1079
+ "planned": 10,
1080
+ "attempted": 10,
1081
+ "completed": 10,
1082
+ "scored": 10,
1083
+ "passed": 6,
1084
+ "ungraded": 0,
1085
+ "not_run": 0,
1086
+ "incomplete": 0
1087
+ }
1088
+ ]
1089
+ },
1090
+ {
1091
+ "run": "huihui-original-q4_k_m-llamacpp-rtx2000ada-mixed24",
1092
+ "model": "Huihui",
1093
+ "device": "RTX 2000 Ada (CPU+GPU)",
1094
+ "benchmarks": [
1095
+ {
1096
+ "benchmark": "ARC-Challenge",
1097
+ "planned": 10,
1098
+ "attempted": 10,
1099
+ "completed": 10,
1100
+ "scored": 10,
1101
+ "passed": 10,
1102
+ "ungraded": 0,
1103
+ "not_run": 0,
1104
+ "incomplete": 0
1105
+ },
1106
+ {
1107
+ "benchmark": "BBH",
1108
+ "planned": 23,
1109
+ "attempted": 11,
1110
+ "completed": 10,
1111
+ "scored": 10,
1112
+ "passed": 6,
1113
+ "ungraded": 1,
1114
+ "not_run": 12,
1115
+ "incomplete": 1
1116
+ },
1117
+ {
1118
+ "benchmark": "ChartQA",
1119
+ "planned": 5,
1120
+ "attempted": 5,
1121
+ "completed": 5,
1122
+ "scored": 5,
1123
+ "passed": 5,
1124
+ "ungraded": 0,
1125
+ "not_run": 0,
1126
+ "incomplete": 0
1127
+ },
1128
+ {
1129
+ "benchmark": "GPQA-Diamond",
1130
+ "planned": 5,
1131
+ "attempted": 5,
1132
+ "completed": 3,
1133
+ "scored": 3,
1134
+ "passed": 3,
1135
+ "ungraded": 2,
1136
+ "not_run": 0,
1137
+ "incomplete": 2
1138
+ },
1139
+ {
1140
+ "benchmark": "GSM8K",
1141
+ "planned": 10,
1142
+ "attempted": 10,
1143
+ "completed": 10,
1144
+ "scored": 10,
1145
+ "passed": 7,
1146
+ "ungraded": 0,
1147
+ "not_run": 0,
1148
+ "incomplete": 0
1149
+ },
1150
+ {
1151
+ "benchmark": "HumanEval-Plus",
1152
+ "planned": 5,
1153
+ "attempted": 5,
1154
+ "completed": 5,
1155
+ "scored": 0,
1156
+ "passed": 0,
1157
+ "ungraded": 5,
1158
+ "not_run": 0,
1159
+ "incomplete": 0
1160
+ },
1161
+ {
1162
+ "benchmark": "IFEval",
1163
+ "planned": 10,
1164
+ "attempted": 10,
1165
+ "completed": 10,
1166
+ "scored": 0,
1167
+ "passed": 0,
1168
+ "ungraded": 10,
1169
+ "not_run": 0,
1170
+ "incomplete": 0
1171
+ },
1172
+ {
1173
+ "benchmark": "LiveCodeBench",
1174
+ "planned": 9,
1175
+ "attempted": 9,
1176
+ "completed": 5,
1177
+ "scored": 0,
1178
+ "passed": 0,
1179
+ "ungraded": 9,
1180
+ "not_run": 0,
1181
+ "incomplete": 4
1182
+ },
1183
+ {
1184
+ "benchmark": "MATH-Level-5",
1185
+ "planned": 10,
1186
+ "attempted": 10,
1187
+ "completed": 5,
1188
+ "scored": 0,
1189
+ "passed": 0,
1190
+ "ungraded": 10,
1191
+ "not_run": 0,
1192
+ "incomplete": 5
1193
+ },
1194
+ {
1195
+ "benchmark": "MBPP-Plus",
1196
+ "planned": 5,
1197
+ "attempted": 5,
1198
+ "completed": 5,
1199
+ "scored": 0,
1200
+ "passed": 0,
1201
+ "ungraded": 5,
1202
+ "not_run": 0,
1203
+ "incomplete": 0
1204
+ },
1205
+ {
1206
+ "benchmark": "MMLU-Pro",
1207
+ "planned": 1,
1208
+ "attempted": 1,
1209
+ "completed": 0,
1210
+ "scored": 0,
1211
+ "passed": 0,
1212
+ "ungraded": 1,
1213
+ "not_run": 0,
1214
+ "incomplete": 1
1215
+ },
1216
+ {
1217
+ "benchmark": "MMMU",
1218
+ "planned": 6,
1219
+ "attempted": 6,
1220
+ "completed": 6,
1221
+ "scored": 6,
1222
+ "passed": 4,
1223
+ "ungraded": 0,
1224
+ "not_run": 0,
1225
+ "incomplete": 0
1226
+ },
1227
+ {
1228
+ "benchmark": "MathVista",
1229
+ "planned": 5,
1230
+ "attempted": 5,
1231
+ "completed": 5,
1232
+ "scored": 5,
1233
+ "passed": 3,
1234
+ "ungraded": 0,
1235
+ "not_run": 0,
1236
+ "incomplete": 0
1237
+ },
1238
+ {
1239
+ "benchmark": "Q36-Agent-Function-Calling",
1240
+ "planned": 5,
1241
+ "attempted": 5,
1242
+ "completed": 5,
1243
+ "scored": 5,
1244
+ "passed": 0,
1245
+ "ungraded": 0,
1246
+ "not_run": 0,
1247
+ "incomplete": 0
1248
+ },
1249
+ {
1250
+ "benchmark": "Q36-Answer-Termination-No-Looping",
1251
+ "planned": 10,
1252
+ "attempted": 10,
1253
+ "completed": 10,
1254
+ "scored": 10,
1255
+ "passed": 10,
1256
+ "ungraded": 0,
1257
+ "not_run": 0,
1258
+ "incomplete": 0
1259
+ },
1260
+ {
1261
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
1262
+ "planned": 20,
1263
+ "attempted": 10,
1264
+ "completed": 10,
1265
+ "scored": 0,
1266
+ "passed": 0,
1267
+ "ungraded": 10,
1268
+ "not_run": 10,
1269
+ "incomplete": 0
1270
+ },
1271
+ {
1272
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
1273
+ "planned": 10,
1274
+ "attempted": 10,
1275
+ "completed": 10,
1276
+ "scored": 10,
1277
+ "passed": 10,
1278
+ "ungraded": 0,
1279
+ "not_run": 0,
1280
+ "incomplete": 0
1281
+ },
1282
+ {
1283
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
1284
+ "planned": 10,
1285
+ "attempted": 10,
1286
+ "completed": 10,
1287
+ "scored": 0,
1288
+ "passed": 0,
1289
+ "ungraded": 10,
1290
+ "not_run": 0,
1291
+ "incomplete": 0
1292
+ },
1293
+ {
1294
+ "benchmark": "Q36-Direct-Style-and-Personality",
1295
+ "planned": 10,
1296
+ "attempted": 10,
1297
+ "completed": 10,
1298
+ "scored": 0,
1299
+ "passed": 0,
1300
+ "ungraded": 10,
1301
+ "not_run": 0,
1302
+ "incomplete": 0
1303
+ },
1304
+ {
1305
+ "benchmark": "Q36-Hermes-Tool-Format",
1306
+ "planned": 5,
1307
+ "attempted": 5,
1308
+ "completed": 5,
1309
+ "scored": 5,
1310
+ "passed": 3,
1311
+ "ungraded": 0,
1312
+ "not_run": 0,
1313
+ "incomplete": 0
1314
+ },
1315
+ {
1316
+ "benchmark": "Q36-JSON-Schema",
1317
+ "planned": 5,
1318
+ "attempted": 5,
1319
+ "completed": 5,
1320
+ "scored": 5,
1321
+ "passed": 3,
1322
+ "ungraded": 0,
1323
+ "not_run": 0,
1324
+ "incomplete": 0
1325
+ },
1326
+ {
1327
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
1328
+ "planned": 10,
1329
+ "attempted": 10,
1330
+ "completed": 10,
1331
+ "scored": 0,
1332
+ "passed": 0,
1333
+ "ungraded": 10,
1334
+ "not_run": 0,
1335
+ "incomplete": 0
1336
+ },
1337
+ {
1338
+ "benchmark": "Q36-Output-Integrity",
1339
+ "planned": 5,
1340
+ "attempted": 5,
1341
+ "completed": 5,
1342
+ "scored": 5,
1343
+ "passed": 5,
1344
+ "ungraded": 0,
1345
+ "not_run": 0,
1346
+ "incomplete": 0
1347
+ },
1348
+ {
1349
+ "benchmark": "TruthfulQA",
1350
+ "planned": 10,
1351
+ "attempted": 10,
1352
+ "completed": 10,
1353
+ "scored": 10,
1354
+ "passed": 7,
1355
+ "ungraded": 0,
1356
+ "not_run": 0,
1357
+ "incomplete": 0
1358
+ }
1359
+ ]
1360
+ }
1361
+ ]
1362
+ }
LAPTOP-CAPABILITY-PLAN.md ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Laptop assistant: from quick tests to useful workflows
2
+
3
+ **Status: local preparation in progress; no laptop winner selected yet.** This is a deployment roadmap, not a claim that the full integration already works.
4
+
5
+ ## πŸ’» Coding and agent skills β€” what we will actually test
6
+
7
+ | Track | Cases | Success evidence | Current status |
8
+ |---|---:|---|---|
9
+ | HumanEval+ | 5 | Generated functions pass the selected task's base and extended tests in an isolated container | Existing outputs; execution/grading must be reconciled |
10
+ | MBPP+ | 5 | Correct results on base and extended tests, including edge cases | Existing outputs; execution/grading must be reconciled |
11
+ | LiveCodeBench | 9 | Program passes task tests within sandbox limits | Existing outputs; execution/grading must be reconciled |
12
+ | Q36 bug fixes | 5 | Reproduce a failing test, apply a patch, pass regression tests | New tasks to prepare |
13
+ | Q36 small app / website tasks | 5 | Build succeeds; required UI behavior passes browser checks | New tasks to prepare |
14
+ | Tool choice and arguments | 5 | Correct tool and schema-valid arguments; no invented tool result | Local smoke fixtures prepared separately |
15
+ | Multi-step planning | 5 | Respect dependencies and reach a verifiable final state | New interactive fixtures to prepare |
16
+ | Recovery from tool errors | 5 | Respond to a simulated failure, choose a valid alternative, verify recovery | New interactive fixtures to prepare |
17
+ | Memory and conflicting evidence | 5 | Retrieve the correct note, cite it, distinguish updated from stale facts | New isolated test vault to prepare |
18
+ | Stopping and task completion | 5 | Stop after success; report missing prerequisites instead of inventing completion | New interactive fixtures to prepare |
19
+ | Transfer to changed constraints | 5 | Solve held-out variants after a changed rule or environment | New tasks to prepare |
20
+
21
+ **64 case slots in this extension**, including 19 existing coding slots. These are compact diagnostics, not full leaderboard scores. The agent tasks measure useful components of general problem solving, **not proof of AGI**. Existing Hermes-format and JSON-schema diagnostics remain in the broad suite and are not silently counted twice.
22
+
23
+ ### How a capability earns a card entry
24
+
25
+ Freeze tasks and evaluators before comparing runtime settings. Count one primary attempt per case. Report **passes / executed-and-graded cases**, task coverage, failures and ungraded cases together. Publish results even when they are weak; only describe a workflow as demonstrated when its actual end-to-end task passes. A successful JSON tool-call test alone does not establish autonomous coding or desktop control.
26
+
27
+ Generated code runs in a disposable container without network access, host credentials or user-folder mounts, with CPU, RAM, process and time limits. Host tools and paid services are not available to benchmark prompts.
28
+
29
+ ## ⚑ Pick a laptop runtime
30
+
31
+ 1. Smoke-test **IQ4_XS, Q3_K_M, IQ2_M and IQ1_M**; exclude the BF16 server package. Start with IQ2_M to check loading and the harness.
32
+ 2. Compare **Ollama and llama.cpp**, each with a documented baseline and an optimized profile: 16 primary configurations. FreeToken is an additional experimental backend only if a compatible build actually runs.
33
+ 3. Use the same prompts, image inputs, context, output cap and thinking setting. Tune only on a separate warm-up set, then freeze settings.
34
+ 4. Record answer quality separately from prefill/decode tokens/s, end-to-end speed, time to first visible output, CPU/RAM/VRAM, temperatures, timeouts and incomplete outputs.
35
+ 5. Select a measured quality/performance/memory compromise, not simply the fastest token rate. An OOM or unsupported backend is a technical failure, not a wrong answer.
36
+
37
+ The target laptop has **32 GB RAM and an RTX 4060 Laptop GPU with 8 GB VRAM**. These 35B-weight models need CPU/RAM offload. MoE reduces active computation, not the need to store the expert weights.
38
+
39
+ ## πŸ› οΈ Hermes, routing and personal memory
40
+
41
+ After selecting the runtime, test it as a local endpoint behind the existing Hermes/router installation. Keep the previous route available for rollback. First use a dedicated test workspace and synthetic notes; connect the real Obsidian vault only after the test flow passes.
42
+
43
+ Useful scenarios to validate:
44
+
45
+ - **Code companion:** inspect a sample repository, fix one bug, run tests and summarize the patch.
46
+ - **Project memory:** find a note, cite its source and use its latest decision in a response.
47
+ - **Research notebook:** turn supplied material into a draft note with provenance, then ask before writing into the real vault.
48
+ - **Task recovery:** recognize an unavailable tool, ask for what is missing or use an explicitly allowed alternative.
49
+ - **Private local assistant:** use the configured local route and verify that external fallback is disabled for private inputs.
50
+
51
+ Obsidian storage, retrieval/indexing, memory updates and routing belong to the host application. They are not built into the model weights. Exact endpoint instructions and working configuration examples will be added after local validation; no production routing has been changed by this document.
README.md CHANGED
@@ -12,7 +12,7 @@ tags:
12
  - abliterated
13
  - heretic
14
  - uncensor
15
- - hermes
16
  - mtp
17
  - coding
18
  - tool-calling
@@ -46,6 +46,12 @@ Meet your local AI companion for ideas, code, images and conversation. **Qwen3.6
46
 
47
  These are things to try, not promises of perfect results. It can make mistakes or repeat itself; check important answers. Adult-oriented training is included (**18+**).
48
 
 
 
 
 
 
 
49
  ## πŸ•ΆοΈ Street knowledge. Business mind. Your style.
50
 
51
  The personality direction is **direct, sharp-witted and business-minded**: a street-smart conversation partner with humor, creative confidence and room for disagreement. The training mix includes internet culture, slang, practical business topics and personality-oriented conversations. Think less formal textbook, more an opinionated partner for brainstorming, writing and exploring alternatives.
@@ -289,6 +295,219 @@ Huihui performed better on the shared automatically assessable RTX 5090 subset;
289
 
290
  </details>
291
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
292
  ## πŸ“œ License, lineage & credits
293
 
294
  [Apache-2.0 license](LICENSE). Dataset sources and their own license information are linked in [DATASETS.md](DATASETS.md); model licensing does not relicense the source datasets. Thanks to the Qwen team, the inherited model and dataset authors, and the Soup, PEFT, Transformers, llama.cpp and Ollama projects.
@@ -297,6 +516,27 @@ Huihui performed better on the shared automatically assessable RTX 5090 subset;
297
 
298
  These names do not imply affiliation or a promise of unrestricted behavior. Full provenance and validation are retained, including the historical Opus4.7-labelled source. No universal superiority, guaranteed compliance or removal of memorization is claimed.
299
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
300
  ## πŸ”— Related collection
301
 
302
  The previous release remains separate: [Qwen3.6 Opus4.7 Heretic Hermes Agent β€” Editions](https://huggingface.co/collections/oktayd/qwen36-opus47-heretic-hermes-agent-editions-6a8d09f6c2eb42ed1b112184).
 
12
  - abliterated
13
  - heretic
14
  - uncensor
15
+ - hermes-lineage
16
  - mtp
17
  - coding
18
  - tool-calling
 
46
 
47
  These are things to try, not promises of perfect results. It can make mistakes or repeat itself; check important answers. Adult-oriented training is included (**18+**).
48
 
49
+ ## 🧠 A team of experts inside one model
50
+
51
+ **MoE means Mixture of Experts.** Think of a team of specialists: for each token, a router chooses which expert networks should contribute. This model has about **35 billion parameters in total**, with roughly **3 billion active per token**; its configuration selects **8 of 256 routed experts**.
52
+
53
+ That saves computation compared with activating every expert at once. It does **not** turn a 35B download into a 3B-sized model: the expert weights still need disk space and accessible RAM/VRAM. Quantized editions make local use more practical. The experts are learned networks, not separate installed apps or named profession-specific agents.
54
+
55
  ## πŸ•ΆοΈ Street knowledge. Business mind. Your style.
56
 
57
  The personality direction is **direct, sharp-witted and business-minded**: a street-smart conversation partner with humor, creative confidence and room for disagreement. The training mix includes internet culture, slang, practical business topics and personality-oriented conversations. Think less formal textbook, more an opinionated partner for brainstorming, writing and exploring alternatives.
 
295
 
296
  </details>
297
 
298
+ <!-- q36-capability-expansion:start -->
299
+ ## πŸ”Ž Broad benchmark β€” tests and passes
300
+
301
+ Expand a device to inspect every test family. **Passes / graded** use the original strict evaluator, including format-sensitive checks. Ungraded answers are not failures or passes. Incomplete is a separate delivery flag and can overlap with ungraded. These are small local subsets, not official leaderboard scores.
302
+
303
+ <details>
304
+ <summary>Q36 Β· H200 β€” per-test breakdown</summary>
305
+
306
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
307
+ |---|---:|---:|---:|---:|---:|
308
+ | ARC-Challenge | 10 / 10 | 8 / 8 | 2 | 2 | 0 |
309
+ | BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
310
+ | ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
311
+ | GPQA-Diamond | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
312
+ | GSM8K | 10 / 10 | 5 / 9 | 1 | 1 | 0 |
313
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
314
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 4 | 0 |
315
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
316
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
317
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
318
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
319
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
320
+ | MathVista | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
321
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
322
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
323
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 2 | 0 |
324
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
325
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
326
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
327
+ | Q36-Hermes-Tool-Format | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
328
+ | Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
329
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
330
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
331
+ | TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
332
+
333
+ </details>
334
+
335
+ <details>
336
+ <summary>Q36 Β· RTX 5090 β€” per-test breakdown</summary>
337
+
338
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
339
+ |---|---:|---:|---:|---:|---:|
340
+ | ARC-Challenge | 10 / 10 | 8 / 8 | 2 | 2 | 0 |
341
+ | BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
342
+ | ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
343
+ | GPQA-Diamond | 5 / 5 | 1 / 5 | 0 | 0 | 0 |
344
+ | GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
345
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 1 | 0 |
346
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
347
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
348
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
349
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
350
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
351
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
352
+ | MathVista | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
353
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
354
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 9 / 10 | 0 | 0 | 0 |
355
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 2 | 0 |
356
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
357
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
358
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
359
+ | Q36-Hermes-Tool-Format | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
360
+ | Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
361
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
362
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
363
+ | TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
364
+
365
+ </details>
366
+
367
+ <details>
368
+ <summary>Huihui Β· RTX 5090 β€” per-test breakdown</summary>
369
+
370
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
371
+ |---|---:|---:|---:|---:|---:|
372
+ | ARC-Challenge | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
373
+ | BBH | 23 / 23 | 16 / 23 | 0 | 0 | 0 |
374
+ | ChartQA | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
375
+ | GPQA-Diamond | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
376
+ | GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
377
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
378
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
379
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 5 | 0 |
380
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
381
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
382
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
383
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
384
+ | MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
385
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
386
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
387
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 0 | 0 |
388
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
389
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
390
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
391
+ | Q36-Hermes-Tool-Format | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
392
+ | Q36-JSON-Schema | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
393
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
394
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
395
+ | TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
396
+
397
+ </details>
398
+
399
+ <details>
400
+ <summary>Q36 Β· RTX 2000 Ada (CPU+GPU) β€” per-test breakdown</summary>
401
+
402
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
403
+ |---|---:|---:|---:|---:|---:|
404
+ | ARC-Challenge | 10 / 10 | 7 / 7 | 3 | 3 | 0 |
405
+ | BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
406
+ | ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
407
+ | GPQA-Diamond | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
408
+ | GSM8K | 10 / 10 | 8 / 10 | 0 | 0 | 0 |
409
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
410
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
411
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
412
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
413
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
414
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
415
+ | MMMU | 6 / 6 | 1 / 6 | 0 | 0 | 0 |
416
+ | MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
417
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
418
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 8 / 10 | 0 | 0 | 0 |
419
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 1 | 0 |
420
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 9 / 10 | 0 | 0 | 0 |
421
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
422
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
423
+ | Q36-Hermes-Tool-Format | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
424
+ | Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
425
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
426
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
427
+ | TruthfulQA | 10 / 10 | 6 / 10 | 0 | 0 | 0 |
428
+
429
+ </details>
430
+
431
+ <details>
432
+ <summary>Huihui Β· RTX 2000 Ada (CPU+GPU) β€” per-test breakdown</summary>
433
+
434
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
435
+ |---|---:|---:|---:|---:|---:|
436
+ | ARC-Challenge | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
437
+ | BBH | 11 / 23 | 6 / 10 | 1 | 1 | 12 |
438
+ | ChartQA | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
439
+ | GPQA-Diamond | 5 / 5 | 3 / 3 | 2 | 2 | 0 |
440
+ | GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
441
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
442
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
443
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 4 | 0 |
444
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 5 | 0 |
445
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
446
+ | MMLU-Pro | 1 / 1 | 0 / 0 | 1 | 1 | 0 |
447
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
448
+ | MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
449
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
450
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
451
+ | Q36-Benign-Compliance-No-Overrefusal | 10 / 20 | 0 / 0 | 10 | 0 | 10 |
452
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
453
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
454
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
455
+ | Q36-Hermes-Tool-Format | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
456
+ | Q36-JSON-Schema | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
457
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
458
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
459
+ | TruthfulQA | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
460
+
461
+ </details>
462
+
463
+ [Machine-readable counts](BENCHMARK-TEST-COUNTS.json). Code generation is not a pass until the relevant execution tests have been graded.
464
+
465
+ ## 🏁 Hard benchmarks on the roadmap
466
+
467
+ All **25 families** below are registered from the [Qwen3.8-27B card](https://huggingface.co/Qwen/Qwen3.8-27B). The matching official adapters and datasets are not yet fully prepared, and **no local Qwen3.8 baseline has been run**. Existing short similarly named diagnostics do not substitute for those runs.
468
+
469
+ <details>
470
+ <summary>Coding, reasoning, agent and visual benchmarks β€” complete planned list</summary>
471
+
472
+ | Benchmark | Planned local cases | Preparation status |
473
+ |---|---:|---|
474
+ | Terminal Bench 2.1 (Terminus) | 5 | Adapter/data preparation pending |
475
+ | SWE-bench Pro | 5 | Adapter/data preparation pending |
476
+ | NL2Repo-Bench | 5 | Adapter/data preparation pending |
477
+ | DeepSWE 1.1 | 5 | Adapter/data preparation pending |
478
+ | QwenSWEBench | TBD | Internal release/access needed |
479
+ | CoWorkBench | TBD | Internal release/access needed |
480
+ | JobBench | 5 | Adapter/data preparation pending |
481
+ | Agents' Last Exam | 5 | Adapter/data preparation pending |
482
+ | IFBench | 20 | Adapter/data preparation pending |
483
+ | GPQA Diamond | 20 | Adapter/data preparation pending |
484
+ | HLE | 10 | Adapter/data preparation pending |
485
+ | LiveCodeBench v6 | 10 | Adapter/data preparation pending |
486
+ | OSWorld-Verified | 5 | Adapter/data preparation pending |
487
+ | WebArena-Verified | 5 | Adapter/data preparation pending |
488
+ | AndroidWorld | 5 | Adapter/data preparation pending |
489
+ | RecreationBench | TBD | Internal release/access needed |
490
+ | ClawEval-MM | 5 | Adapter/data preparation pending |
491
+ | SWE-MM | 5 | Adapter/data preparation pending |
492
+ | Vision2Web | 5 | Adapter/data preparation pending |
493
+ | MathVision | 10 | Adapter/data preparation pending |
494
+ | BabyVision | 10 | Adapter/data preparation pending |
495
+ | CharXiv (RQ) | 10 | Adapter/data preparation pending |
496
+ | OmniDocBench 1.5 | 10 | Adapter/data preparation pending |
497
+ | RealWorldQA | 20 | Adapter/data preparation pending |
498
+ | ERQA | 10 | Adapter/data preparation pending |
499
+
500
+ </details>
501
+
502
+ Comparison graphics will follow measured results, with our local samples and publisher-reported scores kept clearly separate. Different harnesses, budgets and trial counts will not be presented as a head-to-head win. [Full protocol and references](BENCHMARK-PLAN.md).
503
+
504
+ ## πŸ’» Coding, agents and your laptop
505
+
506
+ The next local checks cover executable coding tests, bug fixes, tool calls, planning, recovery, memory and stopping at the right time. **64 case slots** are specified, including 19 existing coding slots. They are compact skill diagnostics, not proof of AGI. Results and failures will both be reported; new capability claims require actual task-level evidence.
507
+
508
+ [πŸ› οΈ Laptop test plan, Hermes routing and Obsidian workflows](LAPTOP-CAPABILITY-PLAN.md). The runtime comparison and full memory integration are **in preparation**, not yet validated. The four device quickstarts above remain separate from these future end-to-end checks.
509
+ <!-- q36-capability-expansion:end -->
510
+
511
  ## πŸ“œ License, lineage & credits
512
 
513
  [Apache-2.0 license](LICENSE). Dataset sources and their own license information are linked in [DATASETS.md](DATASETS.md); model licensing does not relicense the source datasets. Thanks to the Qwen team, the inherited model and dataset authors, and the Soup, PEFT, Transformers, llama.cpp and Ollama projects.
 
516
 
517
  These names do not imply affiliation or a promise of unrestricted behavior. Full provenance and validation are retained, including the historical Opus4.7-labelled source. No universal superiority, guaranteed compliance or removal of memorization is claimed.
518
 
519
+ <details>
520
+ <summary>🧬 What Ablit, Heretic, Uncensor, Hermes, MTP and Vision mean</summary>
521
+
522
+ This release builds on the [previous project's model card](https://huggingface.co/oktayd/Q36-35B-A3B-Opus4.7-Ablit-Heretic-OBLITERATUS-Hermes-MTP-Vision-FT), which records the following stages. These are inherited stages, not new operations performed while packaging this release.
523
+
524
+ | Name / stage | What it contributes |
525
+ |---|---|
526
+ | **Qwen3.6 / MoE** | The underlying language-and-vision architecture and mixture-of-experts backbone. |
527
+ | **Reasoning-distilled lineage** | The earlier card traces a lordx64 derivative followed by the huihui-ai derivative. The historical Opus4.7 label describes inherited reasoning-distillation provenance, not inclusion of proprietary Claude weights. |
528
+ | **Ablit / abliterated** | The huihui-ai source underwent a weight-modification stage aimed at reducing refusal behavior. It is not a separate tool library. |
529
+ | **Heretic** | A subsequent custom, fused-MoE-aware modification stage documented by the previous release. It is distinct from the earlier abliteration. |
530
+ | **OBLITERATUS Nuclear / Uncensor** | Another separately recorded inherited modification stage aimed at reducing refusal behavior. **Uncensor** is the concise release-name label; OBLITERATUS remains part of the provenance. |
531
+ | **Hermes** | Tool-oriented supervised training: function-call structure, coding, terminal/file/repository workflows and multi-tool coordination, using [Hermes Function Calling](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) and [Hermes Agent reasoning traces](https://huggingface.co/datasets/lambda/hermes-agent-reasoning-traces). Your host application still supplies, authorizes and executes tools. |
532
+ | **v2 fine-tuning** | Three additional Soup/PEFT training runs with 34,000 record uses spanning knowledge, instructions, coding, personality and selected vision data. See [datasets and counts](DATASETS.md). |
533
+ | **MTP** | Preserved multi-token-prediction tensors. Preservation is not evidence that speculative decoding is enabled or faster in your runtime. |
534
+ | **Vision** | Image-input architecture; GGUF runtimes also need the matching projector. It does not itself supply browser control, memory or a 3D engine. |
535
+
536
+ The old card reports **23,220 training and 1,179 validation examples** for its own earlier SFT stage; those are separate from the current release's 34,000 record uses. It also records protection of 333 vision tensors and 19 MTP tensors in that earlier build. These historical checks are not new laptop benchmark results.
537
+
538
+ </details>
539
+
540
  ## πŸ”— Related collection
541
 
542
  The previous release remains separate: [Qwen3.6 Opus4.7 Heretic Hermes Agent β€” Editions](https://huggingface.co/collections/oktayd/qwen36-opus47-heretic-hermes-agent-editions-6a8d09f6c2eb42ed1b112184).
SHA256SUMS CHANGED
@@ -2,11 +2,13 @@
2
  807d54b88902798393fc104c3ca80d3e424d64270e8d1e1a34a44d4574f52c7e BENCHMARK-DIAGNOSTIC.json
3
  0e712dd8e4bc65088e170826e98bdc440fb19a2cfdb7613c643e9dabb624ecd8 BENCHMARK-EXTENSION-QWEN38.json
4
  67d070e3b12bcd185626054c81cdf4d528a9a1af26774050e1d50be370b5ee14 BENCHMARK-PLAN.md
 
5
  5c1a6fa0bd2d21e400b4c84001abec8085e3f5d7fbe5d217e7a0af7f32f42c6d DATASETS.md
6
  b483301314df6773850288b6291d49d082a7f816f7266d8cd62ef803b4f4e154 DEVICE-QUICKSTART.md
7
  5fa5bdc4687e5ed723af58eb436f545cc62589e5395d5041722639f03bb5ec9b LAPTOP-BENCHMARK-MATRIX.json
 
8
  20a2a90fa761fe5081d31d25989d94656ad7b9766103b4244b71891e714dcc22 LICENSE
9
- 99bcfca97a75a836cc05bd9032c2731ec9de3e51da9fa4a8a4ab79787bb53874 README.md
10
  cd2b979e9e94047d3ec45a8fe6c6610765a0045be6a09e4e730e8c501405290c RELEASE-NAMING.json
11
  b36b05ac4d31eb371f841af813f3b3fdfad5c04375c58214013a94dd929c2ad0 RELEASE-VALIDATION.json
12
  3836329a27f1f7c4320c8aef72fd617f8ab4440e88e7419ed5cf785408c48cb2 RUNTIME-PATCH-VALIDATION.json
 
2
  807d54b88902798393fc104c3ca80d3e424d64270e8d1e1a34a44d4574f52c7e BENCHMARK-DIAGNOSTIC.json
3
  0e712dd8e4bc65088e170826e98bdc440fb19a2cfdb7613c643e9dabb624ecd8 BENCHMARK-EXTENSION-QWEN38.json
4
  67d070e3b12bcd185626054c81cdf4d528a9a1af26774050e1d50be370b5ee14 BENCHMARK-PLAN.md
5
+ cfd2a06c82f3f41ceaf8c333fb0ee0e38247d0c9419084c0501d5c2430bb7b68 BENCHMARK-TEST-COUNTS.json
6
  5c1a6fa0bd2d21e400b4c84001abec8085e3f5d7fbe5d217e7a0af7f32f42c6d DATASETS.md
7
  b483301314df6773850288b6291d49d082a7f816f7266d8cd62ef803b4f4e154 DEVICE-QUICKSTART.md
8
  5fa5bdc4687e5ed723af58eb436f545cc62589e5395d5041722639f03bb5ec9b LAPTOP-BENCHMARK-MATRIX.json
9
+ 42f57a1314669908fc20a4c657e32aafe4bedc9085508a31d1784cc3efd6c1c4 LAPTOP-CAPABILITY-PLAN.md
10
  20a2a90fa761fe5081d31d25989d94656ad7b9766103b4244b71891e714dcc22 LICENSE
11
+ b41257b9fb42fd8dfb4333d1c3a782bec4eaff2cd64273d4fb516e93a12180c7 README.md
12
  cd2b979e9e94047d3ec45a8fe6c6610765a0045be6a09e4e730e8c501405290c RELEASE-NAMING.json
13
  b36b05ac4d31eb371f841af813f3b3fdfad5c04375c58214013a94dd929c2ad0 RELEASE-VALIDATION.json
14
  3836329a27f1f7c4320c8aef72fd617f8ab4440e88e7419ed5cf785408c48cb2 RUNTIME-PATCH-VALIDATION.json