oktayd commited on
Commit
c468165
Β·
verified Β·
1 Parent(s): e75cb2c

Explain MoE and inherited stages; add per-test benchmark dropdowns and laptop capability roadmap

Browse files
Files changed (4) hide show
  1. BENCHMARK-TEST-COUNTS.json +1362 -0
  2. LAPTOP-CAPABILITY-PLAN.md +51 -0
  3. README.md +241 -1
  4. SHA256SUMS +3 -1
BENCHMARK-TEST-COUNTS.json ADDED
@@ -0,0 +1,1362 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": "q36.public-strict-counts.v1",
3
+ "source_sha256": "9aef69af26b03d3b26a9250a1f6c21fdd4972a63522806d7724454b66b820b17",
4
+ "scoring": "Original strict evaluator, not semantic regrade. Ungraded and incomplete may overlap.",
5
+ "runs": [
6
+ {
7
+ "run": "q36-final-q4_k_m-llamacpp-h200",
8
+ "model": "Q36",
9
+ "device": "H200",
10
+ "benchmarks": [
11
+ {
12
+ "benchmark": "ARC-Challenge",
13
+ "planned": 10,
14
+ "attempted": 10,
15
+ "completed": 8,
16
+ "scored": 8,
17
+ "passed": 8,
18
+ "ungraded": 2,
19
+ "not_run": 0,
20
+ "incomplete": 2
21
+ },
22
+ {
23
+ "benchmark": "BBH",
24
+ "planned": 23,
25
+ "attempted": 23,
26
+ "completed": 20,
27
+ "scored": 20,
28
+ "passed": 3,
29
+ "ungraded": 3,
30
+ "not_run": 0,
31
+ "incomplete": 3
32
+ },
33
+ {
34
+ "benchmark": "ChartQA",
35
+ "planned": 5,
36
+ "attempted": 5,
37
+ "completed": 5,
38
+ "scored": 5,
39
+ "passed": 4,
40
+ "ungraded": 0,
41
+ "not_run": 0,
42
+ "incomplete": 0
43
+ },
44
+ {
45
+ "benchmark": "GPQA-Diamond",
46
+ "planned": 5,
47
+ "attempted": 5,
48
+ "completed": 5,
49
+ "scored": 5,
50
+ "passed": 2,
51
+ "ungraded": 0,
52
+ "not_run": 0,
53
+ "incomplete": 0
54
+ },
55
+ {
56
+ "benchmark": "GSM8K",
57
+ "planned": 10,
58
+ "attempted": 10,
59
+ "completed": 9,
60
+ "scored": 9,
61
+ "passed": 5,
62
+ "ungraded": 1,
63
+ "not_run": 0,
64
+ "incomplete": 1
65
+ },
66
+ {
67
+ "benchmark": "HumanEval-Plus",
68
+ "planned": 5,
69
+ "attempted": 5,
70
+ "completed": 5,
71
+ "scored": 0,
72
+ "passed": 0,
73
+ "ungraded": 5,
74
+ "not_run": 0,
75
+ "incomplete": 0
76
+ },
77
+ {
78
+ "benchmark": "IFEval",
79
+ "planned": 10,
80
+ "attempted": 10,
81
+ "completed": 6,
82
+ "scored": 0,
83
+ "passed": 0,
84
+ "ungraded": 10,
85
+ "not_run": 0,
86
+ "incomplete": 4
87
+ },
88
+ {
89
+ "benchmark": "LiveCodeBench",
90
+ "planned": 9,
91
+ "attempted": 9,
92
+ "completed": 9,
93
+ "scored": 0,
94
+ "passed": 0,
95
+ "ungraded": 9,
96
+ "not_run": 0,
97
+ "incomplete": 0
98
+ },
99
+ {
100
+ "benchmark": "MATH-Level-5",
101
+ "planned": 10,
102
+ "attempted": 10,
103
+ "completed": 8,
104
+ "scored": 0,
105
+ "passed": 0,
106
+ "ungraded": 10,
107
+ "not_run": 0,
108
+ "incomplete": 2
109
+ },
110
+ {
111
+ "benchmark": "MBPP-Plus",
112
+ "planned": 5,
113
+ "attempted": 5,
114
+ "completed": 5,
115
+ "scored": 0,
116
+ "passed": 0,
117
+ "ungraded": 5,
118
+ "not_run": 0,
119
+ "incomplete": 0
120
+ },
121
+ {
122
+ "benchmark": "MMLU-Pro",
123
+ "planned": 1,
124
+ "attempted": 1,
125
+ "completed": 1,
126
+ "scored": 1,
127
+ "passed": 0,
128
+ "ungraded": 0,
129
+ "not_run": 0,
130
+ "incomplete": 0
131
+ },
132
+ {
133
+ "benchmark": "MMMU",
134
+ "planned": 6,
135
+ "attempted": 6,
136
+ "completed": 6,
137
+ "scored": 6,
138
+ "passed": 4,
139
+ "ungraded": 0,
140
+ "not_run": 0,
141
+ "incomplete": 0
142
+ },
143
+ {
144
+ "benchmark": "MathVista",
145
+ "planned": 5,
146
+ "attempted": 5,
147
+ "completed": 5,
148
+ "scored": 5,
149
+ "passed": 2,
150
+ "ungraded": 0,
151
+ "not_run": 0,
152
+ "incomplete": 0
153
+ },
154
+ {
155
+ "benchmark": "Q36-Agent-Function-Calling",
156
+ "planned": 5,
157
+ "attempted": 5,
158
+ "completed": 5,
159
+ "scored": 5,
160
+ "passed": 0,
161
+ "ungraded": 0,
162
+ "not_run": 0,
163
+ "incomplete": 0
164
+ },
165
+ {
166
+ "benchmark": "Q36-Answer-Termination-No-Looping",
167
+ "planned": 10,
168
+ "attempted": 10,
169
+ "completed": 10,
170
+ "scored": 10,
171
+ "passed": 7,
172
+ "ungraded": 0,
173
+ "not_run": 0,
174
+ "incomplete": 0
175
+ },
176
+ {
177
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
178
+ "planned": 20,
179
+ "attempted": 20,
180
+ "completed": 18,
181
+ "scored": 0,
182
+ "passed": 0,
183
+ "ungraded": 20,
184
+ "not_run": 0,
185
+ "incomplete": 2
186
+ },
187
+ {
188
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
189
+ "planned": 10,
190
+ "attempted": 10,
191
+ "completed": 10,
192
+ "scored": 10,
193
+ "passed": 10,
194
+ "ungraded": 0,
195
+ "not_run": 0,
196
+ "incomplete": 0
197
+ },
198
+ {
199
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
200
+ "planned": 10,
201
+ "attempted": 10,
202
+ "completed": 8,
203
+ "scored": 0,
204
+ "passed": 0,
205
+ "ungraded": 10,
206
+ "not_run": 0,
207
+ "incomplete": 2
208
+ },
209
+ {
210
+ "benchmark": "Q36-Direct-Style-and-Personality",
211
+ "planned": 10,
212
+ "attempted": 10,
213
+ "completed": 9,
214
+ "scored": 0,
215
+ "passed": 0,
216
+ "ungraded": 10,
217
+ "not_run": 0,
218
+ "incomplete": 1
219
+ },
220
+ {
221
+ "benchmark": "Q36-Hermes-Tool-Format",
222
+ "planned": 5,
223
+ "attempted": 5,
224
+ "completed": 5,
225
+ "scored": 5,
226
+ "passed": 5,
227
+ "ungraded": 0,
228
+ "not_run": 0,
229
+ "incomplete": 0
230
+ },
231
+ {
232
+ "benchmark": "Q36-JSON-Schema",
233
+ "planned": 5,
234
+ "attempted": 5,
235
+ "completed": 5,
236
+ "scored": 5,
237
+ "passed": 0,
238
+ "ungraded": 0,
239
+ "not_run": 0,
240
+ "incomplete": 0
241
+ },
242
+ {
243
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
244
+ "planned": 10,
245
+ "attempted": 10,
246
+ "completed": 10,
247
+ "scored": 0,
248
+ "passed": 0,
249
+ "ungraded": 10,
250
+ "not_run": 0,
251
+ "incomplete": 0
252
+ },
253
+ {
254
+ "benchmark": "Q36-Output-Integrity",
255
+ "planned": 5,
256
+ "attempted": 5,
257
+ "completed": 5,
258
+ "scored": 5,
259
+ "passed": 5,
260
+ "ungraded": 0,
261
+ "not_run": 0,
262
+ "incomplete": 0
263
+ },
264
+ {
265
+ "benchmark": "TruthfulQA",
266
+ "planned": 10,
267
+ "attempted": 10,
268
+ "completed": 9,
269
+ "scored": 9,
270
+ "passed": 6,
271
+ "ungraded": 1,
272
+ "not_run": 0,
273
+ "incomplete": 1
274
+ }
275
+ ]
276
+ },
277
+ {
278
+ "run": "q36-final-q4_k_m-llamacpp-rtx5090",
279
+ "model": "Q36",
280
+ "device": "RTX 5090",
281
+ "benchmarks": [
282
+ {
283
+ "benchmark": "ARC-Challenge",
284
+ "planned": 10,
285
+ "attempted": 10,
286
+ "completed": 8,
287
+ "scored": 8,
288
+ "passed": 8,
289
+ "ungraded": 2,
290
+ "not_run": 0,
291
+ "incomplete": 2
292
+ },
293
+ {
294
+ "benchmark": "BBH",
295
+ "planned": 23,
296
+ "attempted": 23,
297
+ "completed": 20,
298
+ "scored": 20,
299
+ "passed": 3,
300
+ "ungraded": 3,
301
+ "not_run": 0,
302
+ "incomplete": 3
303
+ },
304
+ {
305
+ "benchmark": "ChartQA",
306
+ "planned": 5,
307
+ "attempted": 5,
308
+ "completed": 5,
309
+ "scored": 5,
310
+ "passed": 4,
311
+ "ungraded": 0,
312
+ "not_run": 0,
313
+ "incomplete": 0
314
+ },
315
+ {
316
+ "benchmark": "GPQA-Diamond",
317
+ "planned": 5,
318
+ "attempted": 5,
319
+ "completed": 5,
320
+ "scored": 5,
321
+ "passed": 1,
322
+ "ungraded": 0,
323
+ "not_run": 0,
324
+ "incomplete": 0
325
+ },
326
+ {
327
+ "benchmark": "GSM8K",
328
+ "planned": 10,
329
+ "attempted": 10,
330
+ "completed": 10,
331
+ "scored": 10,
332
+ "passed": 7,
333
+ "ungraded": 0,
334
+ "not_run": 0,
335
+ "incomplete": 0
336
+ },
337
+ {
338
+ "benchmark": "HumanEval-Plus",
339
+ "planned": 5,
340
+ "attempted": 5,
341
+ "completed": 4,
342
+ "scored": 0,
343
+ "passed": 0,
344
+ "ungraded": 5,
345
+ "not_run": 0,
346
+ "incomplete": 1
347
+ },
348
+ {
349
+ "benchmark": "IFEval",
350
+ "planned": 10,
351
+ "attempted": 10,
352
+ "completed": 8,
353
+ "scored": 0,
354
+ "passed": 0,
355
+ "ungraded": 10,
356
+ "not_run": 0,
357
+ "incomplete": 2
358
+ },
359
+ {
360
+ "benchmark": "LiveCodeBench",
361
+ "planned": 9,
362
+ "attempted": 9,
363
+ "completed": 9,
364
+ "scored": 0,
365
+ "passed": 0,
366
+ "ungraded": 9,
367
+ "not_run": 0,
368
+ "incomplete": 0
369
+ },
370
+ {
371
+ "benchmark": "MATH-Level-5",
372
+ "planned": 10,
373
+ "attempted": 10,
374
+ "completed": 9,
375
+ "scored": 0,
376
+ "passed": 0,
377
+ "ungraded": 10,
378
+ "not_run": 0,
379
+ "incomplete": 1
380
+ },
381
+ {
382
+ "benchmark": "MBPP-Plus",
383
+ "planned": 5,
384
+ "attempted": 5,
385
+ "completed": 5,
386
+ "scored": 0,
387
+ "passed": 0,
388
+ "ungraded": 5,
389
+ "not_run": 0,
390
+ "incomplete": 0
391
+ },
392
+ {
393
+ "benchmark": "MMLU-Pro",
394
+ "planned": 1,
395
+ "attempted": 1,
396
+ "completed": 1,
397
+ "scored": 1,
398
+ "passed": 0,
399
+ "ungraded": 0,
400
+ "not_run": 0,
401
+ "incomplete": 0
402
+ },
403
+ {
404
+ "benchmark": "MMMU",
405
+ "planned": 6,
406
+ "attempted": 6,
407
+ "completed": 6,
408
+ "scored": 6,
409
+ "passed": 4,
410
+ "ungraded": 0,
411
+ "not_run": 0,
412
+ "incomplete": 0
413
+ },
414
+ {
415
+ "benchmark": "MathVista",
416
+ "planned": 5,
417
+ "attempted": 5,
418
+ "completed": 5,
419
+ "scored": 5,
420
+ "passed": 2,
421
+ "ungraded": 0,
422
+ "not_run": 0,
423
+ "incomplete": 0
424
+ },
425
+ {
426
+ "benchmark": "Q36-Agent-Function-Calling",
427
+ "planned": 5,
428
+ "attempted": 5,
429
+ "completed": 5,
430
+ "scored": 5,
431
+ "passed": 0,
432
+ "ungraded": 0,
433
+ "not_run": 0,
434
+ "incomplete": 0
435
+ },
436
+ {
437
+ "benchmark": "Q36-Answer-Termination-No-Looping",
438
+ "planned": 10,
439
+ "attempted": 10,
440
+ "completed": 10,
441
+ "scored": 10,
442
+ "passed": 9,
443
+ "ungraded": 0,
444
+ "not_run": 0,
445
+ "incomplete": 0
446
+ },
447
+ {
448
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
449
+ "planned": 20,
450
+ "attempted": 20,
451
+ "completed": 18,
452
+ "scored": 0,
453
+ "passed": 0,
454
+ "ungraded": 20,
455
+ "not_run": 0,
456
+ "incomplete": 2
457
+ },
458
+ {
459
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
460
+ "planned": 10,
461
+ "attempted": 10,
462
+ "completed": 10,
463
+ "scored": 10,
464
+ "passed": 10,
465
+ "ungraded": 0,
466
+ "not_run": 0,
467
+ "incomplete": 0
468
+ },
469
+ {
470
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
471
+ "planned": 10,
472
+ "attempted": 10,
473
+ "completed": 8,
474
+ "scored": 0,
475
+ "passed": 0,
476
+ "ungraded": 10,
477
+ "not_run": 0,
478
+ "incomplete": 2
479
+ },
480
+ {
481
+ "benchmark": "Q36-Direct-Style-and-Personality",
482
+ "planned": 10,
483
+ "attempted": 10,
484
+ "completed": 10,
485
+ "scored": 0,
486
+ "passed": 0,
487
+ "ungraded": 10,
488
+ "not_run": 0,
489
+ "incomplete": 0
490
+ },
491
+ {
492
+ "benchmark": "Q36-Hermes-Tool-Format",
493
+ "planned": 5,
494
+ "attempted": 5,
495
+ "completed": 5,
496
+ "scored": 5,
497
+ "passed": 3,
498
+ "ungraded": 0,
499
+ "not_run": 0,
500
+ "incomplete": 0
501
+ },
502
+ {
503
+ "benchmark": "Q36-JSON-Schema",
504
+ "planned": 5,
505
+ "attempted": 5,
506
+ "completed": 5,
507
+ "scored": 5,
508
+ "passed": 0,
509
+ "ungraded": 0,
510
+ "not_run": 0,
511
+ "incomplete": 0
512
+ },
513
+ {
514
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
515
+ "planned": 10,
516
+ "attempted": 10,
517
+ "completed": 10,
518
+ "scored": 0,
519
+ "passed": 0,
520
+ "ungraded": 10,
521
+ "not_run": 0,
522
+ "incomplete": 0
523
+ },
524
+ {
525
+ "benchmark": "Q36-Output-Integrity",
526
+ "planned": 5,
527
+ "attempted": 5,
528
+ "completed": 5,
529
+ "scored": 5,
530
+ "passed": 5,
531
+ "ungraded": 0,
532
+ "not_run": 0,
533
+ "incomplete": 0
534
+ },
535
+ {
536
+ "benchmark": "TruthfulQA",
537
+ "planned": 10,
538
+ "attempted": 10,
539
+ "completed": 9,
540
+ "scored": 9,
541
+ "passed": 6,
542
+ "ungraded": 1,
543
+ "not_run": 0,
544
+ "incomplete": 1
545
+ }
546
+ ]
547
+ },
548
+ {
549
+ "run": "huihui-original-q4_k_m-llamacpp-rtx5090",
550
+ "model": "Huihui",
551
+ "device": "RTX 5090",
552
+ "benchmarks": [
553
+ {
554
+ "benchmark": "ARC-Challenge",
555
+ "planned": 10,
556
+ "attempted": 10,
557
+ "completed": 10,
558
+ "scored": 10,
559
+ "passed": 10,
560
+ "ungraded": 0,
561
+ "not_run": 0,
562
+ "incomplete": 0
563
+ },
564
+ {
565
+ "benchmark": "BBH",
566
+ "planned": 23,
567
+ "attempted": 23,
568
+ "completed": 23,
569
+ "scored": 23,
570
+ "passed": 16,
571
+ "ungraded": 0,
572
+ "not_run": 0,
573
+ "incomplete": 0
574
+ },
575
+ {
576
+ "benchmark": "ChartQA",
577
+ "planned": 5,
578
+ "attempted": 5,
579
+ "completed": 5,
580
+ "scored": 5,
581
+ "passed": 5,
582
+ "ungraded": 0,
583
+ "not_run": 0,
584
+ "incomplete": 0
585
+ },
586
+ {
587
+ "benchmark": "GPQA-Diamond",
588
+ "planned": 5,
589
+ "attempted": 5,
590
+ "completed": 5,
591
+ "scored": 5,
592
+ "passed": 5,
593
+ "ungraded": 0,
594
+ "not_run": 0,
595
+ "incomplete": 0
596
+ },
597
+ {
598
+ "benchmark": "GSM8K",
599
+ "planned": 10,
600
+ "attempted": 10,
601
+ "completed": 10,
602
+ "scored": 10,
603
+ "passed": 7,
604
+ "ungraded": 0,
605
+ "not_run": 0,
606
+ "incomplete": 0
607
+ },
608
+ {
609
+ "benchmark": "HumanEval-Plus",
610
+ "planned": 5,
611
+ "attempted": 5,
612
+ "completed": 5,
613
+ "scored": 0,
614
+ "passed": 0,
615
+ "ungraded": 5,
616
+ "not_run": 0,
617
+ "incomplete": 0
618
+ },
619
+ {
620
+ "benchmark": "IFEval",
621
+ "planned": 10,
622
+ "attempted": 10,
623
+ "completed": 10,
624
+ "scored": 0,
625
+ "passed": 0,
626
+ "ungraded": 10,
627
+ "not_run": 0,
628
+ "incomplete": 0
629
+ },
630
+ {
631
+ "benchmark": "LiveCodeBench",
632
+ "planned": 9,
633
+ "attempted": 9,
634
+ "completed": 4,
635
+ "scored": 0,
636
+ "passed": 0,
637
+ "ungraded": 9,
638
+ "not_run": 0,
639
+ "incomplete": 5
640
+ },
641
+ {
642
+ "benchmark": "MATH-Level-5",
643
+ "planned": 10,
644
+ "attempted": 10,
645
+ "completed": 8,
646
+ "scored": 0,
647
+ "passed": 0,
648
+ "ungraded": 10,
649
+ "not_run": 0,
650
+ "incomplete": 2
651
+ },
652
+ {
653
+ "benchmark": "MBPP-Plus",
654
+ "planned": 5,
655
+ "attempted": 5,
656
+ "completed": 5,
657
+ "scored": 0,
658
+ "passed": 0,
659
+ "ungraded": 5,
660
+ "not_run": 0,
661
+ "incomplete": 0
662
+ },
663
+ {
664
+ "benchmark": "MMLU-Pro",
665
+ "planned": 1,
666
+ "attempted": 1,
667
+ "completed": 1,
668
+ "scored": 1,
669
+ "passed": 0,
670
+ "ungraded": 0,
671
+ "not_run": 0,
672
+ "incomplete": 0
673
+ },
674
+ {
675
+ "benchmark": "MMMU",
676
+ "planned": 6,
677
+ "attempted": 6,
678
+ "completed": 6,
679
+ "scored": 6,
680
+ "passed": 4,
681
+ "ungraded": 0,
682
+ "not_run": 0,
683
+ "incomplete": 0
684
+ },
685
+ {
686
+ "benchmark": "MathVista",
687
+ "planned": 5,
688
+ "attempted": 5,
689
+ "completed": 5,
690
+ "scored": 5,
691
+ "passed": 3,
692
+ "ungraded": 0,
693
+ "not_run": 0,
694
+ "incomplete": 0
695
+ },
696
+ {
697
+ "benchmark": "Q36-Agent-Function-Calling",
698
+ "planned": 5,
699
+ "attempted": 5,
700
+ "completed": 5,
701
+ "scored": 5,
702
+ "passed": 0,
703
+ "ungraded": 0,
704
+ "not_run": 0,
705
+ "incomplete": 0
706
+ },
707
+ {
708
+ "benchmark": "Q36-Answer-Termination-No-Looping",
709
+ "planned": 10,
710
+ "attempted": 10,
711
+ "completed": 10,
712
+ "scored": 10,
713
+ "passed": 10,
714
+ "ungraded": 0,
715
+ "not_run": 0,
716
+ "incomplete": 0
717
+ },
718
+ {
719
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
720
+ "planned": 20,
721
+ "attempted": 20,
722
+ "completed": 20,
723
+ "scored": 0,
724
+ "passed": 0,
725
+ "ungraded": 20,
726
+ "not_run": 0,
727
+ "incomplete": 0
728
+ },
729
+ {
730
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
731
+ "planned": 10,
732
+ "attempted": 10,
733
+ "completed": 10,
734
+ "scored": 10,
735
+ "passed": 10,
736
+ "ungraded": 0,
737
+ "not_run": 0,
738
+ "incomplete": 0
739
+ },
740
+ {
741
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
742
+ "planned": 10,
743
+ "attempted": 10,
744
+ "completed": 10,
745
+ "scored": 0,
746
+ "passed": 0,
747
+ "ungraded": 10,
748
+ "not_run": 0,
749
+ "incomplete": 0
750
+ },
751
+ {
752
+ "benchmark": "Q36-Direct-Style-and-Personality",
753
+ "planned": 10,
754
+ "attempted": 10,
755
+ "completed": 10,
756
+ "scored": 0,
757
+ "passed": 0,
758
+ "ungraded": 10,
759
+ "not_run": 0,
760
+ "incomplete": 0
761
+ },
762
+ {
763
+ "benchmark": "Q36-Hermes-Tool-Format",
764
+ "planned": 5,
765
+ "attempted": 5,
766
+ "completed": 5,
767
+ "scored": 5,
768
+ "passed": 4,
769
+ "ungraded": 0,
770
+ "not_run": 0,
771
+ "incomplete": 0
772
+ },
773
+ {
774
+ "benchmark": "Q36-JSON-Schema",
775
+ "planned": 5,
776
+ "attempted": 5,
777
+ "completed": 5,
778
+ "scored": 5,
779
+ "passed": 2,
780
+ "ungraded": 0,
781
+ "not_run": 0,
782
+ "incomplete": 0
783
+ },
784
+ {
785
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
786
+ "planned": 10,
787
+ "attempted": 10,
788
+ "completed": 10,
789
+ "scored": 0,
790
+ "passed": 0,
791
+ "ungraded": 10,
792
+ "not_run": 0,
793
+ "incomplete": 0
794
+ },
795
+ {
796
+ "benchmark": "Q36-Output-Integrity",
797
+ "planned": 5,
798
+ "attempted": 5,
799
+ "completed": 5,
800
+ "scored": 5,
801
+ "passed": 5,
802
+ "ungraded": 0,
803
+ "not_run": 0,
804
+ "incomplete": 0
805
+ },
806
+ {
807
+ "benchmark": "TruthfulQA",
808
+ "planned": 10,
809
+ "attempted": 10,
810
+ "completed": 9,
811
+ "scored": 9,
812
+ "passed": 6,
813
+ "ungraded": 1,
814
+ "not_run": 0,
815
+ "incomplete": 1
816
+ }
817
+ ]
818
+ },
819
+ {
820
+ "run": "q36-final-q4_k_m-llamacpp-rtx2000ada-mixed24",
821
+ "model": "Q36",
822
+ "device": "RTX 2000 Ada (CPU+GPU)",
823
+ "benchmarks": [
824
+ {
825
+ "benchmark": "ARC-Challenge",
826
+ "planned": 10,
827
+ "attempted": 10,
828
+ "completed": 7,
829
+ "scored": 7,
830
+ "passed": 7,
831
+ "ungraded": 3,
832
+ "not_run": 0,
833
+ "incomplete": 3
834
+ },
835
+ {
836
+ "benchmark": "BBH",
837
+ "planned": 23,
838
+ "attempted": 23,
839
+ "completed": 20,
840
+ "scored": 20,
841
+ "passed": 3,
842
+ "ungraded": 3,
843
+ "not_run": 0,
844
+ "incomplete": 3
845
+ },
846
+ {
847
+ "benchmark": "ChartQA",
848
+ "planned": 5,
849
+ "attempted": 5,
850
+ "completed": 5,
851
+ "scored": 5,
852
+ "passed": 4,
853
+ "ungraded": 0,
854
+ "not_run": 0,
855
+ "incomplete": 0
856
+ },
857
+ {
858
+ "benchmark": "GPQA-Diamond",
859
+ "planned": 5,
860
+ "attempted": 5,
861
+ "completed": 5,
862
+ "scored": 5,
863
+ "passed": 2,
864
+ "ungraded": 0,
865
+ "not_run": 0,
866
+ "incomplete": 0
867
+ },
868
+ {
869
+ "benchmark": "GSM8K",
870
+ "planned": 10,
871
+ "attempted": 10,
872
+ "completed": 10,
873
+ "scored": 10,
874
+ "passed": 8,
875
+ "ungraded": 0,
876
+ "not_run": 0,
877
+ "incomplete": 0
878
+ },
879
+ {
880
+ "benchmark": "HumanEval-Plus",
881
+ "planned": 5,
882
+ "attempted": 5,
883
+ "completed": 5,
884
+ "scored": 0,
885
+ "passed": 0,
886
+ "ungraded": 5,
887
+ "not_run": 0,
888
+ "incomplete": 0
889
+ },
890
+ {
891
+ "benchmark": "IFEval",
892
+ "planned": 10,
893
+ "attempted": 10,
894
+ "completed": 9,
895
+ "scored": 0,
896
+ "passed": 0,
897
+ "ungraded": 10,
898
+ "not_run": 0,
899
+ "incomplete": 1
900
+ },
901
+ {
902
+ "benchmark": "LiveCodeBench",
903
+ "planned": 9,
904
+ "attempted": 9,
905
+ "completed": 9,
906
+ "scored": 0,
907
+ "passed": 0,
908
+ "ungraded": 9,
909
+ "not_run": 0,
910
+ "incomplete": 0
911
+ },
912
+ {
913
+ "benchmark": "MATH-Level-5",
914
+ "planned": 10,
915
+ "attempted": 10,
916
+ "completed": 9,
917
+ "scored": 0,
918
+ "passed": 0,
919
+ "ungraded": 10,
920
+ "not_run": 0,
921
+ "incomplete": 1
922
+ },
923
+ {
924
+ "benchmark": "MBPP-Plus",
925
+ "planned": 5,
926
+ "attempted": 5,
927
+ "completed": 5,
928
+ "scored": 0,
929
+ "passed": 0,
930
+ "ungraded": 5,
931
+ "not_run": 0,
932
+ "incomplete": 0
933
+ },
934
+ {
935
+ "benchmark": "MMLU-Pro",
936
+ "planned": 1,
937
+ "attempted": 1,
938
+ "completed": 1,
939
+ "scored": 1,
940
+ "passed": 0,
941
+ "ungraded": 0,
942
+ "not_run": 0,
943
+ "incomplete": 0
944
+ },
945
+ {
946
+ "benchmark": "MMMU",
947
+ "planned": 6,
948
+ "attempted": 6,
949
+ "completed": 6,
950
+ "scored": 6,
951
+ "passed": 1,
952
+ "ungraded": 0,
953
+ "not_run": 0,
954
+ "incomplete": 0
955
+ },
956
+ {
957
+ "benchmark": "MathVista",
958
+ "planned": 5,
959
+ "attempted": 5,
960
+ "completed": 5,
961
+ "scored": 5,
962
+ "passed": 3,
963
+ "ungraded": 0,
964
+ "not_run": 0,
965
+ "incomplete": 0
966
+ },
967
+ {
968
+ "benchmark": "Q36-Agent-Function-Calling",
969
+ "planned": 5,
970
+ "attempted": 5,
971
+ "completed": 5,
972
+ "scored": 5,
973
+ "passed": 0,
974
+ "ungraded": 0,
975
+ "not_run": 0,
976
+ "incomplete": 0
977
+ },
978
+ {
979
+ "benchmark": "Q36-Answer-Termination-No-Looping",
980
+ "planned": 10,
981
+ "attempted": 10,
982
+ "completed": 10,
983
+ "scored": 10,
984
+ "passed": 8,
985
+ "ungraded": 0,
986
+ "not_run": 0,
987
+ "incomplete": 0
988
+ },
989
+ {
990
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
991
+ "planned": 20,
992
+ "attempted": 20,
993
+ "completed": 19,
994
+ "scored": 0,
995
+ "passed": 0,
996
+ "ungraded": 20,
997
+ "not_run": 0,
998
+ "incomplete": 1
999
+ },
1000
+ {
1001
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
1002
+ "planned": 10,
1003
+ "attempted": 10,
1004
+ "completed": 10,
1005
+ "scored": 10,
1006
+ "passed": 9,
1007
+ "ungraded": 0,
1008
+ "not_run": 0,
1009
+ "incomplete": 0
1010
+ },
1011
+ {
1012
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
1013
+ "planned": 10,
1014
+ "attempted": 10,
1015
+ "completed": 9,
1016
+ "scored": 0,
1017
+ "passed": 0,
1018
+ "ungraded": 10,
1019
+ "not_run": 0,
1020
+ "incomplete": 1
1021
+ },
1022
+ {
1023
+ "benchmark": "Q36-Direct-Style-and-Personality",
1024
+ "planned": 10,
1025
+ "attempted": 10,
1026
+ "completed": 10,
1027
+ "scored": 0,
1028
+ "passed": 0,
1029
+ "ungraded": 10,
1030
+ "not_run": 0,
1031
+ "incomplete": 0
1032
+ },
1033
+ {
1034
+ "benchmark": "Q36-Hermes-Tool-Format",
1035
+ "planned": 5,
1036
+ "attempted": 5,
1037
+ "completed": 5,
1038
+ "scored": 5,
1039
+ "passed": 5,
1040
+ "ungraded": 0,
1041
+ "not_run": 0,
1042
+ "incomplete": 0
1043
+ },
1044
+ {
1045
+ "benchmark": "Q36-JSON-Schema",
1046
+ "planned": 5,
1047
+ "attempted": 5,
1048
+ "completed": 5,
1049
+ "scored": 5,
1050
+ "passed": 0,
1051
+ "ungraded": 0,
1052
+ "not_run": 0,
1053
+ "incomplete": 0
1054
+ },
1055
+ {
1056
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
1057
+ "planned": 10,
1058
+ "attempted": 10,
1059
+ "completed": 10,
1060
+ "scored": 0,
1061
+ "passed": 0,
1062
+ "ungraded": 10,
1063
+ "not_run": 0,
1064
+ "incomplete": 0
1065
+ },
1066
+ {
1067
+ "benchmark": "Q36-Output-Integrity",
1068
+ "planned": 5,
1069
+ "attempted": 5,
1070
+ "completed": 5,
1071
+ "scored": 5,
1072
+ "passed": 5,
1073
+ "ungraded": 0,
1074
+ "not_run": 0,
1075
+ "incomplete": 0
1076
+ },
1077
+ {
1078
+ "benchmark": "TruthfulQA",
1079
+ "planned": 10,
1080
+ "attempted": 10,
1081
+ "completed": 10,
1082
+ "scored": 10,
1083
+ "passed": 6,
1084
+ "ungraded": 0,
1085
+ "not_run": 0,
1086
+ "incomplete": 0
1087
+ }
1088
+ ]
1089
+ },
1090
+ {
1091
+ "run": "huihui-original-q4_k_m-llamacpp-rtx2000ada-mixed24",
1092
+ "model": "Huihui",
1093
+ "device": "RTX 2000 Ada (CPU+GPU)",
1094
+ "benchmarks": [
1095
+ {
1096
+ "benchmark": "ARC-Challenge",
1097
+ "planned": 10,
1098
+ "attempted": 10,
1099
+ "completed": 10,
1100
+ "scored": 10,
1101
+ "passed": 10,
1102
+ "ungraded": 0,
1103
+ "not_run": 0,
1104
+ "incomplete": 0
1105
+ },
1106
+ {
1107
+ "benchmark": "BBH",
1108
+ "planned": 23,
1109
+ "attempted": 11,
1110
+ "completed": 10,
1111
+ "scored": 10,
1112
+ "passed": 6,
1113
+ "ungraded": 1,
1114
+ "not_run": 12,
1115
+ "incomplete": 1
1116
+ },
1117
+ {
1118
+ "benchmark": "ChartQA",
1119
+ "planned": 5,
1120
+ "attempted": 5,
1121
+ "completed": 5,
1122
+ "scored": 5,
1123
+ "passed": 5,
1124
+ "ungraded": 0,
1125
+ "not_run": 0,
1126
+ "incomplete": 0
1127
+ },
1128
+ {
1129
+ "benchmark": "GPQA-Diamond",
1130
+ "planned": 5,
1131
+ "attempted": 5,
1132
+ "completed": 3,
1133
+ "scored": 3,
1134
+ "passed": 3,
1135
+ "ungraded": 2,
1136
+ "not_run": 0,
1137
+ "incomplete": 2
1138
+ },
1139
+ {
1140
+ "benchmark": "GSM8K",
1141
+ "planned": 10,
1142
+ "attempted": 10,
1143
+ "completed": 10,
1144
+ "scored": 10,
1145
+ "passed": 7,
1146
+ "ungraded": 0,
1147
+ "not_run": 0,
1148
+ "incomplete": 0
1149
+ },
1150
+ {
1151
+ "benchmark": "HumanEval-Plus",
1152
+ "planned": 5,
1153
+ "attempted": 5,
1154
+ "completed": 5,
1155
+ "scored": 0,
1156
+ "passed": 0,
1157
+ "ungraded": 5,
1158
+ "not_run": 0,
1159
+ "incomplete": 0
1160
+ },
1161
+ {
1162
+ "benchmark": "IFEval",
1163
+ "planned": 10,
1164
+ "attempted": 10,
1165
+ "completed": 10,
1166
+ "scored": 0,
1167
+ "passed": 0,
1168
+ "ungraded": 10,
1169
+ "not_run": 0,
1170
+ "incomplete": 0
1171
+ },
1172
+ {
1173
+ "benchmark": "LiveCodeBench",
1174
+ "planned": 9,
1175
+ "attempted": 9,
1176
+ "completed": 5,
1177
+ "scored": 0,
1178
+ "passed": 0,
1179
+ "ungraded": 9,
1180
+ "not_run": 0,
1181
+ "incomplete": 4
1182
+ },
1183
+ {
1184
+ "benchmark": "MATH-Level-5",
1185
+ "planned": 10,
1186
+ "attempted": 10,
1187
+ "completed": 5,
1188
+ "scored": 0,
1189
+ "passed": 0,
1190
+ "ungraded": 10,
1191
+ "not_run": 0,
1192
+ "incomplete": 5
1193
+ },
1194
+ {
1195
+ "benchmark": "MBPP-Plus",
1196
+ "planned": 5,
1197
+ "attempted": 5,
1198
+ "completed": 5,
1199
+ "scored": 0,
1200
+ "passed": 0,
1201
+ "ungraded": 5,
1202
+ "not_run": 0,
1203
+ "incomplete": 0
1204
+ },
1205
+ {
1206
+ "benchmark": "MMLU-Pro",
1207
+ "planned": 1,
1208
+ "attempted": 1,
1209
+ "completed": 0,
1210
+ "scored": 0,
1211
+ "passed": 0,
1212
+ "ungraded": 1,
1213
+ "not_run": 0,
1214
+ "incomplete": 1
1215
+ },
1216
+ {
1217
+ "benchmark": "MMMU",
1218
+ "planned": 6,
1219
+ "attempted": 6,
1220
+ "completed": 6,
1221
+ "scored": 6,
1222
+ "passed": 4,
1223
+ "ungraded": 0,
1224
+ "not_run": 0,
1225
+ "incomplete": 0
1226
+ },
1227
+ {
1228
+ "benchmark": "MathVista",
1229
+ "planned": 5,
1230
+ "attempted": 5,
1231
+ "completed": 5,
1232
+ "scored": 5,
1233
+ "passed": 3,
1234
+ "ungraded": 0,
1235
+ "not_run": 0,
1236
+ "incomplete": 0
1237
+ },
1238
+ {
1239
+ "benchmark": "Q36-Agent-Function-Calling",
1240
+ "planned": 5,
1241
+ "attempted": 5,
1242
+ "completed": 5,
1243
+ "scored": 5,
1244
+ "passed": 0,
1245
+ "ungraded": 0,
1246
+ "not_run": 0,
1247
+ "incomplete": 0
1248
+ },
1249
+ {
1250
+ "benchmark": "Q36-Answer-Termination-No-Looping",
1251
+ "planned": 10,
1252
+ "attempted": 10,
1253
+ "completed": 10,
1254
+ "scored": 10,
1255
+ "passed": 10,
1256
+ "ungraded": 0,
1257
+ "not_run": 0,
1258
+ "incomplete": 0
1259
+ },
1260
+ {
1261
+ "benchmark": "Q36-Benign-Compliance-No-Overrefusal",
1262
+ "planned": 20,
1263
+ "attempted": 10,
1264
+ "completed": 10,
1265
+ "scored": 0,
1266
+ "passed": 0,
1267
+ "ungraded": 10,
1268
+ "not_run": 10,
1269
+ "incomplete": 0
1270
+ },
1271
+ {
1272
+ "benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
1273
+ "planned": 10,
1274
+ "attempted": 10,
1275
+ "completed": 10,
1276
+ "scored": 10,
1277
+ "passed": 10,
1278
+ "ungraded": 0,
1279
+ "not_run": 0,
1280
+ "incomplete": 0
1281
+ },
1282
+ {
1283
+ "benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
1284
+ "planned": 10,
1285
+ "attempted": 10,
1286
+ "completed": 10,
1287
+ "scored": 0,
1288
+ "passed": 0,
1289
+ "ungraded": 10,
1290
+ "not_run": 0,
1291
+ "incomplete": 0
1292
+ },
1293
+ {
1294
+ "benchmark": "Q36-Direct-Style-and-Personality",
1295
+ "planned": 10,
1296
+ "attempted": 10,
1297
+ "completed": 10,
1298
+ "scored": 0,
1299
+ "passed": 0,
1300
+ "ungraded": 10,
1301
+ "not_run": 0,
1302
+ "incomplete": 0
1303
+ },
1304
+ {
1305
+ "benchmark": "Q36-Hermes-Tool-Format",
1306
+ "planned": 5,
1307
+ "attempted": 5,
1308
+ "completed": 5,
1309
+ "scored": 5,
1310
+ "passed": 3,
1311
+ "ungraded": 0,
1312
+ "not_run": 0,
1313
+ "incomplete": 0
1314
+ },
1315
+ {
1316
+ "benchmark": "Q36-JSON-Schema",
1317
+ "planned": 5,
1318
+ "attempted": 5,
1319
+ "completed": 5,
1320
+ "scored": 5,
1321
+ "passed": 3,
1322
+ "ungraded": 0,
1323
+ "not_run": 0,
1324
+ "incomplete": 0
1325
+ },
1326
+ {
1327
+ "benchmark": "Q36-Legal-Alternatives-and-Boundaries",
1328
+ "planned": 10,
1329
+ "attempted": 10,
1330
+ "completed": 10,
1331
+ "scored": 0,
1332
+ "passed": 0,
1333
+ "ungraded": 10,
1334
+ "not_run": 0,
1335
+ "incomplete": 0
1336
+ },
1337
+ {
1338
+ "benchmark": "Q36-Output-Integrity",
1339
+ "planned": 5,
1340
+ "attempted": 5,
1341
+ "completed": 5,
1342
+ "scored": 5,
1343
+ "passed": 5,
1344
+ "ungraded": 0,
1345
+ "not_run": 0,
1346
+ "incomplete": 0
1347
+ },
1348
+ {
1349
+ "benchmark": "TruthfulQA",
1350
+ "planned": 10,
1351
+ "attempted": 10,
1352
+ "completed": 10,
1353
+ "scored": 10,
1354
+ "passed": 7,
1355
+ "ungraded": 0,
1356
+ "not_run": 0,
1357
+ "incomplete": 0
1358
+ }
1359
+ ]
1360
+ }
1361
+ ]
1362
+ }
LAPTOP-CAPABILITY-PLAN.md ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Laptop assistant: from quick tests to useful workflows
2
+
3
+ **Status: local preparation in progress; no laptop winner selected yet.** This is a deployment roadmap, not a claim that the full integration already works.
4
+
5
+ ## πŸ’» Coding and agent skills β€” what we will actually test
6
+
7
+ | Track | Cases | Success evidence | Current status |
8
+ |---|---:|---|---|
9
+ | HumanEval+ | 5 | Generated functions pass the selected task's base and extended tests in an isolated container | Existing outputs; execution/grading must be reconciled |
10
+ | MBPP+ | 5 | Correct results on base and extended tests, including edge cases | Existing outputs; execution/grading must be reconciled |
11
+ | LiveCodeBench | 9 | Program passes task tests within sandbox limits | Existing outputs; execution/grading must be reconciled |
12
+ | Q36 bug fixes | 5 | Reproduce a failing test, apply a patch, pass regression tests | New tasks to prepare |
13
+ | Q36 small app / website tasks | 5 | Build succeeds; required UI behavior passes browser checks | New tasks to prepare |
14
+ | Tool choice and arguments | 5 | Correct tool and schema-valid arguments; no invented tool result | Local smoke fixtures prepared separately |
15
+ | Multi-step planning | 5 | Respect dependencies and reach a verifiable final state | New interactive fixtures to prepare |
16
+ | Recovery from tool errors | 5 | Respond to a simulated failure, choose a valid alternative, verify recovery | New interactive fixtures to prepare |
17
+ | Memory and conflicting evidence | 5 | Retrieve the correct note, cite it, distinguish updated from stale facts | New isolated test vault to prepare |
18
+ | Stopping and task completion | 5 | Stop after success; report missing prerequisites instead of inventing completion | New interactive fixtures to prepare |
19
+ | Transfer to changed constraints | 5 | Solve held-out variants after a changed rule or environment | New tasks to prepare |
20
+
21
+ **64 case slots in this extension**, including 19 existing coding slots. These are compact diagnostics, not full leaderboard scores. The agent tasks measure useful components of general problem solving, **not proof of AGI**. Existing Hermes-format and JSON-schema diagnostics remain in the broad suite and are not silently counted twice.
22
+
23
+ ### How a capability earns a card entry
24
+
25
+ Freeze tasks and evaluators before comparing runtime settings. Count one primary attempt per case. Report **passes / executed-and-graded cases**, task coverage, failures and ungraded cases together. Publish results even when they are weak; only describe a workflow as demonstrated when its actual end-to-end task passes. A successful JSON tool-call test alone does not establish autonomous coding or desktop control.
26
+
27
+ Generated code runs in a disposable container without network access, host credentials or user-folder mounts, with CPU, RAM, process and time limits. Host tools and paid services are not available to benchmark prompts.
28
+
29
+ ## ⚑ Pick a laptop runtime
30
+
31
+ 1. Smoke-test **IQ4_XS, Q3_K_M, IQ2_M and IQ1_M**; exclude the BF16 server package. Start with IQ2_M to check loading and the harness.
32
+ 2. Compare **Ollama and llama.cpp**, each with a documented baseline and an optimized profile: 16 primary configurations. FreeToken is an additional experimental backend only if a compatible build actually runs.
33
+ 3. Use the same prompts, image inputs, context, output cap and thinking setting. Tune only on a separate warm-up set, then freeze settings.
34
+ 4. Record answer quality separately from prefill/decode tokens/s, end-to-end speed, time to first visible output, CPU/RAM/VRAM, temperatures, timeouts and incomplete outputs.
35
+ 5. Select a measured quality/performance/memory compromise, not simply the fastest token rate. An OOM or unsupported backend is a technical failure, not a wrong answer.
36
+
37
+ The target laptop has **32 GB RAM and an RTX 4060 Laptop GPU with 8 GB VRAM**. These 35B-weight models need CPU/RAM offload. MoE reduces active computation, not the need to store the expert weights.
38
+
39
+ ## πŸ› οΈ Hermes, routing and personal memory
40
+
41
+ After selecting the runtime, test it as a local endpoint behind the existing Hermes/router installation. Keep the previous route available for rollback. First use a dedicated test workspace and synthetic notes; connect the real Obsidian vault only after the test flow passes.
42
+
43
+ Useful scenarios to validate:
44
+
45
+ - **Code companion:** inspect a sample repository, fix one bug, run tests and summarize the patch.
46
+ - **Project memory:** find a note, cite its source and use its latest decision in a response.
47
+ - **Research notebook:** turn supplied material into a draft note with provenance, then ask before writing into the real vault.
48
+ - **Task recovery:** recognize an unavailable tool, ask for what is missing or use an explicitly allowed alternative.
49
+ - **Private local assistant:** use the configured local route and verify that external fallback is disabled for private inputs.
50
+
51
+ Obsidian storage, retrieval/indexing, memory updates and routing belong to the host application. They are not built into the model weights. Exact endpoint instructions and working configuration examples will be added after local validation; no production routing has been changed by this document.
README.md CHANGED
@@ -12,7 +12,7 @@ tags:
12
  - abliterated
13
  - heretic
14
  - uncensor
15
- - hermes
16
  - mtp
17
  - coding
18
  - tool-calling
@@ -46,6 +46,12 @@ Meet your local AI companion for ideas, code, images and conversation. **Qwen3.6
46
 
47
  These are things to try, not promises of perfect results. It can make mistakes or repeat itself; check important answers. Adult-oriented training is included (**18+**).
48
 
 
 
 
 
 
 
49
  ## πŸ•ΆοΈ Street knowledge. Business mind. Your style.
50
 
51
  The personality direction is **direct, sharp-witted and business-minded**: a street-smart conversation partner with humor, creative confidence and room for disagreement. The training mix includes internet culture, slang, practical business topics and personality-oriented conversations. Think less formal textbook, more an opinionated partner for brainstorming, writing and exploring alternatives.
@@ -148,6 +154,219 @@ Huihui performed better on the shared automatically assessable RTX 5090 subset;
148
 
149
  </details>
150
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
  ## πŸ“œ License, lineage & credits
152
 
153
  [Apache-2.0 license](LICENSE). Dataset sources and their own license information are linked in [DATASETS.md](DATASETS.md); model licensing does not relicense the source datasets. Thanks to the Qwen team, the inherited model and dataset authors, and the Soup, PEFT, Transformers, llama.cpp and Ollama projects.
@@ -156,6 +375,27 @@ Huihui performed better on the shared automatically assessable RTX 5090 subset;
156
 
157
  These names do not imply affiliation or a promise of unrestricted behavior. Full provenance and validation are retained, including the historical Opus4.7-labelled source. No universal superiority, guaranteed compliance or removal of memorization is claimed.
158
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
159
  ## πŸ”— Related collection
160
 
161
  The previous release remains separate: [Qwen3.6 Opus4.7 Heretic Hermes Agent β€” Editions](https://huggingface.co/collections/oktayd/qwen36-opus47-heretic-hermes-agent-editions-6a8d09f6c2eb42ed1b112184).
 
12
  - abliterated
13
  - heretic
14
  - uncensor
15
+ - hermes-lineage
16
  - mtp
17
  - coding
18
  - tool-calling
 
46
 
47
  These are things to try, not promises of perfect results. It can make mistakes or repeat itself; check important answers. Adult-oriented training is included (**18+**).
48
 
49
+ ## 🧠 A team of experts inside one model
50
+
51
+ **MoE means Mixture of Experts.** Think of a team of specialists: for each token, a router chooses which expert networks should contribute. This model has about **35 billion parameters in total**, with roughly **3 billion active per token**; its configuration selects **8 of 256 routed experts**.
52
+
53
+ That saves computation compared with activating every expert at once. It does **not** turn a 35B download into a 3B-sized model: the expert weights still need disk space and accessible RAM/VRAM. Quantized editions make local use more practical. The experts are learned networks, not separate installed apps or named profession-specific agents.
54
+
55
  ## πŸ•ΆοΈ Street knowledge. Business mind. Your style.
56
 
57
  The personality direction is **direct, sharp-witted and business-minded**: a street-smart conversation partner with humor, creative confidence and room for disagreement. The training mix includes internet culture, slang, practical business topics and personality-oriented conversations. Think less formal textbook, more an opinionated partner for brainstorming, writing and exploring alternatives.
 
154
 
155
  </details>
156
 
157
+ <!-- q36-capability-expansion:start -->
158
+ ## πŸ”Ž Broad benchmark β€” tests and passes
159
+
160
+ Expand a device to inspect every test family. **Passes / graded** use the original strict evaluator, including format-sensitive checks. Ungraded answers are not failures or passes. Incomplete is a separate delivery flag and can overlap with ungraded. These are small local subsets, not official leaderboard scores.
161
+
162
+ <details>
163
+ <summary>Q36 Β· H200 β€” per-test breakdown</summary>
164
+
165
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
166
+ |---|---:|---:|---:|---:|---:|
167
+ | ARC-Challenge | 10 / 10 | 8 / 8 | 2 | 2 | 0 |
168
+ | BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
169
+ | ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
170
+ | GPQA-Diamond | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
171
+ | GSM8K | 10 / 10 | 5 / 9 | 1 | 1 | 0 |
172
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
173
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 4 | 0 |
174
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
175
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
176
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
177
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
178
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
179
+ | MathVista | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
180
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
181
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
182
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 2 | 0 |
183
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
184
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
185
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
186
+ | Q36-Hermes-Tool-Format | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
187
+ | Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
188
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
189
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
190
+ | TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
191
+
192
+ </details>
193
+
194
+ <details>
195
+ <summary>Q36 Β· RTX 5090 β€” per-test breakdown</summary>
196
+
197
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
198
+ |---|---:|---:|---:|---:|---:|
199
+ | ARC-Challenge | 10 / 10 | 8 / 8 | 2 | 2 | 0 |
200
+ | BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
201
+ | ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
202
+ | GPQA-Diamond | 5 / 5 | 1 / 5 | 0 | 0 | 0 |
203
+ | GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
204
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 1 | 0 |
205
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
206
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
207
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
208
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
209
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
210
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
211
+ | MathVista | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
212
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
213
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 9 / 10 | 0 | 0 | 0 |
214
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 2 | 0 |
215
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
216
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
217
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
218
+ | Q36-Hermes-Tool-Format | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
219
+ | Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
220
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
221
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
222
+ | TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
223
+
224
+ </details>
225
+
226
+ <details>
227
+ <summary>Huihui Β· RTX 5090 β€” per-test breakdown</summary>
228
+
229
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
230
+ |---|---:|---:|---:|---:|---:|
231
+ | ARC-Challenge | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
232
+ | BBH | 23 / 23 | 16 / 23 | 0 | 0 | 0 |
233
+ | ChartQA | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
234
+ | GPQA-Diamond | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
235
+ | GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
236
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
237
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
238
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 5 | 0 |
239
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
240
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
241
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
242
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
243
+ | MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
244
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
245
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
246
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 0 | 0 |
247
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
248
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
249
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
250
+ | Q36-Hermes-Tool-Format | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
251
+ | Q36-JSON-Schema | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
252
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
253
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
254
+ | TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
255
+
256
+ </details>
257
+
258
+ <details>
259
+ <summary>Q36 Β· RTX 2000 Ada (CPU+GPU) β€” per-test breakdown</summary>
260
+
261
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
262
+ |---|---:|---:|---:|---:|---:|
263
+ | ARC-Challenge | 10 / 10 | 7 / 7 | 3 | 3 | 0 |
264
+ | BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
265
+ | ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
266
+ | GPQA-Diamond | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
267
+ | GSM8K | 10 / 10 | 8 / 10 | 0 | 0 | 0 |
268
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
269
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
270
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
271
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
272
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
273
+ | MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
274
+ | MMMU | 6 / 6 | 1 / 6 | 0 | 0 | 0 |
275
+ | MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
276
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
277
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 8 / 10 | 0 | 0 | 0 |
278
+ | Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 1 | 0 |
279
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 9 / 10 | 0 | 0 | 0 |
280
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
281
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
282
+ | Q36-Hermes-Tool-Format | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
283
+ | Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
284
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
285
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
286
+ | TruthfulQA | 10 / 10 | 6 / 10 | 0 | 0 | 0 |
287
+
288
+ </details>
289
+
290
+ <details>
291
+ <summary>Huihui Β· RTX 2000 Ada (CPU+GPU) β€” per-test breakdown</summary>
292
+
293
+ | Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
294
+ |---|---:|---:|---:|---:|---:|
295
+ | ARC-Challenge | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
296
+ | BBH | 11 / 23 | 6 / 10 | 1 | 1 | 12 |
297
+ | ChartQA | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
298
+ | GPQA-Diamond | 5 / 5 | 3 / 3 | 2 | 2 | 0 |
299
+ | GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
300
+ | HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
301
+ | IFEval | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
302
+ | LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 4 | 0 |
303
+ | MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 5 | 0 |
304
+ | MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
305
+ | MMLU-Pro | 1 / 1 | 0 / 0 | 1 | 1 | 0 |
306
+ | MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
307
+ | MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
308
+ | Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
309
+ | Q36-Answer-Termination-No-Looping | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
310
+ | Q36-Benign-Compliance-No-Overrefusal | 10 / 20 | 0 / 0 | 10 | 0 | 10 |
311
+ | Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
312
+ | Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
313
+ | Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
314
+ | Q36-Hermes-Tool-Format | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
315
+ | Q36-JSON-Schema | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
316
+ | Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
317
+ | Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
318
+ | TruthfulQA | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
319
+
320
+ </details>
321
+
322
+ [Machine-readable counts](BENCHMARK-TEST-COUNTS.json). Code generation is not a pass until the relevant execution tests have been graded.
323
+
324
+ ## 🏁 Hard benchmarks on the roadmap
325
+
326
+ All **25 families** below are registered from the [Qwen3.8-27B card](https://huggingface.co/Qwen/Qwen3.8-27B). The matching official adapters and datasets are not yet fully prepared, and **no local Qwen3.8 baseline has been run**. Existing short similarly named diagnostics do not substitute for those runs.
327
+
328
+ <details>
329
+ <summary>Coding, reasoning, agent and visual benchmarks β€” complete planned list</summary>
330
+
331
+ | Benchmark | Planned local cases | Preparation status |
332
+ |---|---:|---|
333
+ | Terminal Bench 2.1 (Terminus) | 5 | Adapter/data preparation pending |
334
+ | SWE-bench Pro | 5 | Adapter/data preparation pending |
335
+ | NL2Repo-Bench | 5 | Adapter/data preparation pending |
336
+ | DeepSWE 1.1 | 5 | Adapter/data preparation pending |
337
+ | QwenSWEBench | TBD | Internal release/access needed |
338
+ | CoWorkBench | TBD | Internal release/access needed |
339
+ | JobBench | 5 | Adapter/data preparation pending |
340
+ | Agents' Last Exam | 5 | Adapter/data preparation pending |
341
+ | IFBench | 20 | Adapter/data preparation pending |
342
+ | GPQA Diamond | 20 | Adapter/data preparation pending |
343
+ | HLE | 10 | Adapter/data preparation pending |
344
+ | LiveCodeBench v6 | 10 | Adapter/data preparation pending |
345
+ | OSWorld-Verified | 5 | Adapter/data preparation pending |
346
+ | WebArena-Verified | 5 | Adapter/data preparation pending |
347
+ | AndroidWorld | 5 | Adapter/data preparation pending |
348
+ | RecreationBench | TBD | Internal release/access needed |
349
+ | ClawEval-MM | 5 | Adapter/data preparation pending |
350
+ | SWE-MM | 5 | Adapter/data preparation pending |
351
+ | Vision2Web | 5 | Adapter/data preparation pending |
352
+ | MathVision | 10 | Adapter/data preparation pending |
353
+ | BabyVision | 10 | Adapter/data preparation pending |
354
+ | CharXiv (RQ) | 10 | Adapter/data preparation pending |
355
+ | OmniDocBench 1.5 | 10 | Adapter/data preparation pending |
356
+ | RealWorldQA | 20 | Adapter/data preparation pending |
357
+ | ERQA | 10 | Adapter/data preparation pending |
358
+
359
+ </details>
360
+
361
+ Comparison graphics will follow measured results, with our local samples and publisher-reported scores kept clearly separate. Different harnesses, budgets and trial counts will not be presented as a head-to-head win. [Full protocol and references](BENCHMARK-PLAN.md).
362
+
363
+ ## πŸ’» Coding, agents and your laptop
364
+
365
+ The next local checks cover executable coding tests, bug fixes, tool calls, planning, recovery, memory and stopping at the right time. **64 case slots** are specified, including 19 existing coding slots. They are compact skill diagnostics, not proof of AGI. Results and failures will both be reported; new capability claims require actual task-level evidence.
366
+
367
+ [πŸ› οΈ Laptop test plan, Hermes routing and Obsidian workflows](LAPTOP-CAPABILITY-PLAN.md). The runtime comparison and full memory integration are **in preparation**, not yet validated. The four device quickstarts above remain separate from these future end-to-end checks.
368
+ <!-- q36-capability-expansion:end -->
369
+
370
  ## πŸ“œ License, lineage & credits
371
 
372
  [Apache-2.0 license](LICENSE). Dataset sources and their own license information are linked in [DATASETS.md](DATASETS.md); model licensing does not relicense the source datasets. Thanks to the Qwen team, the inherited model and dataset authors, and the Soup, PEFT, Transformers, llama.cpp and Ollama projects.
 
375
 
376
  These names do not imply affiliation or a promise of unrestricted behavior. Full provenance and validation are retained, including the historical Opus4.7-labelled source. No universal superiority, guaranteed compliance or removal of memorization is claimed.
377
 
378
+ <details>
379
+ <summary>🧬 What Ablit, Heretic, Uncensor, Hermes, MTP and Vision mean</summary>
380
+
381
+ This release builds on the [previous project's model card](https://huggingface.co/oktayd/Q36-35B-A3B-Opus4.7-Ablit-Heretic-OBLITERATUS-Hermes-MTP-Vision-FT), which records the following stages. These are inherited stages, not new operations performed while packaging this release.
382
+
383
+ | Name / stage | What it contributes |
384
+ |---|---|
385
+ | **Qwen3.6 / MoE** | The underlying language-and-vision architecture and mixture-of-experts backbone. |
386
+ | **Reasoning-distilled lineage** | The earlier card traces a lordx64 derivative followed by the huihui-ai derivative. The historical Opus4.7 label describes inherited reasoning-distillation provenance, not inclusion of proprietary Claude weights. |
387
+ | **Ablit / abliterated** | The huihui-ai source underwent a weight-modification stage aimed at reducing refusal behavior. It is not a separate tool library. |
388
+ | **Heretic** | A subsequent custom, fused-MoE-aware modification stage documented by the previous release. It is distinct from the earlier abliteration. |
389
+ | **OBLITERATUS Nuclear / Uncensor** | Another separately recorded inherited modification stage aimed at reducing refusal behavior. **Uncensor** is the concise release-name label; OBLITERATUS remains part of the provenance. |
390
+ | **Hermes** | Tool-oriented supervised training: function-call structure, coding, terminal/file/repository workflows and multi-tool coordination, using [Hermes Function Calling](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) and [Hermes Agent reasoning traces](https://huggingface.co/datasets/lambda/hermes-agent-reasoning-traces). Your host application still supplies, authorizes and executes tools. |
391
+ | **v2 fine-tuning** | Three additional Soup/PEFT training runs with 34,000 record uses spanning knowledge, instructions, coding, personality and selected vision data. See [datasets and counts](DATASETS.md). |
392
+ | **MTP** | Preserved multi-token-prediction tensors. Preservation is not evidence that speculative decoding is enabled or faster in your runtime. |
393
+ | **Vision** | Image-input architecture; GGUF runtimes also need the matching projector. It does not itself supply browser control, memory or a 3D engine. |
394
+
395
+ The old card reports **23,220 training and 1,179 validation examples** for its own earlier SFT stage; those are separate from the current release's 34,000 record uses. It also records protection of 333 vision tensors and 19 MTP tensors in that earlier build. These historical checks are not new laptop benchmark results.
396
+
397
+ </details>
398
+
399
  ## πŸ”— Related collection
400
 
401
  The previous release remains separate: [Qwen3.6 Opus4.7 Heretic Hermes Agent β€” Editions](https://huggingface.co/collections/oktayd/qwen36-opus47-heretic-hermes-agent-editions-6a8d09f6c2eb42ed1b112184).
SHA256SUMS CHANGED
@@ -2,9 +2,11 @@
2
  807d54b88902798393fc104c3ca80d3e424d64270e8d1e1a34a44d4574f52c7e BENCHMARK-DIAGNOSTIC.json
3
  0e712dd8e4bc65088e170826e98bdc440fb19a2cfdb7613c643e9dabb624ecd8 BENCHMARK-EXTENSION-QWEN38.json
4
  67d070e3b12bcd185626054c81cdf4d528a9a1af26774050e1d50be370b5ee14 BENCHMARK-PLAN.md
 
5
  5c1a6fa0bd2d21e400b4c84001abec8085e3f5d7fbe5d217e7a0af7f32f42c6d DATASETS.md
6
  b483301314df6773850288b6291d49d082a7f816f7266d8cd62ef803b4f4e154 DEVICE-QUICKSTART.md
7
  5fa5bdc4687e5ed723af58eb436f545cc62589e5395d5041722639f03bb5ec9b LAPTOP-BENCHMARK-MATRIX.json
 
8
  ad3521db2dbcdf0b2ff300349d312c0fb44a1ebccc84571712602c512bf4a733 Qwen3.6-35B-v2-IQ1_M.gguf
9
  61a1a351bdc64ad14616277b0f7c630abe297733a29d96659da3bb7b369c769d Qwen3.6-35B-v2-IQ2_M.gguf
10
  80b45e93018023c6a9098cdee83f2a36ccc57dc490625cba2b783c6b3eb832cc Qwen3.6-35B-v2-IQ4_XS.gguf
@@ -12,7 +14,7 @@ ad3521db2dbcdf0b2ff300349d312c0fb44a1ebccc84571712602c512bf4a733 Qwen3.6-35B-v2
12
  3cc8cd0bad0c37fba8e038f68c089cdf2e5f55619d48487fdd9bfff0686ef830 Qwen3.6-35B-v2-Q4_K_M.gguf
13
  a3a5b8afaf96d6d4c9dfb257e54406a04a8363791f980647f788c67ff3966615 Qwen3.6-35B-v2-Q5_K_M.gguf
14
  71099584272a77b22c4aa1152271bf7d85595e1b869ffc883440a9d0e225c073 Qwen3.6-35B-v2-Q8_0.gguf
15
- 81ee951e2874a3bd16458ce27fea3288edd0a34da193720c15f50c276ee88dc3 README.md
16
  a69838e3728d2aa08941191fce1d7c5af6f0703a9a4444e8d6f0ef18cb4ecf8a RELEASE-NAMING.json
17
  d431e82d0ab961c261e525fc08fc3e64c0916fbd4d3f52b839b3d2eda0766338 assets/qwen3.6-35b-v2-banner.png
18
  fc5f718f531acdfe2e45bb09655d40f525eab939f9a3ab501287758f901cb3ad mmproj-Qwen3.6-35B-v2-F16.gguf
 
2
  807d54b88902798393fc104c3ca80d3e424d64270e8d1e1a34a44d4574f52c7e BENCHMARK-DIAGNOSTIC.json
3
  0e712dd8e4bc65088e170826e98bdc440fb19a2cfdb7613c643e9dabb624ecd8 BENCHMARK-EXTENSION-QWEN38.json
4
  67d070e3b12bcd185626054c81cdf4d528a9a1af26774050e1d50be370b5ee14 BENCHMARK-PLAN.md
5
+ cfd2a06c82f3f41ceaf8c333fb0ee0e38247d0c9419084c0501d5c2430bb7b68 BENCHMARK-TEST-COUNTS.json
6
  5c1a6fa0bd2d21e400b4c84001abec8085e3f5d7fbe5d217e7a0af7f32f42c6d DATASETS.md
7
  b483301314df6773850288b6291d49d082a7f816f7266d8cd62ef803b4f4e154 DEVICE-QUICKSTART.md
8
  5fa5bdc4687e5ed723af58eb436f545cc62589e5395d5041722639f03bb5ec9b LAPTOP-BENCHMARK-MATRIX.json
9
+ 42f57a1314669908fc20a4c657e32aafe4bedc9085508a31d1784cc3efd6c1c4 LAPTOP-CAPABILITY-PLAN.md
10
  ad3521db2dbcdf0b2ff300349d312c0fb44a1ebccc84571712602c512bf4a733 Qwen3.6-35B-v2-IQ1_M.gguf
11
  61a1a351bdc64ad14616277b0f7c630abe297733a29d96659da3bb7b369c769d Qwen3.6-35B-v2-IQ2_M.gguf
12
  80b45e93018023c6a9098cdee83f2a36ccc57dc490625cba2b783c6b3eb832cc Qwen3.6-35B-v2-IQ4_XS.gguf
 
14
  3cc8cd0bad0c37fba8e038f68c089cdf2e5f55619d48487fdd9bfff0686ef830 Qwen3.6-35B-v2-Q4_K_M.gguf
15
  a3a5b8afaf96d6d4c9dfb257e54406a04a8363791f980647f788c67ff3966615 Qwen3.6-35B-v2-Q5_K_M.gguf
16
  71099584272a77b22c4aa1152271bf7d85595e1b869ffc883440a9d0e225c073 Qwen3.6-35B-v2-Q8_0.gguf
17
+ 2a5701e733ab7b19f4b073fa13ff2e44d45eef7a54c6ad52679f940249031bab README.md
18
  a69838e3728d2aa08941191fce1d7c5af6f0703a9a4444e8d6f0ef18cb4ecf8a RELEASE-NAMING.json
19
  d431e82d0ab961c261e525fc08fc3e64c0916fbd4d3f52b839b3d2eda0766338 assets/qwen3.6-35b-v2-banner.png
20
  fc5f718f531acdfe2e45bb09655d40f525eab939f9a3ab501287758f901cb3ad mmproj-Qwen3.6-35B-v2-F16.gguf