evaluation/report.md

Image21-MLX informal evaluation

One seed per case, one machine, same pinned MLX runtime. Sequential BF16-then-Q8 desktop run without repeated or interleaved trials; order, thermal state and other applications may affect timing. Pixel similarity is not a perceptual quality score; no cross-runtime parity claim.

Case	BF16 s	Q8 s	BF16 peak GiB	Q8 peak GiB	RGB PSNR dB
portrait	440.91	568.86	16.41	16.19	42.03
english_text	464.32	566.88	16.43	16.20	19.89
chinese_text	474.14	567.43	16.44	16.20	36.97
composition	480.91	562.52	16.44	16.20	37.17
texture	487.64	445.07	16.43	16.20	38.41
rgba	485.45	557.30	16.42	16.20	29.81
edit	606.29	665.00	18.68	16.20	51.09

MLX peak allocated memory excludes the OS, other applications and some process allocations. It is not minimum machine RAM.

Timing includes prompt encoding, phase-by-phase component loading, denoising and VAE decoding; PNG writing is excluded. The after-image RSS field is a snapshot after component release, not a peak.

Existing system swap is recorded separately from per-image swap change. This desktop-session run leaves other applications open. See system-context.json and the per-model environment.json files.

All editing pairs use the same BF16 portrait input. Visual findings and release limitations are in visual-review.json.

Visual review / 视觉检查

逐对检查七类样本，未发现明显严重的量化退化：中英文文字均正确，杯子数量与顺序、毛衣改色及真实透明通道均保留。英文海报的字体和台灯结构有可见变化，龙贴纸的表情与细节也有变化，其余样本更接近。两种精度的人物图都比提示词要求裁得更紧，透明图均有少量接近零的背景 alpha 残留。这是助手进行的非盲法、每类单种子视觉检查，不代表无损或统计等价。

Assistant inspection of all seven paired cases found no obvious severe quality regression in this small sample. English and Chinese text are correct in both precisions; cup count/order and sweater recoloring succeed; real RGBA transparency is retained. The English poster changes visibly in typeface and lamp structure, and the dragon changes expression/details. Other pairs are closer. Both portraits are cropped more tightly than requested, and both transparency samples contain slight near-zero background alpha. This is an unblinded, single-seed visual check, not evidence of lossless quantization or statistical equivalence.

• portrait: Very close to BF16 in facial features, framing, silver hair, blue knit sweater and window light. Small local skin/hair/knit texture changes are visible, without an obvious severe degradation. Same chest-up rather than waist-up crop limitation.
• english_text: Exact, legible CREATE WITH LIGHT and September 2026; no extra text. Compared with BF16 the serif typeface/underline and lamp stem/base change visibly, but both remain coherent and satisfy the prompt. This demonstrates non-identical generation, not an obvious readability regression.
• chinese_text: Both Chinese lines and comma are correct and legible, matching BF16. Layout, cream backdrop and cup placement remain very close; small latte/foam texture variations are visible. No obvious text accuracy regression in this sample.
• composition: Exactly three cups in the requested red/blue/yellow order and one green apple in front of the blue cup. Overall layout is very close to BF16. Minor handle/shadow geometry changes are visible; no counting or spatial-order regression.
• texture: Very close to BF16: coherent bird anatomy, fine blue/orange feather detail, wet moss and droplets. Minor local texture/brightness changes without obvious smoothing or broken detail.
• rgba: Full-body green dragon with orange wings and white sticker outline. Expression and surface details change visibly from BF16 while silhouette and requested content remain coherent. 60.69% of pixels have alpha exactly zero (BF16 60.47%); both have canvas-edge alpha at most 4/255. No loss of actual transparency; neither output is a perfectly binary matte.
• edit: Successful red sweater recoloring with facial features, hair, pose and window background very close to the BF16 edit. Minor local texture changes; no obvious edit-consistency regression. Neither output guarantees pixel-identical preservation outside the garment.