| # Full Swift Mac cache validation — 2026-09-25 |
|
|
| Both complete, existing Swift checkpoints passed native Metal text generation |
| and two cached follow-up requests on an AWS M4 Pro Mac with 48 GiB RAM. |
| All weight shards were verified against their recorded SHA-256 hashes before |
| loading. No quantization, tensor edits, context changes or manual memory-limit overrides |
| were performed. Hub offline mode was enabled during testing. |
|
|
| The input is our own synthetic repeated-record conversation, followed by a short |
| request to reply READY. It is a memory/cache regression test, not a quality |
| benchmark, an exact reproduction of another person's conversation, or a claim |
| that every context fits. The follow-ups must reuse at least half the prompt to |
| pass; the test fails if the worker dies or the retained-cache budget is exceeded. |
|
|
| | Checkpoint | Initial prompt tokens | First request, seconds | Follow-ups, seconds | Follow-up cached tokens | Peak MLX allocation, GiB | |
| |---|---:|---:|---|---|---:| |
| | 4-bit | 86,004 | 893.64 | 1.46, 1.42 | 86000, 86020 | 31.57 | |
| | 5-bit | 86,004 | 907.00 | 1.45, 1.45 | 86000, 86020 | 34.79 | |
|
|
| The first request includes model loading and initial prompt processing. MLX peak |
| allocation is not total process or system memory. Complete request results, |
| generated replies, cache limits and sampled swap observations are recorded in |
| `compatibility/cache-tests/full-mac-4bit-results.json` and |
| `compatibility/cache-tests/full-mac-5bit-results.json`. |
|
|
| Environment: macOS 26.7 (25G229), Python 3.12.13, MLX 0.32.2, MLX-LM 0.32.0, Transformers 5.14.1, |
| Hugging Face Hub 1.31.0. Official MLX-LM base revision: |
| `c69d1288440a0dc4e6401fc417098b07598dccd5`. |
| The shared runtime uses the existing Swift architecture patch, its 5-bit support |
| extension, and the unchanged server-cache patch. The 5-bit extension also accepts |
| the original 4-bit format. The cache defaults are two retained entries, automatic |
| byte budgeting, one prompt/decode stream, and 512-token prefill steps. |
|
|
| To reproduce, install the pinned environment and all three patches in the order |
| documented by the 5-bit release, then use a complete local snapshot. Run one model |
| at a time on an Apple Silicon Mac with at least 48 GiB RAM: |
|
|
| ```bash |
| HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python \ |
| Swift-1.5-4bit-MLX/compatibility/cache-tests/validate_full_mac_cache.py \ |
| --snapshot Swift-1.5-4bit-MLX --output full-mac-4bit-results |
| HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python \ |
| Swift-1.5-5bit-MLX/compatibility/cache-tests/validate_full_mac_cache.py \ |
| --snapshot Swift-1.5-5bit-MLX --output full-mac-5bit-results |
| ``` |
|
|
| Output directories must be new. The harness uses the real server and complete |
| released weights, without building or quantizing any synthetic model. It suppresses |
| the server startup helper's wired-limit call. The unchanged stock BatchGenerator |
| still calls MLX's set_wired_limit with Apple's recommended working-set size; |
| macOS memory settings are not changed. The legacy result field |
| `memory_limit_overrides=false` denotes no manual override, not suppression of this |
| normal batch-generator behavior. The actual Metal device is recorded in the results. |
|
|
| These results establish the tested text workload on the stated 48 GiB machine. |
| They do not establish 24 GiB operation, arbitrary 262k-token workloads, GUI/plugin |
| integration, image/video chat, speculative MTP generation, or broad output quality. |
| Existing installations still need to apply the runtime patch and restart the server. |
| The recorded checks are independently executed tests, not a hosted CI status. |
|
|
