File size: 3,562 Bytes
c62a35c 80896f2 c62a35c 80896f2 c62a35c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # Full Swift Mac cache validation — 2026-09-25
Both complete, existing Swift checkpoints passed native Metal text generation
and two cached follow-up requests on an AWS M4 Pro Mac with 48 GiB RAM.
All weight shards were verified against their recorded SHA-256 hashes before
loading. No quantization, tensor edits, context changes or manual memory-limit overrides
were performed. Hub offline mode was enabled during testing.
The input is our own synthetic repeated-record conversation, followed by a short
request to reply READY. It is a memory/cache regression test, not a quality
benchmark, an exact reproduction of another person's conversation, or a claim
that every context fits. The follow-ups must reuse at least half the prompt to
pass; the test fails if the worker dies or the retained-cache budget is exceeded.
| Checkpoint | Initial prompt tokens | First request, seconds | Follow-ups, seconds | Follow-up cached tokens | Peak MLX allocation, GiB |
|---|---:|---:|---|---|---:|
| 4-bit | 86,004 | 893.64 | 1.46, 1.42 | 86000, 86020 | 31.57 |
| 5-bit | 86,004 | 907.00 | 1.45, 1.45 | 86000, 86020 | 34.79 |
The first request includes model loading and initial prompt processing. MLX peak
allocation is not total process or system memory. Complete request results,
generated replies, cache limits and sampled swap observations are recorded in
`compatibility/cache-tests/full-mac-4bit-results.json` and
`compatibility/cache-tests/full-mac-5bit-results.json`.
Environment: macOS 26.7 (25G229), Python 3.12.13, MLX 0.32.2, MLX-LM 0.32.0, Transformers 5.14.1,
Hugging Face Hub 1.31.0. Official MLX-LM base revision:
`c69d1288440a0dc4e6401fc417098b07598dccd5`.
The shared runtime uses the existing Swift architecture patch, its 5-bit support
extension, and the unchanged server-cache patch. The 5-bit extension also accepts
the original 4-bit format. The cache defaults are two retained entries, automatic
byte budgeting, one prompt/decode stream, and 512-token prefill steps.
To reproduce, install the pinned environment and all three patches in the order
documented by the 5-bit release, then use a complete local snapshot. Run one model
at a time on an Apple Silicon Mac with at least 48 GiB RAM:
```bash
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python \
Swift-1.5-4bit-MLX/compatibility/cache-tests/validate_full_mac_cache.py \
--snapshot Swift-1.5-4bit-MLX --output full-mac-4bit-results
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python \
Swift-1.5-5bit-MLX/compatibility/cache-tests/validate_full_mac_cache.py \
--snapshot Swift-1.5-5bit-MLX --output full-mac-5bit-results
```
Output directories must be new. The harness uses the real server and complete
released weights, without building or quantizing any synthetic model. It suppresses
the server startup helper's wired-limit call. The unchanged stock BatchGenerator
still calls MLX's set_wired_limit with Apple's recommended working-set size;
macOS memory settings are not changed. The legacy result field
`memory_limit_overrides=false` denotes no manual override, not suppression of this
normal batch-generator behavior. The actual Metal device is recorded in the results.
These results establish the tested text workload on the stated 48 GiB machine.
They do not establish 24 GiB operation, arbitrary 262k-token workloads, GUI/plugin
integration, image/video chat, speculative MTP generation, or broad output quality.
Existing installations still need to apply the runtime patch and restart the server.
The recorded checks are independently executed tests, not a hosted CI status.
|