Bound retained server cache while keeping prompt reuse enabled

The pinned server constructed its retained-cache LRU without the configured byte limit. Entries saved during prefill or completion could therefore exceed `--prompt-cache-bytes` after the earlier request-admission trim. This patch enforces the retention limit on each insertion and keeps prompt reuse enabled with bounded defaults.

The Swift server profile defaults to two retained entries, one prompt and one decode stream, and 512-token prefill steps. A Metal memory estimate sets the default retained-cache budget; an explicit byte limit overrides it. The budget excludes active-request and total process memory, so this is not a promise that every context length fits.

Validation completed on clean copies of the pinned source for both architecture patch stacks:

- PASS: 44 unit/upstream server tests per patch stack.
- PASS: 36 offline HTTP requests per format, including warm cache reuse, explicit byte-cap enforcement, streaming, sequential generation and disabled retention.
- PASS: clean patch application, runtime source identity, unchanged model implementation, reverse-apply check, Black 25.1.0, isort 6.0.0 and Ruff 0.16.6.

These are reproducible local Apple Silicon checks, not hosted CI statuses. Results and the offline regression harness are included under `compatibility/cache-tests/`. They use tiny random model fixtures; full Swift long-context capacity remains unverified by this change.

Only runtime patch, tests, instructions and file manifest change. Checkpoint weights, quantization, architecture code and tokenizer assets are unchanged. Existing installations must apply the patch and restart; fetching model files alone does not update an installed server. `SERVER_CACHE_UPDATE.md` includes installation, offline startup, validation and rollback instructions.

This PR is for review and has not been merged into main.

SERVER_CACHE_UPDATE.md

@@ -0,0 +1,93 @@
+# Server cache update
+
+This update changes the pinned MLX-LM server, not the checkpoint. It is a Swift
+runtime patch based on official MLX-LM `c69d1288440a0dc4e6401fc417098b07598dccd5`, not an upstream release.
+The existing architecture patch remains required.
+
+## Apply to an existing installation
+
+Stop the server. Activate the same Python environment used for serving. From the
+directory containing `swift5-mlx-lm` and `Swift-1.5-5bit-MLX`, after downloading this patch:
+
+```bash
+git -C swift5-mlx-lm rev-parse HEAD
+git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
+git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
+python -m pip install --no-deps -e ./swift5-mlx-lm
+python Swift-1.5-5bit-MLX/compatibility/cache-tests/verify_server_patch.py
+```
+
+The revision must equal the pinned revision above. Stop if the patch check fails;
+do not force it onto different server code or apply it twice. For a fresh install,
+follow [USAGE.md](USAGE.md), which applies the architecture patches first.
+The server verifier checks the active Python installation without loading weights.
+
+Once dependencies and the complete checkpoint are already local, startup works
+without Hub access:
+
+```bash
+HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m mlx_lm.server \
+  --model ./Swift-1.5-5bit-MLX --host 127.0.0.1 --port 8080
+```
+
+Remove an earlier `--prompt-cache-size 0` override to enable reuse. Explicit
+arguments override the new defaults. Installing/downloading prerequisites still
+requires internet; offline operation requires those prerequisites already present.
+
+## Behavior and limits
+
+Defaults: two retained cache entries, prompt concurrency 1, decode concurrency 1,
+and prefill steps of 512 tokens. The byte cap is now passed to the LRU cache, so
+every insertion enforces it. Previously the CLI byte limit only triggered a trim
+when adding a batched request; later retained entries could exceed that setting.
+
+The automatic Metal budget uses the reported recommended working-set size `R`,
+the loaded model parameter bytes `W`, and workspace reserve `max(4 GiB, R/8)`.
+It budgets half the remaining space for retained entries:
+`max(0, (R - W - reserve) / 2)`. This is a conservative estimate, not a guarantee
+that an arbitrary active context fits. It does not change macOS or MLX limits.
+When device metadata is unavailable, the entry limit still applies; supply an
+explicit byte limit if needed. `--prompt-cache-bytes 2GB`, for example, sets a
+retention budget of 2,000,000,000 bytes; it is not a universal recommended value.
+
+The active request, temporary tensors, allocator and other apps use additional
+memory. Cache byte accounting is logical and may include shared buffers. Entries
+that cannot fit are evicted whole; the client's input history is not truncated.
+Reuse can therefore vary with context size. This patch does not add KV
+quantization, change attention or alter any checkpoint tensors.
+
+## Reproduce the checks
+
+Use the patched environment with Python 3.12, MLX 0.32.2 and MLX-LM 0.32.0.
+`validation.json` and the two `offline-*-results.json` files record our local
+results. No hosted CI status is claimed. The tests use random tiny models and the
+local tokenizer; they never open the released Swift weight shards.
+
+```bash
+python -m pip install 'pytest==9.1.1' 'requests==2.32.5'
+python -m pytest -q swift5-mlx-lm/tests/test_server_cache_budget.py swift5-mlx-lm/tests/test_server.py
+python Swift-1.5-5bit-MLX/compatibility/cache-tests/test_server_cache_http.py \
+  --assets ./Swift-1.5-5bit-MLX \
+  --config Swift-1.5-5bit-MLX/compatibility/cache-tests/synthetic-config.json \
+  --bits 5 --output cache-http-results.json
+```
+
+The upstream server suite may fetch its small upstream test model if not already
+cached. The separate HTTP harness sets offline mode and needs only local assets.
+It covers 12 requests each with automatic budgeting, an explicit byte limit, and
+disabled retention; streaming and seeded sequential execution are included.
+Synthetic output is deliberately fixed for transport/cache checks, not quality.
+Full Swift long-context execution on a larger Mac has not been independently
+retested by this change. GUI runtimes need their own integration.
+
+## Roll back this runtime patch
+
+Stop the server, then reverse only this patch and restart with explicit settings:
+
+```bash
+git -C swift5-mlx-lm apply --reverse --check ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
+git -C swift5-mlx-lm apply --reverse ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
+```
+
+This leaves the architecture patch and model files intact. An editable install
+uses the source change after restart; reinstall if a non-editable copy was used.

TROUBLESHOOTING.md

@@ -43,35 +43,28 @@ separate runtime. GUI integration has not been validated for these full checkpoi
 Do not remove vision/MTP parameters or change the config to force a stock text
 loader to accept the file. All 2,379 saved tensor entries belong to this release.
 
-## Server settings for limited memory
-
-The pinned MLX-LM server retains prompt caches for reuse by default. For a setup
-that favors lower memory use over reuse speed, stop the existing server and start
-it from the same patched Python environment with:
-
-```bash
-mlx_lm.server --model ukisai/Swift-1.5-5bit-MLX \
-  --host 127.0.0.1 --port 8080 \
-  --prompt-cache-size 0 \
-  --prompt-concurrency 1 --decode-concurrency 1 \
-  --prefill-step-size 512
-```
-
-This disables retained prompt caches, limits concurrency to one, and processes
-prefill in smaller chunks. The model weights and their quantization are unchanged.
-The active request still needs its own cache and temporary memory; these options
-do not set a total process-memory cap or guarantee that any context length fits.
-
-Disabling reuse can slow later turns because their history must be processed
-again. It does not remove history that the client includes in the next request.
-Start with a fresh short conversation and increase history while monitoring RAM.
-The model's configured context limit is distinct from the memory needed to run it.
-
-Validation scope: three successive HTTP requests, including a streaming request,
-passed on each small synthetic 4-bit and 5-bit architecture with these settings.
-Retained-cache accounting stayed at zero. These tests did not load the complete
-Swift checkpoint or establish full-model long-context capacity on any Mac.
-No macOS or MLX memory-limit overrides were used in those tests.
+## Server cache update
+
+The supplied server patch keeps prompt reuse enabled with at most two retained
+entries, enforces a retained-cache byte budget at insertion, and defaults to one
+prompt and one decode stream with 512-token prefill steps. On Metal the default
+budget estimates headroom after model weights and workspace; an explicit
+`--prompt-cache-bytes` overrides it. `--prompt-cache-size 2` means two entries,
+not two GB.
+
+See [SERVER_CACHE_UPDATE.md](SERVER_CACHE_UPDATE.md) for installation, offline
+startup, validation and rollback. Existing installations must apply this runtime
+update and restart. No model-weight download or re-quantization is needed.
+
+The retained-cache limit does not cap active-request or total process memory.
+An entry larger than the budget is evicted, so reuse depends on what fits. Use
+`--prompt-cache-size 0` only when intentionally disabling reuse. The model's
+configured context length is not a promise that it fits every Mac.
+
+Validation: 44 unit/upstream server tests and 72 offline HTTP requests passed
+across tiny 4-bit and 5-bit fixtures. These checks cover retention, warm reuse,
+streaming and sequential generation. They do not establish full-model
+long-context capacity or GUI integration.
 
 ## 3. If a server reports `404 generation thread died`
 

UPLOAD_MANIFEST.json

@@ -36,17 +36,20 @@
       "sha256": "e0ce3327646a3dde90ee001a3bb819c8cb24e550a83d28340671f2542e7ad55b",
       "git_blob_sha1": "008e4ab150bc188e372cad12fda89317afda850e"
     },
+    {
+      "path": "SERVER_CACHE_UPDATE.md",
+      "bytes": 4765,
+      "sha256": "31ae8115dcf0ccba9e91dd03f9eab30dc56a6a752fa26955c05555cc655bdcd7"
+    },
     {
       "path": "TROUBLESHOOTING.md",
-      "bytes": 5865,
-      "sha256": "d94c7c741398c55b8f9abaa1f1542d7046dc6fab1df7e9ea4b13b59af3945dee",
-      "git_blob_sha1": "3af3c505aff07e4e527e1d2d01b7b8f4cdb5288f"
+      "bytes": 5623,
+      "sha256": "8d631dcdacfd6f98d0ac37c4d2532f1a8139ab2e336c8ae27bfe03c251444327"
     },
     {
       "path": "USAGE.md",
-      "bytes": 6912,
-      "sha256": "5c401d9f19fdd22c7bdc495bfdc925bc4ef6f9f829827a22918e26e0ba67d0e4",
-      "git_blob_sha1": "8532eaa9e4c661635164a08e3fb00c597191b1a0"
+      "bytes": 6858,
+      "sha256": "946ba223dd2a7cf1aceb069cf4fd82f4bf609366abacba61017f08e2bf9a29b4"
     },
     {
       "path": "chat_template.jinja",
@@ -66,6 +69,36 @@
       "sha256": "ccfab7ccb2ea306f71531c8ca77bb55507606cd90768b1e32b8b52ab5b48cf01",
       "git_blob_sha1": "98ff47b9ef9d4ac9f1a4bde3db13dd27456c5ea1"
     },
+    {
+      "path": "compatibility/cache-tests/offline-4bit-results.json",
+      "bytes": 5014,
+      "sha256": "ddcfa5f200aa57d1cdadc6225299329b592e858bea2502e9bf07d313f4cf149a"
+    },
+    {
+      "path": "compatibility/cache-tests/offline-5bit-results.json",
+      "bytes": 5014,
+      "sha256": "f50ca1c21eafc43623f1461999e33e99b9e684a990d926aa77fbe59cb919fe25"
+    },
+    {
+      "path": "compatibility/cache-tests/synthetic-config.json",
+      "bytes": 1814,
+      "sha256": "852fa793054753245559ffeae66d4367b6009f3ebdb7ca611747ec7564d785c7"
+    },
+    {
+      "path": "compatibility/cache-tests/test_server_cache_http.py",
+      "bytes": 9689,
+      "sha256": "e7aff50b441145338575a50c7abc5fb6d40fdac3d11bbf6433d909ecfe22483b"
+    },
+    {
+      "path": "compatibility/cache-tests/validation.json",
+      "bytes": 1984,
+      "sha256": "288450e5a3b34d4c45ce16489de749f49c7bd03cd2d28d4940d4bd94953022a5"
+    },
+    {
+      "path": "compatibility/cache-tests/verify_server_patch.py",
+      "bytes": 504,
+      "sha256": "4c3acf33b4b9ffe0131c45ad0584785767a81cf455f9f60d9ec814e4b7eaf61a"
+    },
     {
       "path": "compatibility/conversion-result.json",
       "bytes": 694,
@@ -126,6 +159,11 @@
       "sha256": "f6f1d0bdafa45863bfbf93dac0398c481c993ea04fdf38b9bae98c643f89eaec",
       "git_blob_sha1": "b739acc2e3b4cb82f073b7280255bcb898bb2a33"
     },
+    {
+      "path": "compatibility/swift15-server-cache.patch",
+      "bytes": 9356,
+      "sha256": "a7fbc0f0524ee7864d9f41a98a2e35acf0e63f53e369ee5b9a87ca12926a9eab"
+    },
     {
       "path": "config.json",
       "bytes": 4577,
@@ -218,5 +256,7 @@
   ],
   "file_count": 37,
   "total_bytes": 19310880490,
-  "self_excluded": true
+  "self_excluded": true,
+  "total_bytes_excluding_manifest": 19310918334,
+  "server_cache_update": "2026-09-25: bounded cache reuse runtime patch and recorded local tests; weights unchanged"
 }

USAGE.md

@@ -38,6 +38,8 @@ git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/swift15-m
 git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/swift15-mlx-lm.patch
 git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/enable-5bit.patch
 git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/enable-5bit.patch
+git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
+git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
 ```
 
 Apple Silicon:
@@ -102,35 +104,28 @@ The explicit `model.mtp_logits` step and `model.visual` encoder have historical
 component evidence. Integrated image/video chat and speculative generation are
 not implemented; unsupported multimodal generation must not be reported as working.
 
-## Server settings for limited memory
+## Server cache update
 
-The pinned MLX-LM server retains prompt caches for reuse by default. For a setup
-that favors lower memory use over reuse speed, stop the existing server and start
-it from the same patched Python environment with:
+The supplied server patch keeps prompt reuse enabled with at most two retained
+entries, enforces a retained-cache byte budget at insertion, and defaults to one
+prompt and one decode stream with 512-token prefill steps. On Metal the default
+budget estimates headroom after model weights and workspace; an explicit
+`--prompt-cache-bytes` overrides it. `--prompt-cache-size 2` means two entries,
+not two GB.
 
-```bash
-mlx_lm.server --model ukisai/Swift-1.5-5bit-MLX \
-  --host 127.0.0.1 --port 8080 \
-  --prompt-cache-size 0 \
-  --prompt-concurrency 1 --decode-concurrency 1 \
-  --prefill-step-size 512
-```
+See [SERVER_CACHE_UPDATE.md](SERVER_CACHE_UPDATE.md) for installation, offline
+startup, validation and rollback. Existing installations must apply this runtime
+update and restart. No model-weight download or re-quantization is needed.
+
+The retained-cache limit does not cap active-request or total process memory.
+An entry larger than the budget is evicted, so reuse depends on what fits. Use
+`--prompt-cache-size 0` only when intentionally disabling reuse. The model's
+configured context length is not a promise that it fits every Mac.
 
-This disables retained prompt caches, limits concurrency to one, and processes
-prefill in smaller chunks. The model weights and their quantization are unchanged.
-The active request still needs its own cache and temporary memory; these options
-do not set a total process-memory cap or guarantee that any context length fits.
-
-Disabling reuse can slow later turns because their history must be processed
-again. It does not remove history that the client includes in the next request.
-Start with a fresh short conversation and increase history while monitoring RAM.
-The model's configured context limit is distinct from the memory needed to run it.
-
-Validation scope: three successive HTTP requests, including a streaming request,
-passed on each small synthetic 4-bit and 5-bit architecture with these settings.
-Retained-cache accounting stayed at zero. These tests did not load the complete
-Swift checkpoint or establish full-model long-context capacity on any Mac.
-No macOS or MLX memory-limit overrides were used in those tests.
+Validation: 44 unit/upstream server tests and 72 offline HTTP requests passed
+across tiny 4-bit and 5-bit fixtures. These checks cover retention, warm reuse,
+streaming and sequential generation. They do not establish full-model
+long-context capacity or GUI integration.
 
 ## Conversion provenance
 

compatibility/cache-tests/offline-4bit-results.json

@@ -0,0 +1,222 @@
+{
+  "scope": "Tiny random weights with real local tokenizer assets. No released model weights loaded or changed; no full-model capacity claim.",
+  "bits": 4,
+  "offline_hub": true,
+  "memory_limit_overrides": false,
+  "machine": "arm64",
+  "python": "3.12.13",
+  "packages": {
+    "mlx": "0.32.2",
+    "mlx-lm": "0.32.0",
+    "transformers": "5.14.1",
+    "huggingface_hub": "1.31.0"
+  },
+  "profiles": [
+    {
+      "name": "defaults",
+      "requests": [
+        {
+          "http_status": 200,
+          "prompt_tokens": 225,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 246,
+          "cached_tokens": 221
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 267,
+          "cached_tokens": 242
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 288,
+          "cached_tokens": 263
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 309,
+          "cached_tokens": 284
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 330,
+          "cached_tokens": 305
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 351,
+          "cached_tokens": 326
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 372,
+          "cached_tokens": 347
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 393,
+          "cached_tokens": 368
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 414,
+          "cached_tokens": 389
+        },
+        {
+          "http_status": 200,
+          "stream_complete": true
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 457,
+          "cached_tokens": 431
+        }
+      ],
+      "byte_limit": 4190923688,
+      "peak_retained_bytes": 246528,
+      "retained_sequences": 2,
+      "insertions": 35
+    },
+    {
+      "name": "explicit_bytes",
+      "requests": [
+        {
+          "http_status": 200,
+          "prompt_tokens": 225,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 246,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 267,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 288,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 309,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 330,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 351,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 372,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 393,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 414,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "stream_complete": true
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 457,
+          "cached_tokens": 112
+        }
+      ],
+      "byte_limit": 100000,
+      "peak_retained_bytes": 82432,
+      "retained_sequences": 1,
+      "insertions": 40
+    },
+    {
+      "name": "no_retention",
+      "requests": [
+        {
+          "http_status": 200,
+          "prompt_tokens": 225,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 246,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 267,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 288,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 309,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 330,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 351,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 372,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 393,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 414,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "stream_complete": true
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 457,
+          "cached_tokens": 0
+        }
+      ],
+      "byte_limit": 4190923688,
+      "peak_retained_bytes": 0,
+      "retained_sequences": 0,
+      "insertions": 45
+    }
+  ],
+  "peak_mlx_bytes": 338846824,
+  "status": "PASS"
+}

compatibility/cache-tests/offline-5bit-results.json

@@ -0,0 +1,222 @@
+{
+  "scope": "Tiny random weights with real local tokenizer assets. No released model weights loaded or changed; no full-model capacity claim.",
+  "bits": 5,
+  "offline_hub": true,
+  "memory_limit_overrides": false,
+  "machine": "arm64",
+  "python": "3.12.13",
+  "packages": {
+    "mlx": "0.32.2",
+    "mlx-lm": "0.32.0",
+    "transformers": "5.14.1",
+    "huggingface_hub": "1.31.0"
+  },
+  "profiles": [
+    {
+      "name": "defaults",
+      "requests": [
+        {
+          "http_status": 200,
+          "prompt_tokens": 225,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 246,
+          "cached_tokens": 221
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 267,
+          "cached_tokens": 242
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 288,
+          "cached_tokens": 263
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 309,
+          "cached_tokens": 284
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 330,
+          "cached_tokens": 305
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 351,
+          "cached_tokens": 326
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 372,
+          "cached_tokens": 347
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 393,
+          "cached_tokens": 368
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 414,
+          "cached_tokens": 389
+        },
+        {
+          "http_status": 200,
+          "stream_complete": true
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 457,
+          "cached_tokens": 431
+        }
+      ],
+      "byte_limit": 4186895080,
+      "peak_retained_bytes": 246528,
+      "retained_sequences": 2,
+      "insertions": 35
+    },
+    {
+      "name": "explicit_bytes",
+      "requests": [
+        {
+          "http_status": 200,
+          "prompt_tokens": 225,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 246,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 267,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 288,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 309,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 330,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 351,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 372,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 393,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 414,
+          "cached_tokens": 112
+        },
+        {
+          "http_status": 200,
+          "stream_complete": true
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 457,
+          "cached_tokens": 112
+        }
+      ],
+      "byte_limit": 100000,
+      "peak_retained_bytes": 82432,
+      "retained_sequences": 1,
+      "insertions": 40
+    },
+    {
+      "name": "no_retention",
+      "requests": [
+        {
+          "http_status": 200,
+          "prompt_tokens": 225,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 246,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 267,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 288,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 309,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 330,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 351,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 372,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 393,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 414,
+          "cached_tokens": 0
+        },
+        {
+          "http_status": 200,
+          "stream_complete": true
+        },
+        {
+          "http_status": 200,
+          "prompt_tokens": 457,
+          "cached_tokens": 0
+        }
+      ],
+      "byte_limit": 4186895080,
+      "peak_retained_bytes": 0,
+      "retained_sequences": 0,
+      "insertions": 45
+    }
+  ],
+  "peak_mlx_bytes": 430070044,
+  "status": "PASS"
+}

compatibility/cache-tests/synthetic-config.json

@@ -0,0 +1,71 @@
+{
+  "model_type": "qwen3_5",
+  "architectures": [
+    "Qwen3_5ForConditionalGeneration"
+  ],
+  "language_model_only": false,
+  "tie_word_embeddings": false,
+  "image_token_id": 248056,
+  "video_token_id": 248057,
+  "vision_start_token_id": 248053,
+  "text_config": {
+    "model_type": "qwen3_5_text",
+    "hidden_size": 128,
+    "intermediate_size": 256,
+    "num_hidden_layers": 4,
+    "num_attention_heads": 4,
+    "num_key_value_heads": 2,
+    "head_dim": 32,
+    "vocab_size": 248320,
+    "full_attention_interval": 4,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_num_key_heads": 2,
+    "linear_num_value_heads": 4,
+    "linear_key_head_dim": 32,
+    "linear_value_head_dim": 32,
+    "linear_conv_kernel_dim": 4,
+    "hidden_act": "silu",
+    "attn_output_gate": true,
+    "output_gate_type": "swish",
+    "mamba_ssm_dtype": "float32",
+    "rms_norm_eps": 1e-06,
+    "max_position_embeddings": 1024,
+    "tie_word_embeddings": false,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "rope_parameters": {
+      "rope_type": "default",
+      "rope_theta": 10000000,
+      "partial_rotary_factor": 0.5,
+      "mrope_interleaved": true,
+      "mrope_section": [
+        3,
+        3,
+        2
+      ]
+    }
+  },
+  "vision_config": {
+    "model_type": "qwen3_5",
+    "depth": 2,
+    "hidden_size": 32,
+    "intermediate_size": 48,
+    "num_heads": 4,
+    "out_hidden_size": 128,
+    "num_position_embeddings": 16,
+    "patch_size": 2,
+    "temporal_patch_size": 2,
+    "spatial_merge_size": 2,
+    "in_channels": 3,
+    "hidden_act": "gelu_pytorch_tanh",
+    "deepstack_visual_indexes": []
+  },
+  "vision_end_token_id": 248054
+}

compatibility/cache-tests/test_server_cache_http.py

@@ -0,0 +1,271 @@
+# Copyright © 2026 Swift contributors.
+
+"""Offline HTTP regression checks on tiny random models, never Swift weights."""
+
+import argparse
+import contextlib
+import copy
+import functools
+import importlib.metadata
+import json
+import logging
+import os
+import platform
+import shutil
+import sys
+import tempfile
+import threading
+import urllib.request
+from http.server import ThreadingHTTPServer
+from pathlib import Path
+from unittest.mock import patch
+
+os.environ["HF_HUB_OFFLINE"] = "1"
+os.environ["TRANSFORMERS_OFFLINE"] = "1"
+
+import mlx.core as mx
+import mlx.nn as nn
+from mlx.utils import tree_flatten
+
+from mlx_lm import server
+from mlx_lm.models.cache import LRUPromptCache
+from mlx_lm.models.qwen3_5_full import Model, ModelArgs
+
+
+class RecordingCache(LRUPromptCache):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.peak_bytes = 0
+        self.insertions = 0
+
+    def insert_cache(self, *args, **kwargs):
+        super().insert_cache(*args, **kwargs)
+        self.insertions += 1
+        self.peak_bytes = max(self.peak_bytes, self.nbytes)
+        assert self.nbytes <= self.max_bytes
+        assert len(self) <= self.max_size
+
+
+def make_fixture(folder, assets, config_path, bits):
+    config = json.loads(config_path.read_text())
+    source = json.loads((assets / "config.json").read_text())
+    config["text_config"]["vocab_size"] = source["text_config"]["vocab_size"]
+    config.pop("quantization", None)
+    mx.random.seed(25)
+    model = Model(ModelArgs.from_dict(copy.deepcopy(config)))
+    model.language_model.lm_head.weight = mx.zeros_like(
+        model.language_model.lm_head.weight
+    )
+    model.apply(
+        lambda value: (
+            value.astype(mx.bfloat16)
+            if mx.issubdtype(value.dtype, mx.floating)
+            else value
+        )
+    )
+    nn.quantize(
+        model,
+        bits=bits,
+        group_size=64,
+        mode="affine",
+        class_predicate=lambda path, layer: hasattr(layer, "to_quantized")
+        and layer.weight.shape[-1] % 64 == 0,
+    )
+    mx.eval(model.parameters())
+    weights = dict(tree_flatten(model.parameters()))
+    mx.save_safetensors(
+        str(folder / "model.safetensors"),
+        weights,
+        metadata={"purpose": "synthetic regression only"},
+    )
+    index = {
+        "metadata": {"total_size": sum(w.nbytes for w in weights.values())},
+        "weight_map": {name: "model.safetensors" for name in weights},
+    }
+    (folder / "model.safetensors.index.json").write_text(json.dumps(index))
+    config["quantization"] = {"bits": bits, "group_size": 64, "mode": "affine"}
+    (folder / "config.json").write_text(json.dumps(config))
+    for name in (
+        "tokenizer.json",
+        "tokenizer_config.json",
+        "vocab.json",
+        "merges.txt",
+        "chat_template.jinja",
+        "generation_config.json",
+    ):
+        shutil.copyfile(assets / name, folder / name)
+    del model, weights
+    mx.clear_cache()
+
+
+@contextlib.contextmanager
+def running(folder, options):
+    captured = []
+    argv = ["mlx_lm.server", "--model", str(folder), "--port", "0", *options]
+    with patch.object(sys, "argv", argv), patch.object(
+        server, "run", side_effect=lambda h, p, m: captured.append(m)
+    ), patch.object(server, "maybe_set_recommended_wired_limit", return_value=None):
+        server.main()
+    provider = captured[0]
+    instances = []
+
+    def start_http(host, port, generator):
+        handler = functools.partial(
+            server.APIHandler, generator, system_fingerprint="synthetic-offline-test"
+        )
+        httpd = ThreadingHTTPServer((host, port), handler)
+        thread = threading.Thread(target=httpd.serve_forever, daemon=True)
+        instances.append((httpd, thread, generator))
+        thread.start()
+
+    with patch.object(server, "LRUPromptCache", RecordingCache), patch.object(
+        server, "_run_http_server", start_http
+    ):
+        server.run("127.0.0.1", 0, provider)
+    httpd, thread, generator = instances[0]
+    try:
+        yield f"http://127.0.0.1:{httpd.server_port}", generator, provider.cli_args
+    finally:
+        httpd.shutdown()
+        httpd.server_close()
+        thread.join(timeout=5)
+        generator.stop_and_join()
+
+
+def request(url, messages, stream=False, seeded=False):
+    data = {
+        "model": "default_model",
+        "messages": messages,
+        "max_tokens": 4,
+        "temperature": 0,
+        "chat_template_kwargs": {"enable_thinking": False},
+        "stream": stream,
+    }
+    if seeded:
+        data["seed"] = 25
+    req = urllib.request.Request(
+        url + "/v1/chat/completions",
+        data=json.dumps(data).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    with urllib.request.urlopen(req, timeout=60) as response:
+        body = response.read().decode()
+        assert response.status == 200
+    if stream:
+        assert "data: [DONE]" in body
+        chunks = [
+            json.loads(line[6:])
+            for line in body.splitlines()
+            if line.startswith("data: ") and line != "data: [DONE]"
+        ]
+        text = "".join(
+            choice.get("delta", {}).get("content", "") or ""
+            for chunk in chunks
+            for choice in chunk.get("choices", [])
+        )
+        assert text == "!!!!", text
+        return {"http_status": 200, "stream_complete": True}
+    result = json.loads(body)
+    assert result["choices"][0]["message"]["content"] == "!!!!"
+    usage = result["usage"]
+    return {
+        "http_status": 200,
+        "prompt_tokens": usage["prompt_tokens"],
+        "cached_tokens": usage["prompt_tokens_details"]["cached_tokens"],
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--assets",
+        type=Path,
+        required=True,
+        help="Local Swift snapshot; only tokenizer and config assets are read",
+    )
+    parser.add_argument("--config", type=Path, required=True)
+    parser.add_argument("--bits", type=int, choices=(4, 5), required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    logging.basicConfig(level=logging.WARNING)
+    report = {
+        "scope": "Tiny random weights with real local tokenizer assets. "
+        "No released model weights loaded or changed; no full-model capacity claim.",
+        "bits": args.bits,
+        "offline_hub": True,
+        "memory_limit_overrides": False,
+        "machine": platform.machine(),
+        "python": platform.python_version(),
+        "packages": {
+            name: importlib.metadata.version(name)
+            for name in ("mlx", "mlx-lm", "transformers", "huggingface_hub")
+        },
+        "profiles": [],
+    }
+    profiles = [
+        ("defaults", []),
+        ("explicit_bytes", ["--prompt-cache-bytes", "100000"]),
+        ("no_retention", ["--prompt-cache-size", "0"]),
+    ]
+    with tempfile.TemporaryDirectory(prefix="swift-cache-test-") as temp:
+        folder = Path(temp)
+        make_fixture(folder, args.assets, args.config, args.bits)
+        for name, options in profiles:
+            with running(folder, options) as (url, generator, cli):
+                assert cli.prompt_concurrency == cli.decode_concurrency == 1
+                assert cli.prefill_step_size == 512
+                messages = [
+                    {
+                        "role": "system",
+                        "content": "Help with coding. "
+                        + "Read the session carefully. " * 20,
+                    },
+                    {
+                        "role": "user",
+                        "content": "Remember this context. " + "sample words " * 40,
+                    },
+                    {"role": "assistant", "content": "Previous response."},
+                    {"role": "user", "content": "Continue briefly."},
+                ]
+                row = {"name": name, "requests": []}
+                for index in range(12):
+                    result = request(
+                        url, messages, stream=index == 10, seeded=index == 11
+                    )
+                    assert generator.generation_available()
+                    row["requests"].append(result)
+                    messages.extend(
+                        [
+                            {"role": "assistant", "content": "!!!!"},
+                            {
+                                "role": "user",
+                                "content": f"Another short answer {index}.",
+                            },
+                        ]
+                    )
+                cache = generator.prompt_cache
+                assert cache.insertions > 0
+                row.update(
+                    byte_limit=cache.max_bytes,
+                    peak_retained_bytes=cache.peak_bytes,
+                    retained_sequences=len(cache),
+                    insertions=cache.insertions,
+                )
+                nonstream = [r for r in row["requests"] if "cached_tokens" in r]
+                if name == "no_retention":
+                    assert cache.peak_bytes == 0
+                    assert all(r["cached_tokens"] == 0 for r in nonstream)
+                else:
+                    assert any(r["cached_tokens"] > 0 for r in nonstream[1:])
+                    assert 0 < cache.max_bytes < 1 << 63
+                if name == "explicit_bytes":
+                    assert cache.max_bytes == 100000
+                report["profiles"].append(row)
+    report["peak_mlx_bytes"] = mx.get_peak_memory()
+    report["status"] = "PASS"
+    args.output.write_text(json.dumps(report, indent=2) + "\n")
+    print(json.dumps(report, indent=2))
+
+
+if __name__ == "__main__":
+    main()

compatibility/cache-tests/validation.json

@@ -0,0 +1,55 @@
+{
+  "status": "PASS_LOCAL_REGRESSION_CHECKS",
+  "base_commit": "c69d1288440a0dc4e6401fc417098b07598dccd5",
+  "mlx_version": "0.32.2",
+  "mlx_lm_version": "0.32.0",
+  "server_sha256_before": "57392576de114af80ede723976b9dd156dc234f856b9943b8b44b8b78bc27f79",
+  "server_sha256_after": "791f8c1eb3431c4d24b4f9084f67b665c38e3d547ee4a307e52c5b2b6f12aba0",
+  "patch_sha256": "a7fbc0f0524ee7864d9f41a98a2e35acf0e63f53e369ee5b9a87ca12926a9eab",
+  "unit_and_upstream_server_tests": {
+    "passed": 44,
+    "failed": 0
+  },
+  "offline_http_tests": {
+    "passed_requests": 72,
+    "failed_requests": 0
+  },
+  "scope": "Local Apple Silicon tests using small fixtures. Full Swift long-context capacity is not established.",
+  "hosted_ci": "NOT_RUN; these are recorded local checks, not a hosted CI status",
+  "weights_changed": false,
+  "quantization_changed": false,
+  "fresh_patch_replay": {
+    "base_commit": "c69d1288440a0dc4e6401fc417098b07598dccd5",
+    "status": "PASS",
+    "releases": [
+      {
+        "bits": 4,
+        "clean_patch_apply": "PASS",
+        "rollback_check": "PASS",
+        "runtime_identity": "PASS",
+        "model_implementation_unchanged": "PASS",
+        "unit_and_upstream_tests_passed": 44,
+        "offline_http_requests_passed": 36,
+        "black": "25.1.0 PASS",
+        "isort": "6.0.0 PASS",
+        "ruff": "0.16.6 PASS",
+        "patch_sha256": "a7fbc0f0524ee7864d9f41a98a2e35acf0e63f53e369ee5b9a87ca12926a9eab",
+        "status": "PASS"
+      },
+      {
+        "bits": 5,
+        "clean_patch_apply": "PASS",
+        "rollback_check": "PASS",
+        "runtime_identity": "PASS",
+        "model_implementation_unchanged": "PASS",
+        "unit_and_upstream_tests_passed": 44,
+        "offline_http_requests_passed": 36,
+        "black": "25.1.0 PASS",
+        "isort": "6.0.0 PASS",
+        "ruff": "0.16.6 PASS",
+        "patch_sha256": "a7fbc0f0524ee7864d9f41a98a2e35acf0e63f53e369ee5b9a87ca12926a9eab",
+        "status": "PASS"
+      }
+    ]
+  }
+}

compatibility/cache-tests/verify_server_patch.py

@@ -0,0 +1,11 @@
+"""Verify the installed server patch without loading weights."""
+import hashlib
+import json
+from pathlib import Path
+from mlx_lm import server
+
+manifest = json.loads((Path(__file__).parent / "validation.json").read_text())
+actual = hashlib.sha256(Path(server.__file__).read_bytes()).hexdigest()
+if actual != manifest["server_sha256_after"]:
+    raise SystemExit("FAIL: this Python environment does not use the checked server patch")
+print("PASS: installed server source matches the checked cache patch")

compatibility/swift15-server-cache.patch

@@ -0,0 +1,253 @@
+diff --git a/mlx_lm/server.py b/mlx_lm/server.py
+index 9462d6e..36f7787 100644
+--- a/mlx_lm/server.py
++++ b/mlx_lm/server.py
+@@ -29,6 +29,7 @@ from typing import (
+ 
+ import mlx.core as mx
+ from huggingface_hub import scan_cache_dir
++from mlx.utils import tree_flatten
+ 
+ from ._version import __version__
+ from .generate import (
+@@ -428,6 +429,26 @@ def _format_top_logprobs(logprobs, top_n, tokenizer) -> Tuple[Dict[str, Any]]:
+     )
+ 
+ 
++def _prompt_cache_byte_limit(cli_args, model):
++    limit = getattr(cli_args, "prompt_cache_bytes", None)
++    if limit is not None:
++        if limit < 0:
++            raise ValueError("--prompt-cache-bytes must be non-negative")
++        return limit
++
++    # Reserve workspace and half the remaining capacity for the active request.
++    try:
++        recommended = mx.device_info().get("max_recommended_working_set_size")
++    except (RuntimeError, ValueError):
++        recommended = None
++    if not recommended:
++        return 1 << 63
++    parameters = {id(p): p for _, p in tree_flatten(model.parameters())}
++    model_bytes = sum(p.nbytes for p in parameters.values())
++    reserve = max(4 * 1024**3, recommended // 8)
++    return max(0, (recommended - model_bytes - reserve) // 2)
++
++
+ class ResponseGenerator:
+     def __init__(self, model_provider: ModelProvider, prompt_cache: LRUPromptCache):
+         self.model_provider = model_provider
+@@ -443,6 +464,13 @@ class ResponseGenerator:
+         self._generation_thread = Thread(target=self._run_generate)
+         self._generation_thread.start()
+ 
++    def _configure_prompt_cache(self, model):
++        limit = _prompt_cache_byte_limit(self.cli_args, model)
++        if self.prompt_cache.max_bytes != limit:
++            self.prompt_cache.max_bytes = limit
++            self.prompt_cache.trim_to(n_bytes=limit)
++            logging.info("Retained prompt-cache limit: %.2f GB", limit / 1e9)
++
+     def _run_generate(self):
+         try:
+             self._generate()
+@@ -772,6 +800,7 @@ class ResponseGenerator:
+                         model, tokenizer = self.model_provider.load(
+                             args.model.model, args.model.adapter, args.model.draft
+                         )
++                        self._configure_prompt_cache(model)
+                     except Exception as e:
+                         rqueue.put(e)
+                         continue
+@@ -1762,7 +1791,13 @@ def run(
+     handler_class=APIHandler,
+ ):
+     group = mx.distributed.init()
+-    prompt_cache = LRUPromptCache(model_provider.cli_args.prompt_cache_size)
++    cache_bytes = model_provider.cli_args.prompt_cache_bytes
++    if cache_bytes is not None and cache_bytes < 0:
++        raise ValueError("--prompt-cache-bytes must be non-negative")
++    prompt_cache = LRUPromptCache(
++        model_provider.cli_args.prompt_cache_size,
++        max_bytes=cache_bytes if cache_bytes is not None else 1 << 63,
++    )
+     response_generator = ResponseGenerator(model_provider, prompt_cache)
+     if group.rank() == 0:
+         _run_http_server(host, port, response_generator)
+@@ -1875,31 +1910,32 @@ def main():
+     parser.add_argument(
+         "--decode-concurrency",
+         type=int,
+-        default=32,
++        default=1,
+         help="When a request is batchable then decode that many requests in parallel",
+     )
+     parser.add_argument(
+         "--prompt-concurrency",
+         type=int,
+-        default=8,
++        default=1,
+         help="When a request is batchable then process that many prompts in parallel",
+     )
+     parser.add_argument(
+         "--prefill-step-size",
+         type=int,
+-        default=2048,
+-        help="Step size for prefill processing (default: 2048)",
++        default=512,
++        help="Step size for prefill processing (default: 512)",
+     )
+     parser.add_argument(
+         "--prompt-cache-size",
+         type=int,
+-        default=10,
+-        help="Maximum number of distinct KV caches to hold in the prompt cache",
++        default=2,
++        help="Maximum retained prompt-cache entries (default: 2)",
+     )
+     parser.add_argument(
+         "--prompt-cache-bytes",
+         type=_parse_size,
+-        help="Maximum size in bytes of the KV caches",
++        help="Maximum retained prompt-cache bytes. Default: automatic on Metal. "
++        "This does not cap active-request or total process memory.",
+     )
+     parser.add_argument(
+         "--kv-bits",
+diff --git a/tests/test_server_cache_budget.py b/tests/test_server_cache_budget.py
+new file mode 100644
+--- /dev/null
++++ b/tests/test_server_cache_budget.py
+@@ -0,0 +1,132 @@
++# Copyright © 2026 Swift contributors.
++
++import sys
++from types import SimpleNamespace
++from unittest.mock import Mock
++
++import pytest
++
++from mlx_lm import server
++from mlx_lm.models.cache import LRUPromptCache
++
++
++class CacheState:
++    def __init__(self, nbytes):
++        self.nbytes = nbytes
++
++    def is_trimmable(self):
++        return False
++
++
++@pytest.mark.parametrize("limit", [0, 100])
++def test_server_factory_enforces_configured_bytes(monkeypatch, limit):
++    caches = []
++    provider = SimpleNamespace(
++        cli_args=SimpleNamespace(prompt_cache_size=10, prompt_cache_bytes=limit)
++    )
++    monkeypatch.setattr(
++        server, "ResponseGenerator", lambda provider, cache: caches.append(cache)
++    )
++    monkeypatch.setattr(server, "_run_http_server", lambda *args: None)
++    server.run("127.0.0.1", 0, provider)
++    cache = caches[0]
++    for i in range(5):
++        cache.insert_cache("model", [i, 1], [CacheState(80)])
++        assert cache.nbytes <= limit
++    if limit:
++        reused, remaining = cache.fetch_nearest_cache("model", [4, 1, 9])
++        assert reused is not None
++        assert remaining == [9]
++
++
++def test_negative_limit_rejected_before_worker_start(monkeypatch):
++    worker = Mock()
++    monkeypatch.setattr(server, "ResponseGenerator", worker)
++    provider = SimpleNamespace(
++        cli_args=SimpleNamespace(prompt_cache_size=2, prompt_cache_bytes=-1)
++    )
++    with pytest.raises(ValueError, match="non-negative"):
++        server.run("127.0.0.1", 0, provider)
++    worker.assert_not_called()
++
++
++def test_explicit_limit_does_not_probe_model(monkeypatch):
++    probe = Mock(side_effect=AssertionError("device probe was not needed"))
++    monkeypatch.setattr(server.mx, "device_info", probe)
++    model = Mock()
++    for limit in (0, 1024, 8 * 1024**3):
++        assert (
++            server._prompt_cache_byte_limit(
++                SimpleNamespace(prompt_cache_bytes=limit), model
++            )
++            == limit
++        )
++    model.parameters.assert_not_called()
++
++
++def test_auto_limit_reserves_room_for_active_request(monkeypatch):
++    gib = 1024**3
++    monkeypatch.setattr(
++        server.mx,
++        "device_info",
++        lambda: {"max_recommended_working_set_size": 36 * gib},
++    )
++    model = SimpleNamespace(parameters=lambda: {"weight": CacheState(15 * gib)})
++    args = SimpleNamespace(prompt_cache_bytes=None)
++    limit = server._prompt_cache_byte_limit(args, model)
++    assert 6 * gib <= limit < (36 - 15) * gib // 2
++    larger = SimpleNamespace(parameters=lambda: {"weight": CacheState(30 * gib)})
++    assert server._prompt_cache_byte_limit(args, larger) < limit
++    full = SimpleNamespace(parameters=lambda: {"weight": CacheState(36 * gib)})
++    assert server._prompt_cache_byte_limit(args, full) == 0
++
++
++def test_model_swap_trims_retained_states(monkeypatch):
++    obj = server.ResponseGenerator.__new__(server.ResponseGenerator)
++    obj.model_provider = SimpleNamespace(
++        cli_args=SimpleNamespace(prompt_cache_bytes=100)
++    )
++    obj.prompt_cache = LRUPromptCache(max_size=10)
++    obj.prompt_cache.insert_cache("old", [1], [CacheState(200)])
++    obj._configure_prompt_cache(Mock())
++    assert obj.prompt_cache.nbytes == 0
++    assert obj.prompt_cache.max_bytes == 100
++
++
++@pytest.mark.parametrize("device_info", [{}, {"max_recommended_working_set_size": 0}])
++def test_auto_limit_without_metal_metadata(monkeypatch, device_info):
++    monkeypatch.setattr(server.mx, "device_info", lambda: device_info)
++    model = Mock()
++    assert (
++        server._prompt_cache_byte_limit(SimpleNamespace(prompt_cache_bytes=None), model)
++        == 1 << 63
++    )
++    model.parameters.assert_not_called()
++
++
++def test_auto_limit_counts_tied_parameters_once(monkeypatch):
++    gib = 1024**3
++    monkeypatch.setattr(
++        server.mx,
++        "device_info",
++        lambda: {"max_recommended_working_set_size": 32 * gib},
++    )
++    weight = CacheState(8 * gib)
++    model = SimpleNamespace(parameters=lambda: {"embed": weight, "head": weight})
++    assert (
++        server._prompt_cache_byte_limit(SimpleNamespace(prompt_cache_bytes=None), model)
++        == 10 * gib
++    )
++
++
++def test_defaults_keep_reuse_enabled_and_limit_concurrency(monkeypatch):
++    captured = []
++    monkeypatch.setattr(sys, "argv", ["mlx_lm.server"])
++    monkeypatch.setattr(server, "maybe_set_recommended_wired_limit", lambda: None)
++    monkeypatch.setattr(server, "run", lambda h, p, m: captured.append(m.cli_args))
++    server.main()
++    args = captured[0]
++    assert args.prompt_cache_size == 2
++    assert args.prompt_cache_bytes is None
++    assert args.prompt_concurrency == args.decode_concurrency == 1
++    assert args.prefill_step_size == 512