Record full 4-bit and 5-bit Mac cache validation

Existing complete checkpoints tested on a 48 GiB M4 Pro with native Metal, an 86k-token synthetic text prompt and cached follow-ups. No weights or runtime code changed.

FULL_MAC_VALIDATION.md

@@ -0,0 +1,55 @@
+# Full Swift Mac cache validation — 2026-09-25
+
+Both complete, existing Swift checkpoints passed native Metal text generation
+and two cached follow-up requests on an AWS M4 Pro Mac with 48 GiB RAM.
+All weight shards were verified against their recorded SHA-256 hashes before
+loading. No quantization, tensor edits, context changes or memory-limit overrides
+were performed. Hub offline mode was enabled during testing.
+
+The input is our own synthetic repeated-record conversation, followed by a short
+request to reply READY. It is a memory/cache regression test, not a quality
+benchmark, an exact reproduction of another person's conversation, or a claim
+that every context fits. The follow-ups must reuse at least half the prompt to
+pass; the test fails if the worker dies or the retained-cache budget is exceeded.
+
+| Checkpoint | Initial prompt tokens | First request, seconds | Follow-ups, seconds | Follow-up cached tokens | Peak MLX allocation, GiB |
+|---|---:|---:|---|---|---:|
+| 4-bit | 86,004 | 893.64 | 1.46, 1.42 | 86000, 86020 | 31.57 |
+| 5-bit | 86,004 | 907.00 | 1.45, 1.45 | 86000, 86020 | 34.79 |
+
+The first request includes model loading and initial prompt processing. MLX peak
+allocation is not total process or system memory. Complete request results,
+generated replies, cache limits and sampled swap observations are recorded in
+`compatibility/cache-tests/full-mac-4bit-results.json` and
+`compatibility/cache-tests/full-mac-5bit-results.json`.
+
+Environment: macOS 26.7 (25G229), Python 3.12.13, MLX 0.32.2, MLX-LM 0.32.0, Transformers 5.14.1,
+Hugging Face Hub 1.31.0. Official MLX-LM base revision:
+`c69d1288440a0dc4e6401fc417098b07598dccd5`.
+The shared runtime uses the existing Swift architecture patch, its 5-bit support
+extension, and the unchanged server-cache patch. The 5-bit extension also accepts
+the original 4-bit format. The cache defaults are two retained entries, automatic
+byte budgeting, one prompt/decode stream, and 512-token prefill steps.
+
+To reproduce, install the pinned environment and all three patches in the order
+documented by the 5-bit release, then use a complete local snapshot. Run one model
+at a time on an Apple Silicon Mac with at least 48 GiB RAM:
+
+```bash
+HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python \
+  Swift-1.5-4bit-MLX/compatibility/cache-tests/validate_full_mac_cache.py \
+  --snapshot Swift-1.5-4bit-MLX --output full-mac-4bit-results
+HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python \
+  Swift-1.5-5bit-MLX/compatibility/cache-tests/validate_full_mac_cache.py \
+  --snapshot Swift-1.5-5bit-MLX --output full-mac-5bit-results
+```
+
+Output directories must be new. The harness uses the real server and complete
+released weights, without building or quantizing any synthetic model. It disables
+the server's optional wired-limit adjustment and reports the actual Metal device.
+
+These results establish the tested text workload on the stated 48 GiB machine.
+They do not establish 24 GiB operation, arbitrary 262k-token workloads, GUI/plugin
+integration, image/video chat, speculative MTP generation, or broad output quality.
+Existing installations still need to apply the runtime patch and restart the server.
+The recorded checks are independently executed tests, not a hosted CI status.

README.md

@@ -35,8 +35,13 @@ its supported generation interface is text-only with the included MLX-LM patches
 **Runtime compatibility:** this complete checkpoint requires both supplied patches, in the order shown
 and the pinned Python installation in [USAGE.md](USAGE.md). The tested unpatched
 MLX-LM 0.32.0 loader rejects 501 saved vision entries. Downloading the model does
-not install these patches into an app's inference engine. GUI compatibility and
-full 27B Apple Silicon generation remain unverified.
+not install these patches into an app's inference engine. GUI compatibility remains unverified.
+
+Full-model follow-up validation is now recorded in
+[FULL_MAC_VALIDATION.md](FULL_MAC_VALIDATION.md): both complete checkpoints passed
+an 86k-token synthetic text conversation and two cached follow-ups on a 48 GiB
+M4 Pro using the patched server. GUI integration and other memory/context sizes
+remain outside that test.
 
 The complete weights total **19.28 GB (17.96 GiB) across 4 required shards**, plus
 config, index and tokenizer files. A single 5–6 GB file is not the complete model.
@@ -120,7 +125,7 @@ The CPU example promotes in-memory floating values to FP32 while retaining packe
 [diagnostic](compatibility/macos-quantized-matmul-diagnostic.json) and
 [package checks](compatibility/package-checks.json).
 
-Full 27B Apple Silicon generation is unverified. Integrated image/video chat and
+Full-model text generation is covered in [FULL_MAC_VALIDATION.md](FULL_MAC_VALIDATION.md). Integrated image/video chat and
 speculative MTP generation are **not implemented** by the patch. Vision/MTP weights
 and component checks do not establish those end-to-end capabilities. Runtime/cache
 and OS memory must be budgeted in addition to the tensor payload.

SERVER_CACHE_UPDATE.md

@@ -77,8 +77,11 @@ cached. The separate HTTP harness sets offline mode and needs only local assets.
 It covers 12 requests each with automatic budgeting, an explicit byte limit, and
 disabled retention; streaming and seeded sequential execution are included.
 Synthetic output is deliberately fixed for transport/cache checks, not quality.
-Full Swift long-context execution on a larger Mac has not been independently
-retested by this change. GUI runtimes need their own integration.
+Full-model follow-up validation is now recorded in
+[FULL_MAC_VALIDATION.md](FULL_MAC_VALIDATION.md): both complete checkpoints passed
+an 86k-token synthetic text conversation and two cached follow-ups on a 48 GiB
+M4 Pro using the patched server. GUI integration and other memory/context sizes
+remain outside that test.
 
 ## Roll back this runtime patch
 

TROUBLESHOOTING.md

@@ -100,3 +100,11 @@ against a dead worker will not recover it; restart only after addressing the cau
 
 These checks do not establish full-model Mac execution or GUI compatibility.
 Diagnosing an individual crash requires its original exception and runtime configuration.
+
+## Full-model Mac follow-up
+
+Full-model follow-up validation is now recorded in
+[FULL_MAC_VALIDATION.md](FULL_MAC_VALIDATION.md): both complete checkpoints passed
+an 86k-token synthetic text conversation and two cached follow-ups on a 48 GiB
+M4 Pro using the patched server. GUI integration and other memory/context sizes
+remain outside that test.

UPLOAD_MANIFEST.json

@@ -6,6 +6,11 @@
       "sha256": "8a9fd1f95733fb82a6cba50f5ea2b1e26ca7578bea7a7cc21b4301ff83b1ce3b",
       "git_blob_sha1": "3727b14f485a05787b6cda82f0346eaeaa2e751b"
     },
+    {
+      "path": "FULL_MAC_VALIDATION.md",
+      "bytes": 3246,
+      "sha256": "a3a164e3affcfbe8a506b0205f4d4e63a69807ea56f4a8baa06b2201b20b1faa"
+    },
     {
       "path": "LICENSE",
       "bytes": 13306,
@@ -32,24 +37,23 @@
     },
     {
       "path": "README.md",
-      "bytes": 8067,
-      "sha256": "e0ce3327646a3dde90ee001a3bb819c8cb24e550a83d28340671f2542e7ad55b",
-      "git_blob_sha1": "008e4ab150bc188e372cad12fda89317afda850e"
+      "bytes": 8392,
+      "sha256": "25708f8dc77158d099de1e1155f64d3eef58bfed06c892d7d65f21125b6381b5"
     },
     {
       "path": "SERVER_CACHE_UPDATE.md",
-      "bytes": 4765,
-      "sha256": "31ae8115dcf0ccba9e91dd03f9eab30dc56a6a752fa26955c05555cc655bdcd7"
+      "bytes": 4941,
+      "sha256": "aad61a1a3bf8352ebf214c14f232869f62b282f4fd16f510de63375d7a04d5c7"
     },
     {
       "path": "TROUBLESHOOTING.md",
-      "bytes": 5623,
-      "sha256": "8d631dcdacfd6f98d0ac37c4d2532f1a8139ab2e336c8ae27bfe03c251444327"
+      "bytes": 5972,
+      "sha256": "79d58bcc5929fe4ea3a2becefedf189d2d105102dce07eb894ef08212aa9be6d"
     },
     {
       "path": "USAGE.md",
-      "bytes": 6858,
-      "sha256": "946ba223dd2a7cf1aceb069cf4fd82f4bf609366abacba61017f08e2bf9a29b4"
+      "bytes": 7207,
+      "sha256": "e8590662c34f04462a5da73cb2e0572a26a5a18c385a2dcd281daaacbf47e963"
     },
     {
       "path": "chat_template.jinja",
@@ -69,6 +73,16 @@
       "sha256": "ccfab7ccb2ea306f71531c8ca77bb55507606cd90768b1e32b8b52ab5b48cf01",
       "git_blob_sha1": "98ff47b9ef9d4ac9f1a4bde3db13dd27456c5ea1"
     },
+    {
+      "path": "compatibility/cache-tests/full-mac-4bit-results.json",
+      "bytes": 2699,
+      "sha256": "6087feec8158cd3f1df6e7c3394a51a5091ee96649066ad875f6b6ac866dd76b"
+    },
+    {
+      "path": "compatibility/cache-tests/full-mac-5bit-results.json",
+      "bytes": 2695,
+      "sha256": "274b30cdac2d75d0258ff79a91ec53ce9b7029f36d1d64edf8783037a9c67459"
+    },
     {
       "path": "compatibility/cache-tests/offline-4bit-results.json",
       "bytes": 5014,
@@ -89,6 +103,11 @@
       "bytes": 9689,
       "sha256": "e7aff50b441145338575a50c7abc5fb6d40fdac3d11bbf6433d909ecfe22483b"
     },
+    {
+      "path": "compatibility/cache-tests/validate_full_mac_cache.py",
+      "bytes": 8156,
+      "sha256": "3e48876b07fb73f88c489e3804c9676872b6c1972650447de5820f1966b5811e"
+    },
     {
       "path": "compatibility/cache-tests/validation.json",
       "bytes": 1984,

USAGE.md

@@ -134,3 +134,11 @@ complete customized Swift BF16 source identified in `QUANTIZATION_MANIFEST.json`
 with its 18 shards verified before conversion, the pinned patched converter, and
 affine / 5-bit / group size 64. Do not fill missing weights with base Qwen or another
 quantization. Existing quantized weights are unchanged by these packaging repairs.
+
+## Full-model Mac follow-up
+
+Full-model follow-up validation is now recorded in
+[FULL_MAC_VALIDATION.md](FULL_MAC_VALIDATION.md): both complete checkpoints passed
+an 86k-token synthetic text conversation and two cached follow-ups on a 48 GiB
+M4 Pro using the patched server. GUI integration and other memory/context sizes
+remain outside that test.

compatibility/cache-tests/full-mac-4bit-results.json

@@ -0,0 +1,97 @@
+{
+  "status": "PASS_FULL_MODEL_SYNTHETIC_LONG_CONTEXT_FOLLOWUPS",
+  "physical_memory_bytes": 51539607552,
+  "offline": true,
+  "model_weights_changed": false,
+  "memory_limit_overrides": false,
+  "device": "Device(gpu, 0)",
+  "device_info": {
+    "device_name": "Apple M4 Pro",
+    "max_recommended_working_set_size": 40200896512,
+    "memory_size": 51539607552,
+    "architecture": "applegpu_g16s",
+    "max_buffer_length": 30150672384,
+    "resource_limit": 499000
+  },
+  "packages": {
+    "mlx": "0.32.2",
+    "mlx-lm": "0.32.0",
+    "transformers": "5.14.1",
+    "huggingface_hub": "1.31.0"
+  },
+  "prompt_origin": "Synthetic repeated records; no third-party conversation",
+  "initial_prompt_tokens": 86004,
+  "requests": [
+    {
+      "turn": 1,
+      "elapsed_seconds": 893.6356901669999,
+      "usage": {
+        "prompt_tokens": 86004,
+        "completion_tokens": 2,
+        "total_tokens": 86006,
+        "prompt_tokens_details": {
+          "cached_tokens": 0
+        }
+      },
+      "generation": "READY",
+      "finish_reason": "stop",
+      "retained_bytes": 5945229312,
+      "retained_limit": 9674659152,
+      "retained_sequences": 2,
+      "peak_mlx_bytes": 33428188365
+    },
+    {
+      "turn": 2,
+      "elapsed_seconds": 1.4563554589999512,
+      "usage": {
+        "prompt_tokens": 86024,
+        "completion_tokens": 2,
+        "total_tokens": 86026,
+        "prompt_tokens_details": {
+          "cached_tokens": 86000
+        }
+      },
+      "generation": "READY",
+      "finish_reason": "stop",
+      "retained_bytes": 5946540032,
+      "retained_limit": 9674659152,
+      "retained_sequences": 2,
+      "peak_mlx_bytes": 33885106168
+    },
+    {
+      "turn": 3,
+      "elapsed_seconds": 1.4185313340001358,
+      "usage": {
+        "prompt_tokens": 86044,
+        "completion_tokens": 2,
+        "total_tokens": 86046,
+        "prompt_tokens_details": {
+          "cached_tokens": 86020
+        }
+      },
+      "generation": "READY",
+      "finish_reason": "stop",
+      "retained_bytes": 5947850752,
+      "retained_limit": 9674659152,
+      "retained_sequences": 2,
+      "peak_mlx_bytes": 33902489592
+    }
+  ],
+  "warm_reuse_observed": true,
+  "warm_reuse_fraction": [
+    0.9997210080907654,
+    0.9997210729394264
+  ],
+  "monitor_summary": {
+    "samples": 90,
+    "peak_active_mlx_bytes": 24817128310,
+    "peak_allocator_cache_bytes": 5144874723,
+    "swap_observations": [
+      "total = 0.00M  used = 0.00M  free = 0.00M  (encrypted)"
+    ],
+    "worker_healthy_in_all_samples": true,
+    "retained_limit_observed_in_all_samples": true
+  },
+  "model_repository": "ukisai/Swift-1.5-4bit-MLX",
+  "tested_revision": "c4765c752caee282928f62685722e11000ec64cf"
+}

compatibility/cache-tests/full-mac-5bit-results.json

@@ -0,0 +1,97 @@
+{
+  "status": "PASS_FULL_MODEL_SYNTHETIC_LONG_CONTEXT_FOLLOWUPS",
+  "physical_memory_bytes": 51539607552,
+  "offline": true,
+  "model_weights_changed": false,
+  "memory_limit_overrides": false,
+  "device": "Device(gpu, 0)",
+  "device_info": {
+    "device_name": "Apple M4 Pro",
+    "max_recommended_working_set_size": 40200896512,
+    "memory_size": 51539607552,
+    "architecture": "applegpu_g16s",
+    "max_buffer_length": 30150672384,
+    "resource_limit": 499000
+  },
+  "packages": {
+    "mlx": "0.32.2",
+    "mlx-lm": "0.32.0",
+    "transformers": "5.14.1",
+    "huggingface_hub": "1.31.0"
+  },
+  "prompt_origin": "Synthetic repeated records; no third-party conversation",
+  "initial_prompt_tokens": 86004,
+  "requests": [
+    {
+      "turn": 1,
+      "elapsed_seconds": 906.998991333,
+      "usage": {
+        "prompt_tokens": 86004,
+        "completion_tokens": 2,
+        "total_tokens": 86006,
+        "prompt_tokens_details": {
+          "cached_tokens": 0
+        }
+      },
+      "generation": "READY",
+      "finish_reason": "stop",
+      "retained_bytes": 5945229312,
+      "retained_limit": 7946990032,
+      "retained_sequences": 2,
+      "peak_mlx_bytes": 36879525149
+    },
+    {
+      "turn": 2,
+      "elapsed_seconds": 1.4516940830003477,
+      "usage": {
+        "prompt_tokens": 86024,
+        "completion_tokens": 2,
+        "total_tokens": 86026,
+        "prompt_tokens_details": {
+          "cached_tokens": 86000
+        }
+      },
+      "generation": "READY",
+      "finish_reason": "stop",
+      "retained_bytes": 5946540032,
+      "retained_limit": 7946990032,
+      "retained_sequences": 2,
+      "peak_mlx_bytes": 37340425976
+    },
+    {
+      "turn": 3,
+      "elapsed_seconds": 1.4471444169998904,
+      "usage": {
+        "prompt_tokens": 86044,
+        "completion_tokens": 2,
+        "total_tokens": 86046,
+        "prompt_tokens_details": {
+          "cached_tokens": 86020
+        }
+      },
+      "generation": "READY",
+      "finish_reason": "stop",
+      "retained_bytes": 5947850752,
+      "retained_limit": 7946990032,
+      "retained_sequences": 2,
+      "peak_mlx_bytes": 37358006008
+    }
+  ],
+  "warm_reuse_observed": true,
+  "warm_reuse_fraction": [
+    0.9997210080907654,
+    0.9997210729394264
+  ],
+  "monitor_summary": {
+    "samples": 91,
+    "peak_active_mlx_bytes": 28351155232,
+    "peak_allocator_cache_bytes": 5353876378,
+    "swap_observations": [
+      "total = 0.00M  used = 0.00M  free = 0.00M  (encrypted)"
+    ],
+    "worker_healthy_in_all_samples": true,
+    "retained_limit_observed_in_all_samples": true
+  },
+  "model_repository": "ukisai/Swift-1.5-5bit-MLX",
+  "tested_revision": "5359607ded7c88e3c7379a0adfb879eaf483cd98"
+}

compatibility/cache-tests/validate_full_mac_cache.py

@@ -0,0 +1,166 @@
+"""Validate the existing full checkpoint on an Apple Silicon Mac with >=48 GiB."""
+
+import argparse
+import importlib.util
+import importlib.metadata
+import json
+import logging
+import os
+from pathlib import Path
+import platform
+import subprocess
+import sys
+import threading
+import time
+import urllib.request
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--snapshot", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--target-prompt-tokens", type=int, default=86000)
+    args = parser.parse_args()
+    if platform.system() != "Darwin" or platform.machine() != "arm64":
+        raise SystemExit("This validation requires a native Apple Silicon Mac.")
+    physical_bytes = int(subprocess.check_output(["sysctl", "-n", "hw.memsize"]))
+    if physical_bytes < 48 * 1024**3:
+        raise SystemExit("Refusing a full checkpoint load: at least 48 GiB is required for this test.")
+    if args.target_prompt_tokens < 4096:
+        raise SystemExit("Use at least 4096 tokens for the full-model follow-up test.")
+    args.snapshot = args.snapshot.resolve(strict=True)
+    args.output.mkdir(parents=True, exist_ok=False)
+    os.environ["HF_HUB_OFFLINE"] = "1"
+    os.environ["TRANSFORMERS_OFFLINE"] = "1"
+    logging.basicConfig(level=logging.INFO)
+
+    # Check files and the installed patch before loading model weights.
+    for command in (
+        [sys.executable, str(args.snapshot / "check_download.py"),
+         str(args.snapshot), "--hash", "--runtime"],
+        [sys.executable, str(args.snapshot / "compatibility/cache-tests/verify_server_patch.py")],
+    ):
+        subprocess.run(command, check=True)
+
+    import mlx.core as mx
+    from transformers import AutoTokenizer
+
+    assert mx.metal.is_available(), "Metal is required; a CPU run cannot validate this crash"
+    assert mx.default_device() != mx.cpu, "The validation must execute on the Metal GPU"
+    assert importlib.metadata.version("mlx") == "0.32.2"
+    assert importlib.metadata.version("mlx-lm") == "0.32.0"
+
+    harness_path = args.snapshot / "compatibility/cache-tests/test_server_cache_http.py"
+    spec = importlib.util.spec_from_file_location("cache_harness", harness_path)
+    harness = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(harness)
+    tokenizer = AutoTokenizer.from_pretrained(args.snapshot, local_files_only=True)
+
+    def prompt_length(messages):
+        tokens = tokenizer.apply_chat_template(
+            messages, tokenize=True, return_dict=False,
+            add_generation_prompt=True, enable_thinking=False,
+        )
+        assert isinstance(tokens, list) and tokens and isinstance(tokens[0], int)
+        return len(tokens)
+
+    def make_messages(count):
+        return [
+            {"role": "system", "content": "Read the records. Reply briefly to the final question."},
+            {"role": "user", "content": ("Record: alpha beta gamma delta epsilon.\n" * count)},
+            {"role": "assistant", "content": "I have read the records."},
+            {"role": "user", "content": "Reply with the single word READY."},
+        ]
+
+    lo, hi = 1, args.target_prompt_tokens
+    while lo < hi:
+        mid = (lo + hi) // 2
+        if prompt_length(make_messages(mid)) < args.target_prompt_tokens:
+            lo = mid + 1
+        else:
+            hi = mid
+    messages = make_messages(lo)
+    initial_tokens = prompt_length(messages)
+    assert args.target_prompt_tokens <= initial_tokens < args.target_prompt_tokens + 128
+    print(f"Validated synthetic prompt length: {initial_tokens} tokens", flush=True)
+    report = {"status": "RUNNING", "physical_memory_bytes": physical_bytes,
+              "offline": True, "model_weights_changed": False,
+              "memory_limit_overrides": False,
+              "device": str(mx.default_device()), "device_info": mx.device_info(),
+              "packages": {name: importlib.metadata.version(name)
+                           for name in ("mlx", "mlx-lm", "transformers", "huggingface_hub")},
+              "prompt_origin": "Synthetic repeated records; no third-party conversation",
+              "initial_prompt_tokens": initial_tokens, "requests": []}
+    output_file = args.output / "results.json"
+    stopped = threading.Event()
+
+    def save():
+        output_file.write_text(json.dumps(report, indent=2) + "\n")
+
+    def monitor(generator):
+        with (args.output / "memory.jsonl").open("w") as stream:
+            while not stopped.is_set():
+                cache = generator.prompt_cache
+                row = {"time": time.time(), "active_mlx_bytes": mx.get_active_memory(),
+                       "allocator_cache_bytes": mx.get_cache_memory(),
+                       "peak_mlx_bytes": mx.get_peak_memory(),
+                       "retained_bytes": cache.nbytes, "retained_limit": cache.max_bytes,
+                       "retained_sequences": len(cache),
+                       "worker_available": generator.generation_available(),
+                       "swap": subprocess.check_output(["sysctl", "-n", "vm.swapusage"], text=True).strip()}
+                stream.write(json.dumps(row) + "\n")
+                stream.flush()
+                stopped.wait(10)
+
+    save()
+    try:
+        with harness.running(args.snapshot, []) as (url, generator, cli):
+            observer = threading.Thread(target=monitor, args=(generator,), daemon=True)
+            observer.start()
+            try:
+                for turn in range(3):
+                    body = {"model": "default_model", "messages": messages, "max_tokens": 16,
+                            "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}}
+                    req = urllib.request.Request(url + "/v1/chat/completions",
+                          data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
+                    started = time.monotonic()
+                    with urllib.request.urlopen(req, timeout=3600) as response:
+                        result = json.load(response)
+                        assert response.status == 200
+                    choice = result["choices"][0]
+                    content = choice["message"].get("content")
+                    assert isinstance(content, str) and content.strip(), result
+                    assert generator.generation_available()
+                    cache = generator.prompt_cache
+                    row = {"turn": turn + 1, "elapsed_seconds": time.monotonic() - started,
+                           "usage": result["usage"], "generation": content,
+                           "finish_reason": choice["finish_reason"],
+                           "retained_bytes": cache.nbytes, "retained_limit": cache.max_bytes,
+                           "retained_sequences": len(cache), "peak_mlx_bytes": mx.get_peak_memory()}
+                    report["requests"].append(row)
+                    save()
+                    print(json.dumps(row), flush=True)
+                    messages.extend([{"role": "assistant", "content": content},
+                                     {"role": "user", "content": "Reply with READY again."}])
+                hits = [row["usage"]["prompt_tokens_details"]["cached_tokens"]
+                        for row in report["requests"][1:]]
+                report["warm_reuse_observed"] = any(hits)
+                report["warm_reuse_fraction"] = [
+                    row["usage"]["prompt_tokens_details"]["cached_tokens"] / row["usage"]["prompt_tokens"]
+                    for row in report["requests"][1:]]
+                assert all(fraction >= 0.5 for fraction in report["warm_reuse_fraction"]), \
+                    "Requests completed, but the cache did not reuse most of the long history"
+                report["status"] = "PASS_FULL_MODEL_SYNTHETIC_LONG_CONTEXT_FOLLOWUPS"
+                save()
+            finally:
+                stopped.set()
+                observer.join(timeout=15)
+    except BaseException as error:
+        report["status"] = "FAIL"
+        report["error"] = f"{type(error).__name__}: {error}"
+        save()
+        raise
+
+
+if __name__ == "__main__":
+    main()