USAGE.md

# Swift 1.5 5-bit — complete MLX architecture

Use only a complete snapshot: all four shards, original index, tokenizer and
processor/runtime files are required. The 19.28 GB tensor payload plus runtime,
cache and OS must fit available memory. Do not load this full model on a 16 GiB Mac
or raise system memory limits to conceal insufficient hardware.

## Pin the snapshot and verify it

Use a new working directory. This repository is public; authentication is optional.
The command below resolves current main once to a full commit and then uses
only that pinned snapshot. For a repeat run, reuse the recorded commit.
Do not use an incomplete historical upload or proceed after verification failure.

```bash
python3.12 -m venv .venv-swift5
source .venv-swift5/bin/activate
python -m pip install 'huggingface_hub==1.31.0'
SWIFT_MLX_REVISION="$(python -c 'from huggingface_hub import HfApi; print(HfApi().model_info("ukisai/Swift-1.5-5bit-MLX").sha)')"
printf 'Pinned model revision: %s\n' "$SWIFT_MLX_REVISION"
hf download ukisai/Swift-1.5-5bit-MLX --revision "$SWIFT_MLX_REVISION" --local-dir Swift-1.5-5bit-MLX
hf cache verify ukisai/Swift-1.5-5bit-MLX --revision "$SWIFT_MLX_REVISION" --local-dir Swift-1.5-5bit-MLX --fail-on-missing-files
python Swift-1.5-5bit-MLX/check_download.py Swift-1.5-5bit-MLX
python Swift-1.5-5bit-MLX/verify_release.py Swift-1.5-5bit-MLX
```

The included checker has no network or model-loading code. Supply its optional
`--manifest-sha256` argument from a trusted release plan to pin the manifest too.
Without that trusted digest it checks consistency, not source authenticity.
Stop if any file, checksum, index entry or payload-boundary check fails.

## Install the patches in this order

```bash
git clone https://github.com/ml-explore/mlx-lm.git swift5-mlx-lm
git -C swift5-mlx-lm checkout --detach c69d1288440a0dc4e6401fc417098b07598dccd5
git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/swift15-mlx-lm.patch
git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/swift15-mlx-lm.patch
git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/enable-5bit.patch
git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/enable-5bit.patch
git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
```

Apple Silicon:

```bash
python -m pip install 'mlx==0.32.2' 'transformers==5.14.1' 'huggingface_hub==1.31.0' 'pillow==12.3.0'
python -m pip install -e ./swift5-mlx-lm
```

Linux CPU, Python 3.12, glibc 2.35 or newer:

```bash
python -m pip install 'mlx[cpu]==0.32.2' 'transformers==5.14.1' 'huggingface_hub==1.31.0' 'pillow==12.3.0'
python -m pip install -e ./swift5-mlx-lm
```

The historical [Linux environment](compatibility/environment-linux.json) records
Hub 1.32.0. The 1.31.0 pin above was separately installed in the independent macOS
audit, where 18 synthetic patch tests passed. These environments are not identical
and the tests did not load the full 27B model. The upstream code
[MIT notice](compatibility/LICENSE-MLX-LM-MIT) is included separately from weight licenses.

Before generating, check the Python environment that will actually run the model:

```bash
python Swift-1.5-5bit-MLX/check_download.py Swift-1.5-5bit-MLX --runtime
```

This checks the complete download and selects the pinned patched architecture
without loading weights. It does not check a separate GUI app's environment,
available inference memory, or generated quality. See [TROUBLESHOOTING.md](TROUBLESHOOTING.md)
if it fails or a server reports `generation thread died`.

## Text generation

```python
import mlx.core as mx
from mlx_lm import generate, load
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("Swift-1.5-5bit-MLX")
if mx.default_device() == mx.cpu:
    model.apply(lambda x: x.astype(mx.float32) if mx.issubdtype(x.dtype, mx.floating) else x)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Reply with exactly: Hello from Swift."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
mx.random.seed(20260922)
print(generate(model, tokenizer, prompt=prompt, max_tokens=32, sampler=make_sampler(temp=0)))
```

The CPU branch changes only in-memory floating types; packed UINT32 weights and
files are unchanged. The historical build reported a Linux BF16 accumulation issue.
Its original diagnostic file was not published. The separately executed
[macOS CPU/Metal diagnostic](compatibility/macos-quantized-matmul-diagnostic.json)
uses synthetic tensors: summing 8,192 ones gives 256 on CPU BF16 and 8,192 with FP32;
Metal BF16/FP32 also give 8,192. It is not a new Linux or full-model generation test.

The template supports `reasoning_effort="low"`, `"medium"`, and `"xhigh"`.
Template support does not establish generated quality for those modes.
The explicit `model.mtp_logits` step and `model.visual` encoder have historical
component evidence. Integrated image/video chat and speculative generation are
not implemented; unsupported multimodal generation must not be reported as working.

## Server cache update

The supplied server patch keeps prompt reuse enabled with at most two retained
entries, enforces a retained-cache byte budget at insertion, and defaults to one
prompt and one decode stream with 512-token prefill steps. On Metal the default
budget estimates headroom after model weights and workspace; an explicit
`--prompt-cache-bytes` overrides it. `--prompt-cache-size 2` means two entries,
not two GB.

See [SERVER_CACHE_UPDATE.md](SERVER_CACHE_UPDATE.md) for installation, offline
startup, validation and rollback. Existing installations must apply this runtime
update and restart. No model-weight download or re-quantization is needed.

The retained-cache limit does not cap active-request or total process memory.
An entry larger than the budget is evicted, so reuse depends on what fits. Use
`--prompt-cache-size 0` only when intentionally disabling reuse. The model's
configured context length is not a promise that it fits every Mac.

Validation: 44 unit/upstream server tests and 72 offline HTTP requests passed
across tiny 4-bit and 5-bit fixtures. These checks cover retention, warm reuse,
streaming and sequential generation. They do not establish full-model
long-context capacity or GUI integration.

## Conversion provenance

Conversion is not part of installation. If separately authorized, use only the
complete customized Swift BF16 source identified in `QUANTIZATION_MANIFEST.json`,
with its 18 shards verified before conversion, the pinned patched converter, and
affine / 5-bit / group size 64. Do not fill missing weights with base Qwen or another
quantization. Existing quantized weights are unchanged by these packaging repairs.

## Full-model Mac follow-up

Full-model follow-up validation is now recorded in
[FULL_MAC_VALIDATION.md](FULL_MAC_VALIDATION.md): both complete checkpoints passed
an 86k-token synthetic text conversation and two cached follow-ups on a 48 GiB
M4 Pro using the patched server. GUI integration and other memory/context sizes
remain outside that test.