USAGE.md

# Load Swift 1.5 with its complete MLX architecture

Use the included patch with the pinned official Apple MLX-LM revision. Unpatched
text-only Qwen support does not preserve this checkpoint's complete parameter tree.

Use Python 3.12 in a new working directory. Install the HF CLI before using it,
and pin the complete model revision as well as the MLX-LM source revision.
This repository is private: after installing the CLI, run `hf auth login`
interactively if not already signed in with an account that has access.

```bash
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install 'huggingface_hub==1.31.0'
SWIFT_MLX_REVISION=d2140379e1fd593002c92fa552b2b37fb6eb1159
hf download ukisai/Swift-1.5-4bit-MLX --revision "$SWIFT_MLX_REVISION" --local-dir Swift-1.5-4bit-MLX
hf cache verify ukisai/Swift-1.5-4bit-MLX --revision "$SWIFT_MLX_REVISION" --local-dir Swift-1.5-4bit-MLX --fail-on-missing-files
git clone https://github.com/ml-explore/mlx-lm.git swift15-mlx-lm
git -C swift15-mlx-lm checkout --detach c69d1288440a0dc4e6401fc417098b07598dccd5
git -C swift15-mlx-lm apply --check ../Swift-1.5-4bit-MLX/compatibility/swift15-mlx-lm.patch
git -C swift15-mlx-lm apply ../Swift-1.5-4bit-MLX/compatibility/swift15-mlx-lm.patch
```

Stop after any missing-file or checksum failure. The checkpoint contains about
15.83 GB of tensor data, before runtime, cache and OS overhead. Do not force the
full model onto a 16 GiB Mac or increase system memory limits. Full-model Apple
generation remains unverified. The recorded small Metal samples are not a full run.

On Apple Silicon:

```bash
pip install 'mlx==0.32.2' 'transformers==5.14.1' 'huggingface_hub==1.31.0' pillow
pip install -e ./swift15-mlx-lm
```

On Linux CPU (Python 3.12 and glibc 2.35 or newer):

```bash
pip install 'mlx[cpu]==0.32.2' 'transformers==5.14.1' 'huggingface_hub==1.31.0' pillow
pip install -e ./swift15-mlx-lm
```

Text generation:

```python
import mlx.core as mx
from mlx_lm import load, generate

model, tokenizer = load("Swift-1.5-4bit-MLX")
if mx.default_device() == mx.cpu:
    model.apply(
        lambda value: value.astype(mx.float32)
        if mx.issubdtype(value.dtype, mx.floating) else value
    )
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Say hello."}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=32))
```

The Linux CPU branch promotes only in-memory floating parameters to FP32. Packed
4-bit weights and all files remain unchanged. This avoids the official MLX 0.32.2
Linux scalar BF16 quantized-matmul accumulation bug reproduced in
`compatibility/cpu-quantized-matmul-diagnostic.json` (8,192 exact ones summed to
256 in BF16, versus the correct 8,192 in FP32). The release's CPU generation,
MTP and vision smoke tests use this FP32 runtime. Apple Silicon inference does
not use this CPU workaround; full-model Apple Silicon execution was not tested.

The original chat template also accepts `reasoning_effort="low"`, `"medium"`,
and `"xhigh"`; this release validates the original low and xhigh formats.
This structural smoke test does not establish long-context or benchmark accuracy.

The patch implements an explicit MTP step (`model.mtp_logits`) and the vision
encoder (`model.visual`). Their weights are retained and the release validation
records their component execution. Speculative generation and integrated image/video
chat are not implemented. Unsupported multimodal generation calls raise an error.

Reproduce conversion only from the complete original Swift BF16 export identified
in `QUANTIZATION_MANIFEST.json`, after verifying its 18 shards and original assets:

```bash
mlx_lm.convert --hf-path /path/to/Swift-1.5-BF16 \
  --mlx-path Swift-1.5-4bit-MLX \
  --quantize --q-mode affine --q-bits 4 --q-group-size 64
```

The converter refuses an existing output directory. It uses official MLX-LM lazy
loading, quantization, sharding, and saving; the patch supplies the complete model
and strict parameter/asset mapping.