Instructions to use ukisai/Swift-1.5-5bit-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ukisai/Swift-1.5-5bit-MLX with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("ukisai/Swift-1.5-5bit-MLX")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use ukisai/Swift-1.5-5bit-MLX with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "ukisai/Swift-1.5-5bit-MLX"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ukisai/Swift-1.5-5bit-MLX"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use ukisai/Swift-1.5-5bit-MLX with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "ukisai/Swift-1.5-5bit-MLX"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "ukisai/Swift-1.5-5bit-MLX"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "ukisai/Swift-1.5-5bit-MLX",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use ukisai/Swift-1.5-5bit-MLX with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "ukisai/Swift-1.5-5bit-MLX"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ukisai/Swift-1.5-5bit-MLX

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ukisai/Swift-1.5-5bit-MLX with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "ukisai/Swift-1.5-5bit-MLX"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ukisai/Swift-1.5-5bit-MLX" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Swift-1.5-5bit-MLX / USAGE.md

ukisai

Record full 4-bit and 5-bit Mac cache validation

c62a35c verified 17 days ago

preview code

Raw History Blame

7.21 kB

Swift 1.5 5-bit — complete MLX architecture

Use only a complete snapshot: all four shards, original index, tokenizer and processor/runtime files are required. The 19.28 GB tensor payload plus runtime, cache and OS must fit available memory. Do not load this full model on a 16 GiB Mac or raise system memory limits to conceal insufficient hardware.

Pin the snapshot and verify it

Use a new working directory. This repository is public; authentication is optional. The command below resolves current main once to a full commit and then uses only that pinned snapshot. For a repeat run, reuse the recorded commit. Do not use an incomplete historical upload or proceed after verification failure.

python3.12 -m venv .venv-swift5
source .venv-swift5/bin/activate
python -m pip install 'huggingface_hub==1.31.0'
SWIFT_MLX_REVISION="$(python -c 'from huggingface_hub import HfApi; print(HfApi().model_info("ukisai/Swift-1.5-5bit-MLX").sha)')"
printf 'Pinned model revision: %s\n' "$SWIFT_MLX_REVISION"
hf download ukisai/Swift-1.5-5bit-MLX --revision "$SWIFT_MLX_REVISION" --local-dir Swift-1.5-5bit-MLX
hf cache verify ukisai/Swift-1.5-5bit-MLX --revision "$SWIFT_MLX_REVISION" --local-dir Swift-1.5-5bit-MLX --fail-on-missing-files
python Swift-1.5-5bit-MLX/check_download.py Swift-1.5-5bit-MLX
python Swift-1.5-5bit-MLX/verify_release.py Swift-1.5-5bit-MLX

The included checker has no network or model-loading code. Supply its optional --manifest-sha256 argument from a trusted release plan to pin the manifest too. Without that trusted digest it checks consistency, not source authenticity. Stop if any file, checksum, index entry or payload-boundary check fails.

Install the patches in this order

git clone https://github.com/ml-explore/mlx-lm.git swift5-mlx-lm
git -C swift5-mlx-lm checkout --detach c69d1288440a0dc4e6401fc417098b07598dccd5
git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/swift15-mlx-lm.patch
git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/swift15-mlx-lm.patch
git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/enable-5bit.patch
git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/enable-5bit.patch
git -C swift5-mlx-lm apply --check ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch
git -C swift5-mlx-lm apply ../Swift-1.5-5bit-MLX/compatibility/swift15-server-cache.patch

Apple Silicon:

python -m pip install 'mlx==0.32.2' 'transformers==5.14.1' 'huggingface_hub==1.31.0' 'pillow==12.3.0'
python -m pip install -e ./swift5-mlx-lm

Linux CPU, Python 3.12, glibc 2.35 or newer:

python -m pip install 'mlx[cpu]==0.32.2' 'transformers==5.14.1' 'huggingface_hub==1.31.0' 'pillow==12.3.0'
python -m pip install -e ./swift5-mlx-lm

The historical Linux environment records Hub 1.32.0. The 1.31.0 pin above was separately installed in the independent macOS audit, where 18 synthetic patch tests passed. These environments are not identical and the tests did not load the full 27B model. The upstream code MIT notice is included separately from weight licenses.

Before generating, check the Python environment that will actually run the model:

python Swift-1.5-5bit-MLX/check_download.py Swift-1.5-5bit-MLX --runtime

This checks the complete download and selects the pinned patched architecture without loading weights. It does not check a separate GUI app's environment, available inference memory, or generated quality. See TROUBLESHOOTING.md if it fails or a server reports generation thread died.

Text generation

import mlx.core as mx
from mlx_lm import generate, load
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("Swift-1.5-5bit-MLX")
if mx.default_device() == mx.cpu:
    model.apply(lambda x: x.astype(mx.float32) if mx.issubdtype(x.dtype, mx.floating) else x)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Reply with exactly: Hello from Swift."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
mx.random.seed(20260922)
print(generate(model, tokenizer, prompt=prompt, max_tokens=32, sampler=make_sampler(temp=0)))

The CPU branch changes only in-memory floating types; packed UINT32 weights and files are unchanged. The historical build reported a Linux BF16 accumulation issue. Its original diagnostic file was not published. The separately executed macOS CPU/Metal diagnostic uses synthetic tensors: summing 8,192 ones gives 256 on CPU BF16 and 8,192 with FP32; Metal BF16/FP32 also give 8,192. It is not a new Linux or full-model generation test.

The template supports reasoning_effort="low", "medium", and "xhigh". Template support does not establish generated quality for those modes. The explicit model.mtp_logits step and model.visual encoder have historical component evidence. Integrated image/video chat and speculative generation are not implemented; unsupported multimodal generation must not be reported as working.

Server cache update

The supplied server patch keeps prompt reuse enabled with at most two retained entries, enforces a retained-cache byte budget at insertion, and defaults to one prompt and one decode stream with 512-token prefill steps. On Metal the default budget estimates headroom after model weights and workspace; an explicit --prompt-cache-bytes overrides it. --prompt-cache-size 2 means two entries, not two GB.

See SERVER_CACHE_UPDATE.md for installation, offline startup, validation and rollback. Existing installations must apply this runtime update and restart. No model-weight download or re-quantization is needed.

The retained-cache limit does not cap active-request or total process memory. An entry larger than the budget is evicted, so reuse depends on what fits. Use --prompt-cache-size 0 only when intentionally disabling reuse. The model's configured context length is not a promise that it fits every Mac.

Validation: 44 unit/upstream server tests and 72 offline HTTP requests passed across tiny 4-bit and 5-bit fixtures. These checks cover retention, warm reuse, streaming and sequential generation. They do not establish full-model long-context capacity or GUI integration.

Conversion provenance

Conversion is not part of installation. If separately authorized, use only the complete customized Swift BF16 source identified in QUANTIZATION_MANIFEST.json, with its 18 shards verified before conversion, the pinned patched converter, and affine / 5-bit / group size 64. Do not fill missing weights with base Qwen or another quantization. Existing quantized weights are unchanged by these packaging repairs.

Full-model Mac follow-up

Full-model follow-up validation is now recorded in FULL_MAC_VALIDATION.md: both complete checkpoints passed an 86k-token synthetic text conversation and two cached follow-ups on a 48 GiB M4 Pro using the patched server. GUI integration and other memory/context sizes remain outside that test.