Update README for swiss-ai/Apertus-v1.1-0.5B-Instruct-MLX-INT3

README.md

@@ -54,7 +54,7 @@ Instead of standard pre-training, Apertus-v1.1 models were created using pre-tra
 
 This model family includes base pre-trained models and instruction-tuned models.
 
-For instruction-tuned models, we additionally provide high-quality quantization-aware distillation (QAD) checkpoints, obtained via the official [`qat-suite`](https://github.com/swiss-ai/qat-suite). We provide FP8 and NVFP4A16 checkpoints with vLLM inference in mind and INT3-6 checkpoints optimized for mobile usage on Apple devices.
+For instruction-tuned models, we additionally provide high-quality quantization-aware distillation (QAD) checkpoints, obtained via the official [`qat-suite`]([https://github.com/swiss-ai/qat-suite](https://github.com/swiss-ai/qat-suite)). We provide FP8 and NVFP4A16 checkpoints with vLLM inference in mind and INT3-6 checkpoints optimized for mobile usage on Apple devices.
 
 The full list of released checkpoints is shown below:
 
@@ -73,22 +73,18 @@ For more details refer to the original Apertus [technical report](https://arxiv.
 
 ## How to use
 
-The modeling code for Apertus is available in transformers `v4.56.0` and later, so make sure to upgrade your transformers version. You can also load the model with the latest `vLLM` which uses transformers as a backend.
+The modeling code for this Apertus MLX quantization is available via `mlx-lm`.
 ```bash
-pip install -U transformers
+pip install mlx-lm
 ```
 
 ```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
+from mlx_lm import load, generate
 
 model_name = "swiss-ai/Apertus-v1.1-0.5B-Instruct-MLX-INT3"
-device = "cuda"  # for GPU usage or "cpu" for CPU usage
 
-# load the tokenizer and the model
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(
-    model_name,
-).to(device)
+# load the model and tokenizer
+model, tokenizer = load(model_name)
 
 # prepare the model input
 prompt = "Give me a brief explanation of gravity in simple terms."
@@ -101,14 +97,9 @@ text = tokenizer.apply_chat_template(
     tokenize=False,
     add_generation_prompt=True,
 )
-model_inputs = tokenizer([text], return_tensors="pt", add_special_tokens=False).to(model.device)
 
 # Generate the output
-generated_ids = model.generate(**model_inputs, max_new_tokens=32768)
-
-# Get and decode the output
-output_ids = generated_ids[0][len(model_inputs.input_ids[0]) :]
-print(tokenizer.decode(output_ids, skip_special_tokens=True))
+response = generate(model, tokenizer, prompt=text, verbose=True, max_tokens=32768)
 ```
 
 >[!TIP]