---
library_name: mlx
license: apache-2.0
base_model: mistralai/Devstral-Small-2-24B-Instruct-2512
tags:
  - mlx
  - mistral
  - devstral
  - code
  - tool-use
  - agentic
  - quantized
  - apple-silicon
language:
  - en
pipeline_tag: text-generation
---

# Devstral Small 2 24B Instruct (MLX 3-bit)

MLX-optimized 3-bit quantization of [mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512) for Apple Silicon Macs.

## Model Details

- **Base Model**: Devstral Small 2 24B Instruct (December 2512 release)
- **Architecture**: Mistral3 (40 layers, GQA)
- **Quantization**: 3-bit affine, group size 64 (~3.5 bits/weight effective)
- **Size**: ~9.6 GB (vs ~48 GB BF16, ~12 GB 4-bit)
- **Framework**: [MLX](https://github.com/ml-explore/mlx) via [mlx-lm](https://github.com/ml-explore/mlx-examples)

## Key Features

- **Native Tool Calling**: Full `[AVAILABLE_TOOLS]` / `[TOOL_CALLS]` / `[TOOL_RESULTS]` support
- **Agentic Coding**: Designed for code generation, editing, and multi-step tool use
- **256K Context**: Supports up to 256K token context window
- **Speculative Decoding Compatible**: Works with Mistral Tekken tokenizer draft models

## Quantization Method

This model was created using a **dequantize-requantize** workflow for optimal 3-bit quality:

```bash
# Step 1: Dequantize the MLX 4-bit model back to BF16
mlx_lm.convert \
  --hf-path mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-4Bit \
  --mlx-path ./devstral-v2-bf16 \
  --dequantize --dtype bfloat16

# Step 2: Requantize from BF16 to 3-bit
mlx_lm.convert \
  --hf-path ./devstral-v2-bf16 \
  --mlx-path ./devstral-v2-3bit \
  --quantize --q-bits 3 --q-group-size 64
```

**Why this works**: Direct 3-bit quantization from the original HuggingFace BF16 weights produces degenerate output ("decay decay decay") due to weight distribution differences during the PyTorch-to-MLX conversion. Going through the MLX 4-bit model's dequantized BF16 preserves the weight structure that MLX inference requires.

## Usage

```python
from mlx_lm import load, generate

model, tokenizer = load("badmadrad/Devstral-Small-2-24B-Instruct-2512-MLX-3bit")

messages = [{"role": "user", "content": "Write a Python function to sort a list."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, max_tokens=500)
print(response)
```

### With Tool Calling

```python
tools = [{"type": "function", "function": {"name": "read_file", "description": "Read a file", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}}]

messages = [{"role": "user", "content": "Read the file main.py"}]
prompt = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, max_tokens=200)
# Output: [TOOL_CALLS]read_file[ARGS]{"path": "main.py"}
```

### With Speculative Decoding

```python
from mlx_lm import load
from mlx_lm.generate import speculative_generate_step

model, tokenizer = load("badmadrad/Devstral-Small-2-24B-Instruct-2512-MLX-3bit")
draft_model, _ = load("badmadrad/Mistral-Small-3.1-DRAFT-0.5B-MLX-4bit")

# Draft model proposes tokens, main model verifies — faster generation
```

## Hardware Requirements

- Apple Silicon Mac (M1/M2/M3/M4)
- Minimum 16 GB unified memory
- macOS 13.5+

## Comparison

| Variant | Size | Quality | Tool Calling |
|---------|------|---------|-------------|
| BF16 (original) | ~48 GB | Best | Yes |
| 4-bit (mlx-community) | ~12 GB | Great | Yes |
| **3-bit (this model)** | **~9.6 GB** | **Good** | **Yes** |

## License

Apache 2.0 (same as base model)