Instructions to use mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit")
config = load_config("mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mlx-community/Devstral-Small-2-24B-Instruct-2512-6bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

prince-canuma commited on Jul 4

Commit

83cb7e7

verified ·

1 Parent(s): 51b3928

Re-upload mistralai/Devstral-Small-2-24B-Instruct-2512 MLX 6bit

Browse files

Converted from mistralai/Devstral-Small-2-24B-Instruct-2512 at revision c599e8e56f3f9110e97f0dc0450ce248e3334d84 with the current mlx-vlm checkout. README.md is intentionally preserved from the existing repository card.

Files changed (7) hide show

config.json +78 -78
model-00001-of-00004.safetensors +2 -2
model-00002-of-00004.safetensors +2 -2
model-00003-of-00004.safetensors +2 -2
model-00004-of-00004.safetensors +2 -2
processor_config.json +0 -1
tokenizer_config.json +1 -1

config.json CHANGED Viewed

@@ -1,83 +1,83 @@
 {
-  "architectures": [
-    "Mistral3ForConditionalGeneration"
-  ],
-  "do_sample": true,
-  "dtype": "bfloat16",
-  "eos_token_id": 2,
-  "generation_config": {
-    "bos_token_id": 1,
     "do_sample": true,
     "eos_token_id": 2,
-    "max_length": 262144,
-    "pad_token_id": 11,
     "temperature": 0.15,
-    "transformers_version": "5.0.0.dev0"
-  },
-  "image_token_index": 10,
-  "model_type": "mistral3",
-  "multimodal_projector_bias": false,
-  "projector_hidden_act": "gelu",
-  "quantization": {
-    "bits": 6,
-    "group_size": 64,
-    "mode": "affine"
-  },
-  "quantization_config": {
-    "bits": 6,
-    "group_size": 64,
-    "mode": "affine"
-  },
-  "spatial_merge_size": 2,
-  "temperature": 0.15,
-  "text_config": {
-    "attention_dropout": 0.0,
-    "head_dim": 128,
-    "hidden_act": "silu",
-    "hidden_size": 5120,
-    "initializer_range": 0.02,
-    "intermediate_size": 32768,
-    "max_position_embeddings": 393216,
-    "model_type": "ministral3",
-    "num_attention_heads": 32,
-    "num_hidden_layers": 40,
-    "num_key_value_heads": 8,
-    "rms_norm_eps": 1e-05,
-    "rope_parameters": {
-      "beta_fast": 32.0,
-      "beta_slow": 1.0,
-      "factor": 48.0,
-      "llama_4_scaling_beta": 0.1,
-      "mscale": 1.0,
-      "mscale_all_dim": 1.0,
-      "original_max_position_embeddings": 8192,
-      "rope_theta": 100000000.0,
-      "rope_type": "yarn",
-      "type": "yarn"
     },
-    "sliding_window": null,
-    "use_cache": true,
-    "vocab_size": 131072
-  },
-  "tie_word_embeddings": false,
-  "transformers_version": "5.0.0.dev0",
-  "vision_config": {
-    "attention_dropout": 0.0,
-    "head_dim": 64,
-    "hidden_act": "silu",
-    "hidden_size": 1024,
-    "image_size": 1540,
-    "initializer_range": 0.02,
-    "intermediate_size": 4096,
-    "model_type": "pixtral",
-    "num_attention_heads": 16,
-    "num_channels": 3,
-    "num_hidden_layers": 24,
-    "patch_size": 14,
-    "rope_parameters": {
-      "rope_theta": 10000.0,
-      "rope_type": "default"
-    }
-  },
-  "vision_feature_layer": -1
-}

 {
+    "architectures": [
+        "Mistral3ForConditionalGeneration"
+    ],
     "do_sample": true,
+    "dtype": "bfloat16",
     "eos_token_id": 2,
+    "generation_config": {
+        "bos_token_id": 1,
+        "eos_token_id": 2,
+        "max_length": 262144,
+        "pad_token_id": 11,
+        "do_sample": true,
+        "temperature": 0.15,
+        "transformers_version": "5.0.0.dev0"
+    },
+    "image_token_index": 10,
+    "model_type": "mistral3",
+    "multimodal_projector_bias": false,
+    "projector_hidden_act": "gelu",
+    "quantization": {
+        "group_size": 64,
+        "bits": 6,
+        "mode": "affine"
+    },
+    "quantization_config": {
+        "group_size": 64,
+        "bits": 6,
+        "mode": "affine"
+    },
+    "spatial_merge_size": 2,
     "temperature": 0.15,
+    "text_config": {
+        "attention_dropout": 0.0,
+        "head_dim": 128,
+        "hidden_act": "silu",
+        "hidden_size": 5120,
+        "initializer_range": 0.02,
+        "intermediate_size": 32768,
+        "max_position_embeddings": 393216,
+        "model_type": "ministral3",
+        "num_attention_heads": 32,
+        "num_hidden_layers": 40,
+        "num_key_value_heads": 8,
+        "rms_norm_eps": 1e-05,
+        "rope_parameters": {
+            "beta_fast": 32.0,
+            "beta_slow": 1.0,
+            "factor": 48.0,
+            "llama_4_scaling_beta": 0.1,
+            "mscale": 1.0,
+            "mscale_all_dim": 1.0,
+            "original_max_position_embeddings": 8192,
+            "rope_theta": 100000000.0,
+            "rope_type": "yarn",
+            "type": "yarn"
+        },
+        "sliding_window": null,
+        "use_cache": true,
+        "vocab_size": 131072
+    },
+    "tie_word_embeddings": false,
+    "transformers_version": "5.0.0.dev0",
+    "vision_config": {
+        "attention_dropout": 0.0,
+        "head_dim": 64,
+        "hidden_act": "silu",
+        "hidden_size": 1024,
+        "image_size": 1540,
+        "initializer_range": 0.02,
+        "intermediate_size": 4096,
+        "model_type": "pixtral",
+        "num_attention_heads": 16,
+        "num_channels": 3,
+        "num_hidden_layers": 24,
+        "patch_size": 14,
+        "rope_parameters": {
+            "rope_theta": 10000.0,
+            "rope_type": "default"
+        }
     },
+    "vision_feature_layer": -1
+}

model-00001-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9c0b63e59e3c61690dd84e202e4488970ff4a0c54bd8bfbe29743b6274df6c83
-size 5350971224

 version https://git-lfs.github.com/spec/v1
+oid sha256:da919d9eaa410b8fa12ad3a88e7f5730411343041a2adaa00eb09373a79787d7
+size 5350971296

model-00002-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9b3bd066a2904bc27a96fffbc49286c3991ddc669c8af2d5d85d4c437c0fbee9
-size 5282482723

 version https://git-lfs.github.com/spec/v1
+oid sha256:953d5833791022803cd35abd085db5251b796357e85fc9bc09ff7d231139985f
+size 5282482737

model-00003-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9155cf054849b3118115b09fb1d8d99896e34874cd46f29199a288609c31274d
-size 5282482779

 version https://git-lfs.github.com/spec/v1
+oid sha256:e1d66c6c3cb9c66dd46b03dd93106dabd5e6f7db9d294e5941e6682797c44c19
+size 5282482745

model-00004-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4ab074979fb116c24944dc80e0f86e2cade8af2044fb1a455fd95923ed411d15
-size 4912119780

 version https://git-lfs.github.com/spec/v1
+oid sha256:f00c93f6969feee62fc76fa75b2f671313f95ff6e5ace66e69342fb9a8939b3b
+size 4912119770

processor_config.json CHANGED Viewed

@@ -24,7 +24,6 @@
     ],
     "pad_size": null,
     "patch_size": 14,
-    "processor_class": "Mistral3Processor",
     "resample": 3,
     "rescale_factor": 0.00392156862745098,
     "size": {

     ],
     "pad_size": null,
     "patch_size": 14,
     "resample": 3,
     "rescale_factor": 0.00392156862745098,
     "size": {

tokenizer_config.json CHANGED Viewed

@@ -1,5 +1,4 @@
 {
-  "additional_special_tokens": null,
   "backend": "tokenizers",
   "bos_token": "<s>",
   "eos_token": "</s>",
@@ -1006,6 +1005,7 @@
     "<SPECIAL_999>"
   ],
   "is_local": true,
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "<pad>",
   "processor_class": "Mistral3Processor",

 {
   "backend": "tokenizers",
   "bos_token": "<s>",
   "eos_token": "</s>",
     "<SPECIAL_999>"
   ],
   "is_local": true,
+  "local_files_only": false,
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "<pad>",
   "processor_class": "Mistral3Processor",