Upload folder using huggingface_hub

README.md

@@ -35,6 +35,26 @@ MLX-optimized 3-bit quantization of [mistralai/Devstral-Small-2-24B-Instruct-251
 - **256K Context**: Supports up to 256K token context window
 - **Speculative Decoding Compatible**: Works with Mistral Tekken tokenizer draft models
 
+## Quantization Method
+
+This model was created using a **dequantize-requantize** workflow for optimal 3-bit quality:
+
+```bash
+# Step 1: Dequantize the MLX 4-bit model back to BF16
+mlx_lm.convert \
+  --hf-path mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-4Bit \
+  --mlx-path ./devstral-v2-bf16 \
+  --dequantize --dtype bfloat16
+
+# Step 2: Requantize from BF16 to 3-bit
+mlx_lm.convert \
+  --hf-path ./devstral-v2-bf16 \
+  --mlx-path ./devstral-v2-3bit \
+  --quantize --q-bits 3 --q-group-size 64
+```
+
+**Why this works**: Direct 3-bit quantization from the original HuggingFace BF16 weights produces degenerate output ("decay decay decay") due to weight distribution differences during the PyTorch-to-MLX conversion. Going through the MLX 4-bit model's dequantized BF16 preserves the weight structure that MLX inference requires.
+
 ## Usage
 
 ```python
@@ -56,6 +76,7 @@ tools = [{"type": "function", "function": {"name": "read_file", "description": "
 messages = [{"role": "user", "content": "Read the file main.py"}]
 prompt = tokenizer.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)
 response = generate(model, tokenizer, prompt=prompt, max_tokens=200)
+# Output: [TOOL_CALLS]read_file[ARGS]{"path": "main.py"}
 ```
 
 ### With Speculative Decoding
@@ -70,27 +91,20 @@ draft_model, _ = load("badmadrad/Mistral-Small-3.1-DRAFT-0.5B-MLX-4bit")
 # Draft model proposes tokens, main model verifies — faster generation
 ```
 
-## Quantization Details
-
-Converted using `mlx_lm.convert`:
-
-```bash
-mlx_lm.convert \
-  --hf-path mistralai/Devstral-Small-2-24B-Instruct-2512 \
-  --mlx-path ./devstral-v2-3bit \
-  --quantize --q-bits 3 --q-group-size 64
-```
-
-- **Method**: Affine quantization (3-bit with group size 64)
-- **Effective bits/weight**: ~3.5
-- **Memory savings**: ~75% vs BF16, ~20% vs 4-bit
-
 ## Hardware Requirements
 
 - Apple Silicon Mac (M1/M2/M3/M4)
-- Minimum 16 GB unified memory (recommended 32 GB+)
+- Minimum 16 GB unified memory
 - macOS 13.5+
 
+## Comparison
+
+| Variant | Size | Quality | Tool Calling |
+|---------|------|---------|-------------|
+| BF16 (original) | ~48 GB | Best | Yes |
+| 4-bit (mlx-community) | ~12 GB | Great | Yes |
+| **3-bit (this model)** | **~9.6 GB** | **Good** | **Yes** |
+
 ## License
 
 Apache 2.0 (same as base model)

generation_config.json

@@ -3,7 +3,5 @@
   "eos_token_id": 2,
   "max_length": 262144,
   "pad_token_id": 11,
-  "do_sample": true,
-  "temperature": 0.15,
   "transformers_version": "5.0.0.dev0"
 }

model-00001-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7225d1c69f6106601fdab9ebc22c9f1f60d333f7aeae97a77746bd280dde1bfa
+oid sha256:86938c62379f1c361c27356ee8aa4f1730032278f9acb2b8de86fa5f08397c85
 size 5326582403

model-00002-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:674190296975560d2f0860d52e03d1c71b9df52c3e3baaf3def71d87d54d7ee5
+oid sha256:05c48505b7530ff654777143594bd67f1a4ca8455cf71f021ec96ee7010b35cf
 size 4987111828