Instructions to use eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M

Use Docker

docker model run hf.co/eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M

Ollama
How to use eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF with Ollama:
```
ollama run hf.co/eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF with Docker Model Runner:
```
docker model run hf.co/eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M
```

Lemonade

How to use eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Mistral-Small-3.2-24B-Instruct-2506-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

eaddario commited on Jul 18, 2025

Commit

cb70951

verified ·

1 Parent(s): 8c916d1

Generate Perplexity, KLD, ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores

Browse files

Files changed (5) hide show

scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.arc +21 -0
scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.hsw +20 -0
scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.mmlu +21 -0
scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.ppx +37 -0
scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.tqa +21 -0

scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.arc ADDED Viewed

	@@ -0,0 +1,21 @@

+build: 5900 (10a0351a) with Apple clang version 17.0.0 (clang-1700.0.13.3) for arm64-apple-darwin24.4.0
+llama_model_load_from_file_impl: using device Metal (Apple M4 Max) - 49151 MiB free
+llama_model_loader: loaded meta data with 46 key-value pairs and 363 tensors from ./Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf (version GGUF V3 (latest))
+Final result: 69.0667 +/- 1.6889
+Random chance: 25.0083 +/- 1.5824
+llama_perf_context_print:        load time =    5870.95 ms
+llama_perf_context_print: prompt eval time =  172816.96 ms / 36666 tokens (    4.71 ms per token,   212.17 tokens per second)
+llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_perf_context_print:       total time =  173679.86 ms / 36667 tokens
+ggml_metal_free: deallocating
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)

scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.hsw ADDED Viewed

	@@ -0,0 +1,20 @@

+build: 5900 (10a0351a) with Apple clang version 17.0.0 (clang-1700.0.13.3) for arm64-apple-darwin24.4.0
+llama_model_load_from_file_impl: using device Metal (Apple M4 Max) - 49151 MiB free
+llama_model_loader: loaded meta data with 46 key-value pairs and 363 tensors from ./Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf (version GGUF V3 (latest))
+750	83.33333333%	[80.4977%, 85.8293%]
+llama_perf_context_print:        load time =     901.73 ms
+llama_perf_context_print: prompt eval time =  611981.24 ms / 129319 tokens (    4.73 ms per token,   211.31 tokens per second)
+llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_perf_context_print:       total time =  615620.44 ms / 129320 tokens
+ggml_metal_free: deallocating
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)

scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.mmlu ADDED Viewed

	@@ -0,0 +1,21 @@

+build: 5900 (10a0351a) with Apple clang version 17.0.0 (clang-1700.0.13.3) for arm64-apple-darwin24.4.0
+llama_model_load_from_file_impl: using device Metal (Apple M4 Max) - 49151 MiB free
+llama_model_loader: loaded meta data with 46 key-value pairs and 363 tensors from ./Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf (version GGUF V3 (latest))
+Final result: 45.0667 +/- 1.8180
+Random chance: 25.0000 +/- 1.5822
+llama_perf_context_print:        load time =     950.33 ms
+llama_perf_context_print: prompt eval time =  323050.25 ms / 68956 tokens (    4.68 ms per token,   213.45 tokens per second)
+llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_perf_context_print:       total time =  324339.12 ms / 68957 tokens
+ggml_metal_free: deallocating
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)

scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.ppx ADDED Viewed

	@@ -0,0 +1,37 @@

+====== Perplexity statistics ======
+Mean PPL(Q)                   :   5.288988 ±   0.030482
+Mean PPL(base)                :   5.056724 ±   0.028289
+Cor(ln(PPL(Q)), ln(PPL(base))):  98.74%
+Mean ln(PPL(Q)/PPL(base))     :   0.044908 ±   0.000917
+Mean PPL(Q)/PPL(base)         :   1.045932 ±   0.000960
+Mean PPL(Q)-PPL(base)         :   0.232265 ±   0.005153
+====== KL divergence statistics ======
+Mean    KLD:   0.046663 ±   0.000405
+Maximum KLD:   8.698647
+99.9%   KLD:   2.124999
+99.0%   KLD:   0.576927
+99.0%   KLD:   0.576927
+Median  KLD:   0.018407
+10.0%   KLD:   0.000476
+ 5.0%   KLD:   0.000131
+ 1.0%   KLD:   0.000017
+Minimum KLD:  -0.000061
+====== Token probability statistics ======
+Mean    Δp: -0.077 ± 0.018 %
+Maximum Δp: 93.922%
+99.9%   Δp: 37.396%
+99.0%   Δp: 16.027%
+95.0%   Δp:  7.966%
+90.0%   Δp:  5.036%
+75.0%   Δp:  1.441%
+Median  Δp:  0.021%
+25.0%   Δp: -0.801%
+10.0%   Δp: -4.350%
+ 5.0%   Δp: -8.091%
+ 1.0%   Δp: -24.916%
+ 0.1%   Δp: -70.360%
+Minimum Δp: -99.157%
+RMS Δp    :  6.914 ± 0.057 %
+Same top p: 91.235 ± 0.073 %

scores/Mistral-Small-3.2-24B-Instruct-2506-q4_k_m.tqa ADDED Viewed

	@@ -0,0 +1,21 @@

+build: 5900 (10a0351a) with Apple clang version 17.0.0 (clang-1700.0.13.3) for arm64-apple-darwin24.4.0
+llama_model_load_from_file_impl: using device Metal (Apple M4 Max) - 49151 MiB free
+llama_model_loader: loaded meta data with 46 key-value pairs and 363 tensors from ./Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf (version GGUF V3 (latest))
+Final result: 37.4667 +/- 1.7686
+Random chance: 19.8992 +/- 1.4588
+llama_perf_context_print:        load time =     942.33 ms
+llama_perf_context_print: prompt eval time =  246605.90 ms / 51053 tokens (    4.83 ms per token,   207.02 tokens per second)
+llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_perf_context_print:       total time =  248227.45 ms / 51054 tokens
+ggml_metal_free: deallocating
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)
+ggml_metal_mem_pool_free: freeing memory pool, num heaps = 0 (total = 0)