Instructions to use alamios/Mistral-Small-3.1-DRAFT-0.5B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use alamios/Mistral-Small-3.1-DRAFT-0.5B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="alamios/Mistral-Small-3.1-DRAFT-0.5B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("alamios/Mistral-Small-3.1-DRAFT-0.5B")
model = AutoModelForCausalLM.from_pretrained("alamios/Mistral-Small-3.1-DRAFT-0.5B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use alamios/Mistral-Small-3.1-DRAFT-0.5B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "alamios/Mistral-Small-3.1-DRAFT-0.5B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alamios/Mistral-Small-3.1-DRAFT-0.5B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/alamios/Mistral-Small-3.1-DRAFT-0.5B

SGLang

How to use alamios/Mistral-Small-3.1-DRAFT-0.5B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "alamios/Mistral-Small-3.1-DRAFT-0.5B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alamios/Mistral-Small-3.1-DRAFT-0.5B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "alamios/Mistral-Small-3.1-DRAFT-0.5B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alamios/Mistral-Small-3.1-DRAFT-0.5B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use alamios/Mistral-Small-3.1-DRAFT-0.5B with Docker Model Runner:
```
docker model run hf.co/alamios/Mistral-Small-3.1-DRAFT-0.5B
```

Thanks for releasing the non-GGUF version!

by rdsm - opened Mar 24, 2025

Discussion

rdsm

Mar 24, 2025

Thanks for releasing the safetensors version!

alamios

Owner Mar 24, 2025

you're welcome!

urtuuuu

Mar 24, 2025

Is there something similar for gemma 3 27b? I actually tried gemma3 1B model as draft in lm studio, but it works even slower than without it. Maybe it's because of my slow hardware.

alamios

Owner Mar 24, 2025

I'll try, but I don't think a light finetune would work better than their model. Anyway, stay tuned!

ElvisM

Mar 25, 2025

I don't think it works, but I wish this draft also worked with the first Mistral Small release. I dislike the 3.1 version for creative writing - it's just too boring. Also haven't had any luck using Gemma 1b as a draft model.

rdsm

Mar 25, 2025

@ElvisM are you talking the 2501 or the 2409? I haven't tested the 2409 but I believe it works fine with the 2501.

ElvisM

Mar 26, 2025

@ElvisM are you talking the 2501 or the 2409? I haven't tested the 2409 but I believe it works fine with the 2501.

Tested and confirmed that it doesn't work. LM Studio doesn't even allow to select it.

rdsm

Mar 29, 2025

@ElvisM you can probably do a transplant.... Actually.. I have it all set up here, I just created a QwenPhi 0.5b, I will try here to make a transplant for the 2409 vocab for this draft model here.

I love this @alamios idea... it is working great here.

rdsm

Mar 29, 2025

@ElvisM actually I am not so sure if I can actually do it... Because the 2409 is distributed under the mrl licence and the Qwen 2.5 is apache, I am trying to understand the compatibility of the mrl with the apache-2 license. I think if I include both licences and distribute under the mrl it is ok , but I will give it another read tomorrow, before releasing it.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment