Instructions to use JetBrains/Mellum2-12B-A2.5B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JetBrains/Mellum2-12B-A2.5B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="JetBrains/Mellum2-12B-A2.5B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("JetBrains/Mellum2-12B-A2.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained("JetBrains/Mellum2-12B-A2.5B-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JetBrains/Mellum2-12B-A2.5B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JetBrains/Mellum2-12B-A2.5B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JetBrains/Mellum2-12B-A2.5B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/JetBrains/Mellum2-12B-A2.5B-Instruct

SGLang

How to use JetBrains/Mellum2-12B-A2.5B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JetBrains/Mellum2-12B-A2.5B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JetBrains/Mellum2-12B-A2.5B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JetBrains/Mellum2-12B-A2.5B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JetBrains/Mellum2-12B-A2.5B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use JetBrains/Mellum2-12B-A2.5B-Instruct with Docker Model Runner:
```
docker model run hf.co/JetBrains/Mellum2-12B-A2.5B-Instruct
```

Mellum2-12B-A2.5B-Instruct

Commit History

Fix eos_token_id: <|im_end|> = 28 (was 0, the pretrain default)

273d9c1
verified

Ar4l commited on 24 days ago

Fix eos_token_id: <|im_end|> = 28 (was 0, the pretrain default)

571df75
verified

Ar4l commited on 24 days ago

Fix broken SWA

717997f
verified

mkannan-jb commited on Jun 12

Update README.md

4ee5751
verified

pavlichenko commited on Jun 1

Update README.md

85d8ab8
verified

pavlichenko commited on May 29

Add post-training evaluation comparison table

376e5ce
verified

kseniialysaniuk commited on May 29

Drop duplicate <picture> block; use dark logo only

1a9b4cf
verified

kseniialysaniuk commited on May 29

Add logo to model card

a5bc4e8
verified

kseniialysaniuk commited on May 29

Add logo, use case note, vLLM serving, and Quickstart (#4)

b52eac2

pavlichenko

kseniialysaniuk commited on May 28

Update model card and add evaluation results (#2)

88fe4ad

pavlichenko

kseniialysaniuk commited on May 28

Finalize config (#1)

19a1c28

pavlichenko

mkannan-jb commited on May 26

Add files using upload-large-folder tool

6340f8d
verified

pavlichenko commited on May 26

initial commit

33a7888
verified

pavlichenko commited on May 26

Commit History

Fix eos_token_id: <|im_end|> = 28 (was 0, the pretrain default) 273d9c1 verified

Fix eos_token_id: <|im_end|> = 28 (was 0, the pretrain default) 571df75 verified

Fix broken SWA 717997f verified

Update README.md 4ee5751 verified

Update README.md 85d8ab8 verified

Add post-training evaluation comparison table 376e5ce verified

Drop duplicate <picture> block; use dark logo only 1a9b4cf verified

Add logo to model card a5bc4e8 verified

Add logo, use case note, vLLM serving, and Quickstart (#4) b52eac2

Update model card and add evaluation results (#2) 88fe4ad

Finalize config (#1) 19a1c28

Add files using upload-large-folder tool 6340f8d verified

initial commit 33a7888 verified

Fix eos_token_id: <|im_end|> = 28 (was 0, the pretrain default)

273d9c1
verified

Fix eos_token_id: <|im_end|> = 28 (was 0, the pretrain default)

571df75
verified

Fix broken SWA

717997f
verified

Update README.md

4ee5751
verified

Update README.md

85d8ab8
verified

Add post-training evaluation comparison table

376e5ce
verified

Drop duplicate <picture> block; use dark logo only

1a9b4cf
verified

Add logo to model card

a5bc4e8
verified

Add logo, use case note, vLLM serving, and Quickstart (#4)

b52eac2

Update model card and add evaluation results (#2)

88fe4ad

Finalize config (#1)

19a1c28

Add files using upload-large-folder tool

6340f8d
verified

initial commit

33a7888
verified