Instructions to use RozGrov/NemoDori-v0.2.1-12B-MN-BT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RozGrov/NemoDori-v0.2.1-12B-MN-BT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RozGrov/NemoDori-v0.2.1-12B-MN-BT")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RozGrov/NemoDori-v0.2.1-12B-MN-BT")
model = AutoModelForCausalLM.from_pretrained("RozGrov/NemoDori-v0.2.1-12B-MN-BT", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RozGrov/NemoDori-v0.2.1-12B-MN-BT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RozGrov/NemoDori-v0.2.1-12B-MN-BT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RozGrov/NemoDori-v0.2.1-12B-MN-BT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RozGrov/NemoDori-v0.2.1-12B-MN-BT

SGLang

How to use RozGrov/NemoDori-v0.2.1-12B-MN-BT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RozGrov/NemoDori-v0.2.1-12B-MN-BT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RozGrov/NemoDori-v0.2.1-12B-MN-BT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RozGrov/NemoDori-v0.2.1-12B-MN-BT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RozGrov/NemoDori-v0.2.1-12B-MN-BT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RozGrov/NemoDori-v0.2.1-12B-MN-BT with Docker Model Runner:
```
docker model run hf.co/RozGrov/NemoDori-v0.2.1-12B-MN-BT
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

NemoDori-v0.2.1-12B-MN-BT

This is a merge of pre-trained language models created using mergekit.

The first child from NemoDori-v0.2-12B-MN-BT.

The purpose is to find a way to increase v0.2 capability to stay aware of the past conversations and follow instructions better, especially the last one (depth-0), while keeping it's creativity and capability to (E)RP. This model is one of the few childs to try to fulfill that.

In my very short testing so far, I haven't found anything that's different from the parent and worth mentioning. But I think this version is slightly degraded somehow, (I don't quite know it, I just felt like it did). Anyway, try it as you may, but I think it's parent (v0.2) is better than this one.

The other child (v0.2.2) is out. I tested it more than this model and it seems to be improved better than this model, but the response format is not very consistent.

You may give me feedback on anything, or guide me how I can fulfill my-ahem it's purpose while keeping it as low as not-70B.
Fine-tune is... pretty expensive for me, and I'm not ready for that (yet, tho i'm interested).

(listen, between you and me, i still don't get it. still learning this new hobby of mine, and it's kind of refreshing in a way. i'll be exploring more other architectures in the future. Yet, this is about how random i pick my straw, just to see how lucky i am.)
(although, i am interested to learn how to make a new merge method. similar to when i'm making a solution for solving specific problem just like good ol days. but hell, this llm stuff is really expensive.)

Merge Details

Merge Method

This model was merged using the breadcrumbs_ties merge method using RozGrov/NemoDori-v0.2-12B-MN-BT as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:


models:
  - model: crestf411/nemo-sunfall-v0.6.1
    parameters:
      weight: 0.33
  - model: UsernameJustAnother/Nemo-12B-Marlin-v5
    parameters:
      weight: 0.2
  - model: unsloth/Mistral-Nemo-Instruct-2407
    parameters:
      weight: 0.37
  - model: RozGrov/NemoDori-v0.2-12B-MN-BT
    parameters:
      weight: 1
merge_method: breadcrumbs_ties
base_model: RozGrov/NemoDori-v0.2-12B-MN-BT
parameters:
    density: 0.93
    gamma: 0.015
dtype: bfloat16

Downloads last month: 12

Safetensors

Model size

12B params

Tensor type

BF16

Model tree for RozGrov/NemoDori-v0.2.1-12B-MN-BT

RozGrov/NemoDori-v0.2-12B-MN-BT

UsernameJustAnother/Nemo-12B-Marlin-v5

crestf411/nemo-sunfall-v0.6.1

unsloth/Mistral-Nemo-Instruct-2407

Merge model

this model

Quantizations

4 models