Instructions to use kromvault/L3.1-Siithamo-v0.1-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kromvault/L3.1-Siithamo-v0.1-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kromvault/L3.1-Siithamo-v0.1-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kromvault/L3.1-Siithamo-v0.1-8B")
model = AutoModelForCausalLM.from_pretrained("kromvault/L3.1-Siithamo-v0.1-8B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kromvault/L3.1-Siithamo-v0.1-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kromvault/L3.1-Siithamo-v0.1-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kromvault/L3.1-Siithamo-v0.1-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kromvault/L3.1-Siithamo-v0.1-8B

SGLang

How to use kromvault/L3.1-Siithamo-v0.1-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kromvault/L3.1-Siithamo-v0.1-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kromvault/L3.1-Siithamo-v0.1-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kromvault/L3.1-Siithamo-v0.1-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kromvault/L3.1-Siithamo-v0.1-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use kromvault/L3.1-Siithamo-v0.1-8B with Docker Model Runner:
```
docker model run hf.co/kromvault/L3.1-Siithamo-v0.1-8B
```

L3.1-Siithamo-v0.1-8B / README.md

kromeurus

Update README.md

2ae9c3b verified about 2 years ago

preview code

Raw

History Blame Contribute Delete

2.7 kB

metadata

base_model:
  - ArliAI/ArliAI-Llama-3-8B-Formax-v1.0
  - Sao10K/L3.1-8B-Niitama-v1.1
  - Sao10K/L3-8B-Tamamo-v1
  - Sao10K/L3-8B-Stheno-v3.3-32K
  - Edgerunners/Lyraea-large-llama-3.1
library_name: transformers
tags:
  - mergekit
  - merge

My first foray into Llama 3.1 and just having fun with the merging process. Testing theories and such.

Updated version with higher context here.

Quants

OG Q8 GGUF by me.

Details & Recommended Settings

Unfortunaely, this model still double lines but its not as often. Dramatic as fuck at times. I haven't tested the context limit yet but I'm sure it suffered somehow.

Outputs a lot, pretty chatty like Stheno. Pulls some chaotic creativity from Niitama but its mellowed out with Tamamo. A little cliche writing, but it's almost endearing in a way. Should follow instructs fine? Stunted a little compared to the original model, don't think that's a negative though.

4K Max context even on L3.1 (DAMN U FORMAX)

Rec. Settings:

Template: L3
Temperature: 1.35
Min P: 0.1
Repeat Penalty: 1.05
Repeat Penalty Tokens: 256

Models Merged & Merge Theory

The following models were included in the merge:

Using Edgerunners Lyraea as the 3.1 base, model stock mereged L3.1 Niitama, Stheno 3.3, and Tamamo a top each other. Then trying to curb L3 tendencies and add some instruct following capabilities, added some Formax in a dare linear merge. At least for updating L3 to L3.1, doing TIES anything results in a 'shittier' model.

Config

models:
    - model: Sao10K/L3.1-8B-Niitama-v1.1
    - model: Sao10K/L3-8B-Stheno-v3.3-32K
    - model: Sao10K/L3-8B-Tamamo-v1
base_model: Edgerunners/Lyraea-large-llama-3.1
parameters:
  normalize: false
  int8_mask: true
merge_method: model_stock
dtype: float32
out_dtype: bfloat16
name: siitamol3.1
---
models: 
  - model: ArliAI/ArliAI-Llama-3-8B-Formax-v1.0
    parameters:
      weight: [0.5, 0.3, 0.2, 0.1]
  - model: siitamol3.1
    parameters:
      weight: [0.5, 0.7, 0.8, 1]
base_model: siitamol3.1
parameters:
  normalize: false
  int8_mask: true
merge_method: dare_linear
dtype: float32
out_dtype: bfloat16