Instructions to use teknium/Mistral-Trismegistus-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use teknium/Mistral-Trismegistus-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="teknium/Mistral-Trismegistus-7B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("teknium/Mistral-Trismegistus-7B")
model = AutoModelForCausalLM.from_pretrained("teknium/Mistral-Trismegistus-7B", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use teknium/Mistral-Trismegistus-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "teknium/Mistral-Trismegistus-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "teknium/Mistral-Trismegistus-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/teknium/Mistral-Trismegistus-7B

SGLang

How to use teknium/Mistral-Trismegistus-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "teknium/Mistral-Trismegistus-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "teknium/Mistral-Trismegistus-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "teknium/Mistral-Trismegistus-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "teknium/Mistral-Trismegistus-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use teknium/Mistral-Trismegistus-7B with Docker Model Runner:
```
docker model run hf.co/teknium/Mistral-Trismegistus-7B
```

Great! one thing.. FIX for: going on after responses with Q&A pairs ie: 'USER:" ...

by quantumalchemy - opened Oct 16, 2023

Discussion

quantumalchemy

Oct 16, 2023

add --> stop=["USER:"]
in your params
GREAT WORK! THANKS!

teknium

Owner Oct 16, 2023

add --> stop=["USER:"]
in your params
GREAT WORK! THANKS!

Does it have trouble keeping it's turns seperated

quantumalchemy changed discussion title from Great! one thing.. FIX for: going on after responces with Q&A pairs ie: 'USER:" ... to Great! one thing.. FIX for: going on after responses with Q&A pairs ie: 'USER:" ... Oct 16, 2023

ikci

Oct 16, 2023

add --> stop=["USER:"]
in your params
GREAT WORK! THANKS!

Does it have trouble keeping it's turns seperated

I'm 90% sure that stuff is handled by the frontend one uses when running the AI, not by the model

quantumalchemy

Oct 18, 2023

most likely cause because I am running this llm with the 'hammer down'
max_tokens=32768
no / max_new_tokens
only seems to occur on short responses to to simple questions.. ie "hello", "who are you" ..
ask something specific and its very good.
cant wait to see the the dataset on this thing - Just fantastic!

quantumalchemy changed discussion status to closed Oct 18, 2023

quantumalchemy changed discussion status to open Oct 18, 2023

teknium

Owner Oct 18, 2023

most likely cause because I am running this llm with the 'hammer down'
max_tokens=32768
no / max_new_tokens
only seems to occur on short responses to to simple questions.. ie "hello", "who are you" ..
ask something specific and its very good.
cant wait to see the the dataset on this thing - Just fantastic!

fyi the dataset is released :) https://huggingface.co/datasets/teknium/trismegistus-project

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment