Instructions to use openGPT-X/Teuken-7B-instruct-research-v0.4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use openGPT-X/Teuken-7B-instruct-research-v0.4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="openGPT-X/Teuken-7B-instruct-research-v0.4")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("openGPT-X/Teuken-7B-instruct-research-v0.4") model = AutoModelForCausalLM.from_pretrained("openGPT-X/Teuken-7B-instruct-research-v0.4", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use openGPT-X/Teuken-7B-instruct-research-v0.4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "openGPT-X/Teuken-7B-instruct-research-v0.4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "openGPT-X/Teuken-7B-instruct-research-v0.4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/openGPT-X/Teuken-7B-instruct-research-v0.4
- SGLang
How to use openGPT-X/Teuken-7B-instruct-research-v0.4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "openGPT-X/Teuken-7B-instruct-research-v0.4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "openGPT-X/Teuken-7B-instruct-research-v0.4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "openGPT-X/Teuken-7B-instruct-research-v0.4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "openGPT-X/Teuken-7B-instruct-research-v0.4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use openGPT-X/Teuken-7B-instruct-research-v0.4 with Docker Model Runner:
docker model run hf.co/openGPT-X/Teuken-7B-instruct-research-v0.4
Difference to commercial? And why two licenses?
What is the exact difference between this model and the one named commercial?
I do see that it is non-commerical here and Apache 2.0 in commercial, but the benchmark shows slightly different values and it also seems odd to do a dual-licensing between a non-commercial license and Apache 2.0 which seems to allow the same.
Yeah I had the same question. What are the differences? And why is this the research version, if usage is so heavily restricted?
What is the exact difference between this model and the one named commercial?
I do see that it is non-commerical here and Apache 2.0 in commercial, but the benchmark shows slightly different values and it also seems odd to do a dual-licensing between a non-commercial license and Apache 2.0 which seems to allow the same.
Thank you for asking. Florian and I discussed the issue here (please feel free to join).
https://discord.com/channels/1280881494029500499/1310964954014285854/1311401599293456515
With regards to performance you can check out our leaderboard :)
https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard
Background: the instruction tuning dataset for the research version does not allow for commercial usage. We are working on an multilingual instruction tuning dataset that performs optimally and permits commercial usage.
Yeah I had the same question. What are the differences? And why is this the research version, if usage is so heavily restricted?
We are working on an instruction tuning dataset that will perform as good as the research one. Personally, I am also not happy that modification etc is not permitted etc.