Instructions to use mlabonne/Beyonder-4x7B-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlabonne/Beyonder-4x7B-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mlabonne/Beyonder-4x7B-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mlabonne/Beyonder-4x7B-v2")
model = AutoModelForCausalLM.from_pretrained("mlabonne/Beyonder-4x7B-v2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mlabonne/Beyonder-4x7B-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mlabonne/Beyonder-4x7B-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlabonne/Beyonder-4x7B-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mlabonne/Beyonder-4x7B-v2

SGLang

How to use mlabonne/Beyonder-4x7B-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mlabonne/Beyonder-4x7B-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlabonne/Beyonder-4x7B-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mlabonne/Beyonder-4x7B-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mlabonne/Beyonder-4x7B-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mlabonne/Beyonder-4x7B-v2 with Docker Model Runner:
```
docker model run hf.co/mlabonne/Beyonder-4x7B-v2
```

Wrong prompt format in tokenizer_config.json?

by wolfram - opened Jan 22, 2024

Discussion

wolfram

Jan 22, 2024

The chat_template specified in tokenizer_config.json is ChatML, but apparently this model uses the (weird) GPT4 Correct prompt format. Please clarify which is the correct prompt format/chat template and kindly state it on the model card, and make sure tokenizer_config.json also has the proper template. Thank you!

mlabonne

Owner Jan 22, 2024

Hi @wolfram thanks for testing this model. I think you used an old GGUF version from TheBloke with the previous wrong tokenizer_config.json. I added an "added_tokens.json" file, maybe this helps.

If not, can you detail what changes I should make? Thanks.

wolfram

Jan 22, 2024

•

edited Jan 25, 2024

@mlabonne What's the actual chat template? In your tokenizer_config.json, the chat_template is set to ChatML, but the models your mix is made of are using a GPT4 Correct prompt format. How do you prompt it properly?

I used TheBloke's GGUF because the HF version crashed with the error message "RuntimeError: CUDA error: device-side assert triggered". Is that a known issue or just a problem on my end?

mlabonne

Owner Jan 27, 2024

Yeah, I managed to make it work with ChatML without any issues but it looks like this depends on your config. There's no pre-defined chat template. As you said, this is a merge of several models that use the GPT4 Correct prompt format, but these tokens are not implemented. I tried a few configs and I'm opting for a modified GPT4 Correct prompt format with a different eos token. I believe it's the best solution but I haven't tested it thoroughly. The CUDA error is also fixed.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment