Instructions to use mistralai/Mistral-7B-Instruct-v0.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mistralai/Mistral-7B-Instruct-v0.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mistralai/Mistral-7B-Instruct-v0.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Install mistral-common:
pip install --upgrade mistral-common
# Start the vLLM server:
vllm serve "mistralai/Mistral-7B-Instruct-v0.2" --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.2

SGLang

How to use mistralai/Mistral-7B-Instruct-v0.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mistralai/Mistral-7B-Instruct-v0.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mistralai/Mistral-7B-Instruct-v0.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-Instruct-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mistralai/Mistral-7B-Instruct-v0.2 with Docker Model Runner:
```
docker model run hf.co/mistralai/Mistral-7B-Instruct-v0.2
```

Failed to download tokenizer mistralai/Mistral-7B-Instruct-v0.2. Falling back to default tokenizer

#145

by didiaodeking - opened Jul 8, 2024

Discussion

didiaodeking

Jul 8, 2024

When I deployed privateGpt locally in windows 10, an error occurred. I tried updating the transforms version above but failed to solve the problem. I did not use a graphics card locally, was it related to nvida?

PS F:\private-gpt-0.5.0\private-gpt-main> $env:PGPT_PROFILES="ollama"
PS F:\private-gpt-0.5.0\private-gpt-main> make run
poetry run python -m private_gpt
10:24:25.303 [INFO ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'ollama']
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
--- Logging error ---
Traceback (most recent call last):
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init_.py", line 798, in get
return self._context[key]
~~~~~~~~~~~~~^^^^^
KeyError: <class 'private_gpt.ui.ui.PrivateGptUi'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init_.py", line 798, in get
return self._context[key]
~~~~~~~~~~~~~^^^^^
KeyError: <class 'private_gpt.server.ingest.ingest_service.IngestService'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\private-gpt-0.5.0\private-gpt-main\private_gpt\components\llm\llm_component.py", line 30, in init
AutoTokenizer.from_pretrained(
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\transformers\models\auto\tokenization_auto.py", line 825, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 2048, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 2287, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\transformers\models\llama\tokenization_llama_fast.py", line 133, in init
super().init(
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\transformers\tokenization_utils_fast.py", line 111, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 40 column 3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\Python311\Lib\logging_init.py", line 1110, in emit
msg = self.format(record)
^^^^^^^^^^^^^^^^^^^
File "D:\Python311\Lib\logging_init.py", line 953, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
File "D:\Python311\Lib\logging_init.py", line 687, in format
record.message = record.getMessage()
^^^^^^^^^^^^^^^^^^^
File "D:\Python311\Lib\logging_init.py", line 377, in getMessage
msg = msg % self.args
~~~~^~~~~~~~~~~
TypeError: not all arguments converted during string formatting
Call stack:
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "F:\private-gpt-0.5.0\private-gpt-main\private_gpt_main.py", line 5, in
from private_gpt.main import app
File "", line 1176, in find_and_load
File "", line 1147, in find_and_load_unlocked
File "", line 690, in load_unlocked
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "F:\private-gpt-0.5.0\private-gpt-main\private_gpt\main.py", line 6, in
app = create_app(global_injector)
File "F:\private-gpt-0.5.0\private-gpt-main\private_gpt\launcher.py", line 63, in create_app
ui = root_injector.get(PrivateGptUi)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 974, in get
provider_instance = scope_instance.get(interface, binding.provider)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 800, in get
instance = self.get_instance(key, provider, self.injector)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 811, in get_instance
return provider.get(injector)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 264, in get
return injector.create_object(self.cls)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 998, in create_object
self.call_with_injection(init, self=instance, kwargs=additional_kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 1031, in call_with_injection
dependencies = self.args_to_inject(
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 1079, in args_to_inject
instance: Any = self.get(interface)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 974, in get
provider_instance = scope_instance.get(interface, binding.provider)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 800, in get
instance = self.get_instance(key, provider, self.injector)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 811, in get_instance
return provider.get(injector)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 264, in get
return injector.create_object(self.cls)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 998, in create_object
self.call_with_injection(init, self=instance, kwargs=additional_kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 1031, in call_with_injection
dependencies = self.args_to_inject(
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 1079, in args_to_inject
instance: Any = self.get(interface)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 974, in get
provider_instance = scope_instance.get(interface, binding.provider)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 91, in wrapper
return function(*args, **kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 800, in get
instance = self.get_instance(key, provider, self.injector)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 811, in get_instance
return provider.get(injector)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 264, in get
return injector.create_object(self.cls)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init.py", line 998, in create_object
self.call_with_injection(init, self=instance, kwargs=additional_kwargs)
File "F:\private-gpt-0.5.0\private-gpt-main.venv\Lib\site-packages\injector_init_.py", line 1040, in call_with_injection
return callable(*full_args, **dependencies)
File "F:\private-gpt-0.5.0\private-gpt-main\private_gpt\components\llm\llm_component.py", line 37, in init
logger.warning(
Message: 'Failed to download tokenizer %s. Falling back to default tokenizer.'
Arguments: ('mistralai/Mistral-7B-Instruct-v0.2', Exception('data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 40 column 3'))
10:24:28.748 [INFO ] private_gpt.components.llm.llm_component - Initializing the LLM in mode=ollama
10:24:29.225 [INFO ] private_gpt.components.embedding.embedding_component - Initializing the embedding model in mode=ollama
10:24:29.227 [INFO ] llama_index.core.indices.loading - Loading all indices.
10:24:29.513 [INFO ] private_gpt.ui.ui - Mounting the gradio UI, at path=/
10:24:29.631 [INFO ] uvicorn.error - Started server process [20140]
10:24:29.631 [INFO ] uvicorn.error - Waiting for application startup.
10:24:29.632 [INFO ] uvicorn.error - Application startup complete.
10:24:29.634 [INFO ] uvicorn.error - Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

ocnimesh

Jul 16, 2024

I am also facing the same error. I use rayllm docker image for deploying the llm.

Logs :

File "/home/ray/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/home/ray/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/serve/_private/replica.py", line 455, in initialize_and_get_metadata
raise RuntimeError(traceback.format_exc()) from None
RuntimeError: Traceback (most recent call last):
File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/serve/_private/replica.py", line 443, in initialize_and_get_metadata
await self._initialize_replica()
File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/serve/_private/replica.py", line 182, in initialize_replica
await sync_to_async(_callable.init)(*init_args, **init_kwargs)
File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/server/vllm/vllm_deployment.py", line 37, in init
await self.engine.start()
File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/llm/vllm/vllm_engine.py", line 88, in start
pg, runtime_env = await self.node_initializer.initialize_node(self.llm_app)
File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/llm/llm_node_initializer.py", line 64, in initialize_node
await self._initialize_local_node(engine_config, scaling_config)
File "/home/ray/anaconda3/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/observability/tracing/threading_propagator.py", line 28, in wrapper
return fn(*args, **kwargs)
File "/home/ray/anaconda3/lib/python3.9/site-packages/rayllm/backend/llm/llm_node_initializer.py", line 95, in _initialize_local_node
_ = AutoTokenizer.from_pretrained(engine_config.actual_hf_model_id)
File "/home/ray/anaconda3/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 837, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/home/ray/anaconda3/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2086, in from_pretrained
return cls._from_pretrained(
File "/home/ray/anaconda3/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2325, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/home/ray/anaconda3/lib/python3.9/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in init
super().init(
File "/home/ray/anaconda3/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 111, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 40 column 3

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment