Instructions to use QuietImpostor/Llama-3.1-Mini-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuietImpostor/Llama-3.1-Mini-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuietImpostor/Llama-3.1-Mini-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("QuietImpostor/Llama-3.1-Mini-Instruct")
model = AutoModelForCausalLM.from_pretrained("QuietImpostor/Llama-3.1-Mini-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuietImpostor/Llama-3.1-Mini-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuietImpostor/Llama-3.1-Mini-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuietImpostor/Llama-3.1-Mini-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuietImpostor/Llama-3.1-Mini-Instruct

SGLang

How to use QuietImpostor/Llama-3.1-Mini-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuietImpostor/Llama-3.1-Mini-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuietImpostor/Llama-3.1-Mini-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuietImpostor/Llama-3.1-Mini-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuietImpostor/Llama-3.1-Mini-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use QuietImpostor/Llama-3.1-Mini-Instruct with Docker Model Runner:
```
docker model run hf.co/QuietImpostor/Llama-3.1-Mini-Instruct
```

QuietImpostor commited on Aug 1, 2024

Commit

f34b11d

verified ·

1 Parent(s): e13f9f7

Update README.md

Browse files

Files changed (1) hide show

README.md +15 -6

README.md CHANGED Viewed

@@ -1,5 +1,4 @@
 ---
-base_model: QuietImpostor/Llama-3.1-Mini
 language:
 - en
 license: apache-2.0
@@ -12,12 +11,22 @@ tags:
 - sft
 ---
-# Uploaded  model
-- **Developed by:** QuietImpostor
-- **License:** apache-2.0
-- **Finetuned from model :** QuietImpostor/Llama-3.1-Mini
-This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
 [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)

 ---
 language:
 - en
 license: apache-2.0
 - sft
 ---
+# Llama 3.1 Mini - LoRA Finetuned
+## Model Description
+This model is a LoRA-finetuned version of the Llama 3.1 Mini model, which is a pruned variant of the Llama 3.1 8B model. The original Llama 3.1 Mini was created by pruning the larger model to approximately 3 billion parameters, and this version has been further adapted using Low-Rank Adaptation (LoRA) to enhance its capabilities.
+## Limitations
+Please note that this model, like its base version, may exhibit biases present in its training data and should be used with appropriate care and consideration.
+## Training Data
+The base model (Llama 3.1 Mini) was trained on my personal Claude 3 Opus and Claude 3.5 Sonnet dataset, with some synthetic pairs added on with Gemma 2 9B it being the user, and Llama 3 70B through Groq being the assistant. I have also used Guanaco alongside everything else.
+[Your citation or acknowledgment for the finetuned version]
+## License
+[Specify the license under which you're releasing this model, ensuring it complies with the original Llama 3.1 license]
 [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)