ArliAI-Llama-3
Collection
Quantizations of ArliAI-Llama-3 models • 15 items • Updated • 2
How to use from
vLLMUse Docker
docker model run hf.co/QuantFactory/ArliAI-Llama-3-8B-Instruct-ORPO-v0.1-GGUF:Quick Links
QuantFactory/ArliAI-Llama-3-8B-Instruct-ORPO-v0.1-GGUF
This is quantized version of OwenArli/ArliAI-Llama-3-8B-Instruct-ORPO-v0.1 created using llama.cpp
Model Description
Based on Meta-Llama-3-8b-Instruct, and is governed by Meta Llama 3 License agreement: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
ORPO fine tuning method using the following datasets:
- https://huggingface.co/datasets/Intel/orca_dpo_pairs
- https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo
- https://huggingface.co/datasets/unalignment/toxic-dpo-v0.2
- https://huggingface.co/datasets/M4-ai/prm_dpo_pairs_cleaned
- https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1
Despite the toxic datasets to reduce refusals, this model is still relatively safe but refuses less than the original Meta model.
As of now ORPO fine tuning seems to improve some metrics while reducing other metrics by a lot:
Instruct format:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>
{{ user_message_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{{ model_answer_1 }}<|eot_id|><|start_header_id|>user<|end_header_id|>
{{ user_message_2 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Quants:
- Downloads last month
- 264
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
Install from pip and serve model
# Install vLLM from pip: pip install vllm# Start the vLLM server: vllm serve "QuantFactory/ArliAI-Llama-3-8B-Instruct-ORPO-v0.1-GGUF"# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "QuantFactory/ArliAI-Llama-3-8B-Instruct-ORPO-v0.1-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'