Instructions to use avinashhm/llama-3.1-8b-dora-finetuned with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use avinashhm/llama-3.1-8b-dora-finetuned with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="avinashhm/llama-3.1-8b-dora-finetuned")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("avinashhm/llama-3.1-8b-dora-finetuned")
model = AutoModelForCausalLM.from_pretrained("avinashhm/llama-3.1-8b-dora-finetuned", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

PEFT
How to use avinashhm/llama-3.1-8b-dora-finetuned with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use avinashhm/llama-3.1-8b-dora-finetuned with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "avinashhm/llama-3.1-8b-dora-finetuned"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "avinashhm/llama-3.1-8b-dora-finetuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/avinashhm/llama-3.1-8b-dora-finetuned

SGLang

How to use avinashhm/llama-3.1-8b-dora-finetuned with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "avinashhm/llama-3.1-8b-dora-finetuned" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "avinashhm/llama-3.1-8b-dora-finetuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "avinashhm/llama-3.1-8b-dora-finetuned" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "avinashhm/llama-3.1-8b-dora-finetuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use avinashhm/llama-3.1-8b-dora-finetuned with Docker Model Runner:
```
docker model run hf.co/avinashhm/llama-3.1-8b-dora-finetuned
```

LLaMA-3.1-8B-Instruct DoRA Fine-Tuned

Developed by: avinashhm
License: apache-2.0
Finetuned from model: devatar/quantized_Llama-3.1-8B-Instruct

This model is a fine-tuned version of devatar/quantized_Llama-3.1-8B-Instruct, adapted using DoRA (Weight-Decomposed Low-Rank Adaptation) on a subset of the mlabonne/FineTome-100k dataset. It is optimized for instruction-following tasks, such as answering questions and explaining concepts, and was fine-tuned on a 40GB GPU with memory-efficient techniques.

Training Details

Dataset: mlabonne/FineTome-100k (5,000 samples)
Fine-Tuning Method: DoRA (r=8, lora_alpha=16, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
Training Steps: 500
Optimizer: Paged AdamW 8-bit
Learning Rate: 2e-5 (cosine scheduler)
Batch Size: Effective batch size of 8 (per_device_train_batch_size=1, gradient_accumulation_steps=8)
Precision: Mixed precision (FP16)
Training Loss: Decreased from 1.4494 to 0.8145 over 500 steps

Dataset

The model was trained on a 5,000-sample subset of mlabonne/FineTome-100k, which contains high-quality instruction-response pairs. Conversations were formatted as ### Human: ... ### Gpt: ... for training, covering tasks like explaining programming concepts and reasoning.

Usage

To use the model for inference:

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("avinashhm/llama-3.1-8b-dora-finetuned", device_map="auto", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("avinashhm/llama-3.1-8b-dora-finetuned")

inputs = tokenizer("Explain boolean operators in programming.", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Note: The base model is 4-bit quantized, and merging DoRA adapters may introduce minor rounding errors in generations.

Requirements

torch
transformers
peft
trl
datasets
bitsandbytes
GPU with at least 40GB VRAM for training (less for inference)

Install dependencies:

pip install torch transformers datasets peft trl bitsandbytes
pip install git+https://github.com/huggingface/peft.git

Limitations

Fine-tuned on a 5,000-sample subset, which may limit generalization.
4-bit quantization may introduce slight performance trade-offs.
Used an older trl version (pre-0.7.0), lacking features like max_seq_length.

Downloads last month: 26

Safetensors

Model size

8B params

Tensor type

BF16

F32

avinashhm
/

llama-3.1-8b-dora-finetuned