Instructions to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp")
model = AutoModelForCausalLM.from_pretrained("newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp

SGLang

How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with Docker Model Runner:
```
docker model run hf.co/newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp
```

Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp / README.md

nmmursit

Update README.md

dc76f2b verified 11 months ago

9.7 kB

language:
  - tr
license: llama3.1
base_model: meta-llama/Llama-3.1-8B-Instruct
tags:
  - legal
  - turkish
  - llama-3.1
  - fp8
  - bfloat16
  - mixed-precision
  - question-answering
  - fsdp-v2
  - pytorch
  - distributed-training
datasets:
  - newmindai/EuroHPC-Legal
library_name: transformers
pipeline_tag: text-generation
model-index:
  - name: Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp
    results: []

Abstract

This repository provides a domain-adapted Turkish legal instruction-tuned model derived from meta-llama/Llama-3.1-8B-Instruct. The model was developed as part of the “Harnessing Fully Sharded Data Parallelism v2 with Float8 Precision for Faster Training” experiment, which compared FSDP2 + FP8 v.s. FSDP2 + BF16 configurations where using FSDP2 with FP8 reduced training time by roughly 39 minutes (13.75% faster) for the llama_3.1_8B model compared with its respective BF16 training. The model was fine-tuned on the newmindai/EuroHPC-Legal corpus (Q/A format) to enhance legal-domain reasoning across multiple subfields. This model was trained in bfloat16 precision using multi-GPU data parallelism on a high-performance computing (HPC) cluster. The resulting model aims to deliver accurate, context-aware legal text generation and question answering in Turkish while demonstrating the viability of FP8 training for production use.

Setups

Precision: Used Half-precision bfloat16 as data type and for computation.
Hardware: HPC (EuroHPC/BSC-class) node with 4 × NVIDIA H100 GPUs.
Framework: PyTorch with torchrun for distributed training.

Experiment Context

This model was trained as part of our study for comparing FSDP2 with bfloat16 precision against FSDP2 with FP8 mixed precision bfp16-fp8. We used meta-llama/Llama-3.1-8B-Instruct. The model has been loaded using torch_dtype = bfloat16 and wrapped at once, also during forward/backward passes bfloat16 has been used for computations.


from torchao.float8 import (
    convert_to_float8_training,
    Float8LinearConfig
)
from dataclasses import replace

config = Float8LinearConfig.from_recipe_name("rowwise_with_gw_hp")
model = convert_to_float8_training(model, config=config)
if use_fp8:
    for i, layer in enumerate(model.model.layers):
        fully_shard(layer, **fsdp_kwargs)
    fully_shard(model.model.embed_tokens, **fsdp_kwargs)
    fully_shard(model.lm_head, **fsdp_kwargs)
    config = replace(config, pad_inner_dim=True)

Base Model Technical Specifications

Parameters: 8 Billion
Architecture Family: Llama 3.1
Maximum Position Embeddings: 131,072
Attention Heads: 32 (num_attention_heads)
Key-Value Heads: 8 (num_key_value_heads)
Hidden Layers: 32 (num_hidden_layers)
Hidden Size: 4,096 (hidden_size)
Intermediate Size: 14,336
Vocabulary Size: 128,256
Precision: bfloat16
RoPE Scaling: type llama3, factor = 8.0
RMS Norm Epsilon: 1e-05
Activation: SiLU

Training Methodology

Training Configuration

Model: meta-llama/Llama-3.1-8B-Instruct
Sequence Length: 4,096 (seq_len)
Epochs: 1
Per-Device Micro Batch Size: 2
Gradient Accumulation: 4
GPUs: 4 (via CUDA_VISIBLE_DEVICES=0,1,2,3)
dtype: bf16 && fp8=false
- Weights: bfloat16
- Activations: bfloat16
Optimizer: AdamW
- Learning Rate: 2e-5
- Weight Decay: 0.01
- Betas: (0.9, 0.95)
- Epsilon: 1e-8
LR Scheduler: Cosine; warmup = 10% (warmup_ratio=0.1) | also warmup_steps=100
Max Grad Norm: 1.0
Gradient Checkpointing: Enabled
Evaluation: every 5 steps (eval_steps=5, eval_samples=1000)
Checkpointing: every 10 steps; keep last 5; select best by eval_loss
Logging: every step to file; Weights & Biases in offline mode
Seed: 100
Distributed Training: torch.distributed.run (single node, multi-GPU)
- FSDP2 (Optimized Fully Sharded Data Parallel)

Dependencies

package	Version
Transformers	4.57.1
torch	2.9.0+cu128
accelerate	0.14.1
datasets	4.3.0
huggingface-hub	0.36.0
tensorboard	2.20.0
tensorboard-data-server	0.7.2
wandb	0.22.1

Performance Evaluation

2-models trained on 1Node with both (bfp16-fp8) with tensorwise && rowwise configurations

perplexity metric results for bfp16-fp8 tensorwise && rowwise configurations	Accuracy metric results for bfp16-fp8 tensorwise && rowwise configurations	Loss metric results for bfp16-fp8 tensorwise && rowwise configurations	Memory allocation for bfp16-fp8 tensorwise && rowwise configurations	Utilization for bfp16-fp8 tensorwise && rowwise configurations

Model	Training Time (mins)	Memory Allocated (avg %)	GPU Utilization (avg %)	Speed vs bf16
Llama-3.1-8B-Instruct_w16a16	138.75267	74.4189	56.6059%	_
Llama-3.1-8B-Instruct-w16a8-tw	123.75267	68.8982	97.5364%	12.11%
Llama-3.1-8B-Instruct_w16a8_rw	115.75364	69.6132	97.7689%	19.87%
Llama-3.1-8B-Instruct_w16a8_rowwise_with_gw_hp	109.00364	69.4806	97.3312%	27.33%
Llama-3.1-8B-Instruct-w16a8-mxtw	64.00328	68.8982	95.5661%	116.82%

Implementation

Gpu && Memory usage Profiling

To visualize the usage of the memory and gpu's vram/utilis and more you can profile them with pytorch-profiler tool or nsight systems profiler tool .

follow the steps to visualize with Pytorch-profiler :
1. pip install the versions that mentioned in the dependencies section of these libs tensorboard and tensorboard-data-server.
2. Visualize pytorch profiles by runing the command provided below.
```
tensorboard --logdir="/gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node" --port="6006" --bind_all
```

follow the steps to visualize with nsight systems:

download the files on to you local machine.
download the full_version of the nsys profiler tool from here : https://developer.nvidia.com/nsight-systems/get-started.
open the nsys-rep profiles using the nsight systems gui .

also you can get a summary of the report using these command lines up to your porpuse :

nsys stats --report cuda_gpu_kern_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep

nsys stats --report cuda_gpu_trace /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep

  nsys stats --report cuda_gpu_mem_size_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep

Usage

Note: the final model has been saved in bfloat16 format. For inference, load the model in bfloat16 or float16 as shown below:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp"
dtype = torch.bfloat16

tok = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=dtype,
    device_map="auto"
)

prompt = "Soru: Kişisel Verilerin Korunması Kanunu uyarınca hangi durumlarda açık rıza aranmaz? Cevap:"

inputs = tok(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False
    )

print(tok.decode(out[0], skip_special_tokens=True))

Ethical Considerations and Disclaimers

Research & development purposes only; not a substitute for professional legal counsel.
Users must ensure compliance with data protection and sector regulations.
Potential biases may exist in domain data and model outputs.

Model & Data Card Metadata

Total Parameters: 8,030,261,248
Serialized Size (approx.): 16,060,522,496 bytes
Config precision: bfloat16
RoPE: llama3 scaling, factor 8.0

References and Citations

Base Model

@misc{meta_llama31_8b_instruct,
  title={Llama 3.1 8B Instruct},
  author={Meta AI},
  year={2024},
  howpublished={\url{https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct}}
}

Training Dataset

@misc{euro_hpc_legal,
  title={EuroHPC-Legal},
  author={newmindai},
  year={2025},
  howpublished={\url{https://huggingface.co/datasets/newmindai/EuroHPC-Legal}}
}