--- language: - tr license: llama3.1 base_model: meta-llama/Llama-3.1-8B-Instruct tags: - legal - turkish - llama-3.1 - fp8 - bfloat16 - mixed-precision - question-answering - fsdp-v2 - pytorch - distributed-training datasets: - newmindai/EuroHPC-Legal library_name: transformers pipeline_tag: text-generation model-index: - name: Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp results: [] --- # Abstract This repository provides a domain-adapted Turkish legal instruction-tuned model derived from meta-llama/Llama-3.1-8B-Instruct. The model was developed as part of the “Harnessing Fully Sharded Data Parallelism v2 with Float8 Precision for Faster Training” experiment, which compared FSDP2 + FP8 v.s. FSDP2 + BF16 configurations where using FSDP2 with FP8 reduced training time by roughly 39 minutes (13.75% faster) for the llama_3.1_8B model compared with its respective BF16 training. The model was fine-tuned on the newmindai/EuroHPC-Legal corpus (Q/A format) to enhance legal-domain reasoning across multiple subfields. This model was trained in bfloat16 precision using multi-GPU data parallelism on a high-performance computing (HPC) cluster. The resulting model aims to deliver accurate, context-aware legal text generation and question answering in Turkish while demonstrating the viability of FP8 training for production use. ### Setups - **Precision**: Used Half-precision bfloat16 as data type and for computation. - **Hardware**: HPC (EuroHPC/BSC-class) node with 4 × NVIDIA H100 GPUs. - **Framework**: PyTorch with `torchrun` for distributed training. ### Experiment Context This model was trained as part of our study for comparing **FSDP2 with bfloat16 precision** against **FSDP2 with FP8 mixed precision bfp16-fp8**. We used `meta-llama/Llama-3.1-8B-Instruct`. The model has been loaded using `torch_dtype = bfloat16` and wrapped at once, also during forward/backward passes `bfloat16` has been used for computations. ```python from torchao.float8 import ( convert_to_float8_training, Float8LinearConfig ) from dataclasses import replace config = Float8LinearConfig.from_recipe_name("rowwise_with_gw_hp") model = convert_to_float8_training(model, config=config) if use_fp8: for i, layer in enumerate(model.model.layers): fully_shard(layer, **fsdp_kwargs) fully_shard(model.model.embed_tokens, **fsdp_kwargs) fully_shard(model.lm_head, **fsdp_kwargs) config = replace(config, pad_inner_dim=True) ```` ## Base Model Technical Specifications - **Parameters**: 8 Billion - **Architecture Family**: Llama 3.1 - **Maximum Position Embeddings**: 131,072 - **Attention Heads**: 32 (`num_attention_heads`) - **Key-Value Heads**: 8 (`num_key_value_heads`) - **Hidden Layers**: 32 (`num_hidden_layers`) - **Hidden Size**: 4,096 (`hidden_size`) - **Intermediate Size**: 14,336 - **Vocabulary Size**: 128,256 - **Precision**: bfloat16 - **RoPE Scaling**: type `llama3`, factor = 8.0 - **RMS Norm Epsilon**: 1e-05 - **Activation**: SiLU ## Training Methodology ### Training Configuration - **Model**: `meta-llama/Llama-3.1-8B-Instruct` - **Sequence Length**: 4,096 (`seq_len`) - **Epochs**: 1 - **Per-Device Micro Batch Size**: 2 - **Gradient Accumulation**: 4 - **GPUs**: 4 (via `CUDA_VISIBLE_DEVICES=0,1,2,3`) - **dtype**: `bf16` && `fp8=false` - Weights: bfloat16 - Activations: bfloat16 - **Optimizer**: AdamW - Learning Rate: 2e-5 - Weight Decay: 0.01 - Betas: (0.9, 0.95) - Epsilon: 1e-8 - **LR Scheduler**: Cosine; warmup = 10% (`warmup_ratio=0.1`) | also `warmup_steps=100` - **Max Grad Norm**: 1.0 - **Gradient Checkpointing**: Enabled - **Evaluation**: every 5 steps (`eval_steps=5`, `eval_samples=1000`) - **Checkpointing**: every 10 steps; keep last 5; select best by `eval_loss` - **Logging**: every step to file; Weights & Biases in offline mode - **Seed**: 100 - **Distributed Training**: `torch.distributed.run` (single node, multi-GPU) - FSDP2 (Optimized Fully Sharded Data Parallel) ### Dependencies | package | Version | |-------|--------| | Transformers | 4.57.1 | | torch | 2.9.0+cu128 | | accelerate | 0.14.1 | | datasets | 4.3.0 | | huggingface-hub | 0.36.0 | | tensorboard | 2.20.0 | | tensorboard-data-server | 0.7.2 | | wandb | 0.22.1 | ## **Performance Evaluation** # *2-models trained on 1Node with both (bfp16-fp8) with tensorwise && rowwise configurations* | perplexity metric results for bfp16-fp8 tensorwise && rowwise configurations | Accuracy metric results for bfp16-fp8 tensorwise && rowwise configurations | Loss metric results for bfp16-fp8 tensorwise && rowwise configurations | Memory allocation for bfp16-fp8 tensorwise && rowwise configurations | Utilization for bfp16-fp8 tensorwise && rowwise configurations | |---------|---------|---------|---------|---------| |![W&B Chart 11_6_2025, 1_28_06 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/DOvLRT82pszUOYStDOBRG.png)| ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/27loZAUgU-V0gQ0jAZz4o.png) | ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/NX_Ni-U6spKd4R7PzBi8i.png)|![W&B Chart 11_6_2025, 1_24_01 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/YezrVMnv1CoJchKNhDA1V.png)| ![W&B Chart 11_6_2025, 1_25_41 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/qPkBYPC9rNKeyz0ONHmlO.png)| | Model | Training Time (mins) | Memory Allocated (avg %) | GPU Utilization (avg %) | Speed vs bf16 | | :-------------------------------------------------- | --------------------: | -----------------------: | -----------------------: | -------------: | | **Llama-3.1-8B-Instruct_w16a16** | 138.75267 | 74.4189 | 56.6059% | _ | | **Llama-3.1-8B-Instruct-w16a8-tw** | 123.75267 | 68.8982 | 97.5364% | 12.11% | | **Llama-3.1-8B-Instruct_w16a8_rw** | 115.75364 | 69.6132 | 97.7689% | 19.87% | | **Llama-3.1-8B-Instruct_w16a8_rowwise_with_gw_hp** | 109.00364 | 69.4806 | 97.3312% | 27.33% | | **Llama-3.1-8B-Instruct-w16a8-mxtw** | 64.00328 | 68.8982 | 95.5661% | 116.82% | ## Implementation ### Gpu && Memory usage Profiling To visualize the usage of the memory and gpu's vram/utilis and more you can profile them with **pytorch-profiler** tool or **nsight systems** profiler tool . - follow the steps to visualize with **Pytorch-profiler** : 1. pip install the versions that mentioned in the dependencies section of these libs tensorboard and tensorboard-data-server. 2. Visualize pytorch profiles by runing the command provided below. ```python tensorboard --logdir="/gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node" --port="6006" --bind_all ```` - follow the steps to visualize with **nsight systems**: 1. download the files on to you local machine. 2. download the full_version of the nsys profiler tool from here : https://developer.nvidia.com/nsight-systems/get-started. 3. open the nsys-rep profiles using the nsight systems gui . also you can get a summary of the report using these command lines up to your porpuse : ```python nsys stats --report cuda_gpu_kern_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep ```` ```python nsys stats --report cuda_gpu_trace /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep ```` ```python nsys stats --report cuda_gpu_mem_size_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep ```` ### Usage **Note**: the final model has been saved in bfloat16 format. For inference, load the model in bfloat16 or float16 as shown below: ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_name = "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp" dtype = torch.bfloat16 tok = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=dtype, device_map="auto" ) prompt = "Soru: Kişisel Verilerin Korunması Kanunu uyarınca hangi durumlarda açık rıza aranmaz? Cevap:" inputs = tok(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): out = model.generate( **inputs, max_new_tokens=256, do_sample=False ) print(tok.decode(out[0], skip_special_tokens=True)) ```` ## Ethical Considerations and Disclaimers * Research & development purposes only; not a substitute for professional legal counsel. * Users must ensure compliance with data protection and sector regulations. * Potential biases may exist in domain data and model outputs. ## Model & Data Card Metadata * **Total Parameters**: 8,030,261,248 * **Serialized Size (approx.)**: 16,060,522,496 bytes * **Config precision**: bfloat16 * **RoPE**: llama3 scaling, factor 8.0 ## References and Citations ### Base Model ```bibtex @misc{meta_llama31_8b_instruct, title={Llama 3.1 8B Instruct}, author={Meta AI}, year={2024}, howpublished={\url{https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct}} } ``` ### Training Dataset ```bibtex @misc{euro_hpc_legal, title={EuroHPC-Legal}, author={newmindai}, year={2025}, howpublished={\url{https://huggingface.co/datasets/newmindai/EuroHPC-Legal}} } ```