Update README.md

README.md

@@ -19,6 +19,203 @@ datasets:
 library_name: transformers
 pipeline_tag: text-generation
 model-index:
-- name: Llama-3.1-8B-Instruct_w16a8_tensorwise
+- name: Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp
   results: []
 ---
+
+# Abstract
+
+This repository provides a domain-adapted Turkish legal instruction-tuned model derived from meta-llama/Llama-3.1-8B-Instruct. The model was developed as part of the “Harnessing Fully Sharded Data Parallelism v2 with Float8 Precision for Faster Training” experiment, which compared FSDP2 + FP8 v.s. FSDP2 + BF16 configurations where using FSDP2 with FP8 reduced training time by roughly 39 minutes (13.75% faster) for the llama_3.1_8B model compared with its respective BF16 training. 
+The model was fine-tuned on the newmindai/EuroHPC-Legal corpus (Q/A format) to enhance legal-domain reasoning across multiple subfields. This model was trained in bfloat16 precision using multi-GPU data parallelism on a high-performance computing (HPC) cluster.
+The resulting model aims to deliver accurate, context-aware legal text generation and question answering in Turkish while demonstrating the viability of FP8 training for production use.
+
+### Setups
+- **Precision**: Used Half-precision bfloat16 as data type and for computation.
+- **Hardware**: HPC (EuroHPC/BSC-class) node with 4 × NVIDIA H100 GPUs. 
+- **Framework**: PyTorch with `torchrun` for distributed training. 
+
+### Experiment Context
+
+This model was trained as part of our study for comparing **FSDP2 with bfloat16 precision** against **FSDP2 with FP8 mixed precision bfp16-fp8**. 
+We used `meta-llama/Llama-3.1-8B-Instruct`. The model has been loaded using `torch_dtype = bfloat16` and wrapped at once, also during forward/backward passes `bfloat16` has been used for computations.
+
+
+```python
+
+from torchao.float8 import (
+    convert_to_float8_training,
+    Float8LinearConfig
+)
+from dataclasses import replace
+
+config = Float8LinearConfig.from_recipe_name("rowwise_with_gw_hp")
+model = convert_to_float8_training(model, config=config)
+if use_fp8:
+    for i, layer in enumerate(model.model.layers):
+        fully_shard(layer, **fsdp_kwargs)
+    fully_shard(model.model.embed_tokens, **fsdp_kwargs)
+    fully_shard(model.lm_head, **fsdp_kwargs)
+    config = replace(config, pad_inner_dim=True)
+````
+
+
+## Base Model Technical Specifications
+- **Parameters**: 8 Billion
+- **Architecture Family**: Llama 3.1
+- **Maximum Position Embeddings**: 131,072
+- **Attention Heads**: 32 (`num_attention_heads`)
+- **Key-Value Heads**: 8 (`num_key_value_heads`)
+- **Hidden Layers**: 32 (`num_hidden_layers`)
+- **Hidden Size**: 4,096 (`hidden_size`)
+- **Intermediate Size**: 14,336
+- **Vocabulary Size**: 128,256
+- **Precision**: bfloat16
+- **RoPE Scaling**: type `llama3`, factor = 8.0
+- **RMS Norm Epsilon**: 1e-05
+- **Activation**: SiLU
+
+## Training Methodology
+
+### Training Configuration
+- **Model**: `meta-llama/Llama-3.1-8B-Instruct`
+- **Sequence Length**: 4,096 (`seq_len`)
+- **Epochs**: 1
+- **Per-Device Micro Batch Size**: 2
+- **Gradient Accumulation**: 4
+- **GPUs**: 4 (via `CUDA_VISIBLE_DEVICES=0,1,2,3`)
+- **dtype**: `bf16` && `fp8=false`
+  - Weights: bfloat16
+  - Activations: bfloat16
+- **Optimizer**: AdamW
+  - Learning Rate: 2e-5
+  - Weight Decay: 0.01
+  - Betas: (0.9, 0.95)
+  - Epsilon: 1e-8
+- **LR Scheduler**: Cosine; warmup = 10% (`warmup_ratio=0.1`) | also `warmup_steps=100`
+- **Max Grad Norm**: 1.0
+- **Gradient Checkpointing**: Enabled
+- **Evaluation**: every 5 steps (`eval_steps=5`, `eval_samples=1000`)
+- **Checkpointing**: every 10 steps; keep last 5; select best by `eval_loss`
+- **Logging**: every step to file; Weights & Biases in offline mode
+- **Seed**: 100
+- **Distributed Training**: `torch.distributed.run` (single node, multi-GPU)
+  - FSDP2 (Optimized Fully Sharded Data Parallel)
+
+### Dependencies
+| package | Version |
+|-------|--------|
+| Transformers  | 4.57.1 |
+| torch  | 2.9.0+cu128 |
+| accelerate  | 0.14.1 |
+| datasets  | 4.3.0 |
+| huggingface-hub  | 0.36.0 |
+| tensorboard | 2.20.0 |
+| tensorboard-data-server | 0.7.2 |
+| wandb  | 0.22.1 |
+## **Performance Evaluation**
+# *2-models trained on 1Node with both (bfp16-fp8) with tensorwise && rowwise configurations*
+| perplexity metric results for bfp16-fp8 tensorwise && rowwise configurations | Accuracy metric results for bfp16-fp8 tensorwise && rowwise configurations | Loss metric results for bfp16-fp8 tensorwise && rowwise configurations | Memory allocation for bfp16-fp8 tensorwise && rowwise configurations | Utilization for bfp16-fp8 tensorwise && rowwise configurations | 
+|---------|---------|---------|---------|---------|
+|![W&B Chart 11_6_2025, 1_28_06 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/DOvLRT82pszUOYStDOBRG.png)| ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/27loZAUgU-V0gQ0jAZz4o.png) | ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/NX_Ni-U6spKd4R7PzBi8i.png)|![W&B Chart 11_6_2025, 1_24_01 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/YezrVMnv1CoJchKNhDA1V.png)| ![W&B Chart 11_6_2025, 1_25_41 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/qPkBYPC9rNKeyz0ONHmlO.png)|
+
+
+
+| Method               | Total Training Time | Avg GPU Utilization | Final Val Loss | Final Val Perplexity | Final Val Accuracy | Memory Allocated (avg %) |     **Speed vs Tensorwise**     |
+| :------------------- | :------------------ | :-----------------: | :------------: | :------------------: | :----------------: | :----------------------: | :-----------------------------: |
+| **w16a8 Tensorwise** | **123 min 50 s**    |        ≈ 98 %       |     0.8394     |        2.3151        |       0.7934       |        ~68 – 80 %        |                —                |
+| **w16a8 Rowwise**    | **115 min 25 s**    |        ≈ 99 %       |     0.8392     |        2.3144        |       0.7941       |        ~66 – 78 %        |                7%               |
+
+
+## Implementation
+### Gpu && Memory usage Profiling
+To visualize the usage of the memory and gpu's vram/utilis and more you can profile them with **pytorch-profiler** tool or **nsight systems** profiler tool . 
+- follow the steps to visualize with **Pytorch-profiler** : 
+  1. pip install the versions that mentioned in the dependencies section of these libs tensorboard and tensorboard-data-server.
+  2. Visualize pytorch profiles by runing the command provided below. 
+  ```python
+  tensorboard --logdir="/gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node" --port="6006" --bind_all
+  ````
+
+- follow the steps to visualize with **nsight systems**:
+  1. download the files on to you local machine.
+  2. download the full_version of the nsys profiler tool from here : https://developer.nvidia.com/nsight-systems/get-started.
+  3. open the nsys-rep profiles using the nsight systems gui .
+
+  also you can get a summary of the report using these command lines up to your porpuse :
+    ```python
+    nsys stats --report cuda_gpu_kern_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep
+    ````
+    ```python
+    nsys stats --report cuda_gpu_trace /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep
+    ````
+    ```python
+      nsys stats --report cuda_gpu_mem_size_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep
+  ````
+
+
+### Usage
+
+**Note**: the final model has been saved in bfloat16 format. For inference, load the model in bfloat16 or float16 as shown below:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_name = "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp"
+dtype = torch.bfloat16
+
+tok = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype=dtype,
+    device_map="auto"
+)
+
+prompt = "Soru: Kişisel Verilerin Korunması Kanunu uyarınca hangi durumlarda açık rıza aranmaz? Cevap:"
+
+inputs = tok(prompt, return_tensors="pt").to(model.device)
+with torch.no_grad():
+    out = model.generate(
+        **inputs,
+        max_new_tokens=256,
+        do_sample=False
+    )
+
+print(tok.decode(out[0], skip_special_tokens=True))
+````
+
+## Ethical Considerations and Disclaimers
+* Research & development purposes only; not a substitute for professional legal counsel.
+* Users must ensure compliance with data protection and sector regulations.
+* Potential biases may exist in domain data and model outputs.
+
+## Model & Data Card Metadata
+
+* **Total Parameters**: 8,030,261,248
+* **Serialized Size (approx.)**: 16,060,522,496 bytes
+* **Config precision**: bfloat16
+* **RoPE**: llama3 scaling, factor 8.0
+
+## References and Citations
+
+### Base Model
+
+```bibtex
+@misc{meta_llama31_8b_instruct,
+  title={Llama 3.1 8B Instruct},
+  author={Meta AI},
+  year={2024},
+  howpublished={\url{https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct}}
+}
+```
+
+### Training Dataset
+
+```bibtex
+@misc{euro_hpc_legal,
+  title={EuroHPC-Legal},
+  author={newmindai},
+  year={2025},
+  howpublished={\url{https://huggingface.co/datasets/newmindai/EuroHPC-Legal}}
+}
+```