Update README.md

README.md

@@ -25,9 +25,8 @@ model-index:
 
 # Abstract
 
-This repository provides a domain-adapted Turkish legal instruction-tuned model derived from meta-llama/Llama-3.1-8B-Instruct. The model was developed as part of the “Harnessing Fully Sharded Data Parallelism v2 with Float8 Precision for Faster Training” experiment, which compared FSDP2 + FP8 v.s. FSDP2 + BF16 configurations where using FSDP2 with FP8 reduced training time by roughly 39 minutes (13.75% faster) for the llama_3.1_8B model compared with its respective BF16 training. 
-The model was fine-tuned on the newmindai/EuroHPC-Legal corpus (Q/A format) to enhance legal-domain reasoning across multiple subfields. This model was trained in bfloat16 precision using multi-GPU data parallelism on a high-performance computing (HPC) cluster.
-The resulting model aims to deliver accurate, context-aware legal text generation and question answering in Turkish while demonstrating the viability of FP8 training for production use.
+This repository provides a domain-adapted Turkish legal instruction-tuned model derived from meta-llama/Llama-3.1-8B-Instruct. The model was developed as part of the “Harnessing Fully Sharded Data Parallelism v2 with Float8 Precision for Faster Training” experiment, which compared FSDP2 + FP8 v.s. FSDP2 + BF16 configurations where using FSDP2 with FP8 reduced training time for the llama_3.1_8B model compared with its respective BF16 training. 
+The training experiments were conducted on the newmindai/EuroHPC-Legal corpus (Q/A format) to enhance legal-domain reasoning across multiple subfields. using mainly 4 recipies : fp8 mixed precision default cast configurations using “tensorwise” and “rowwise”,"rowwise_with_gw_hp"  recipes, also a “costum” cast configuration for each of the input, weight, grad_output and compared with the baseline FSDP2+bfp16. In fp8 mixed precision, the model weights to be updated were stored in BF16, and a copy of the model input, weight and grad_output beside the activations were quantized to FP8 varients using Torchao. 
 
 ### Setups
 - **Precision**: Used Half-precision bfloat16 as data type and for computation.
@@ -36,29 +35,14 @@ The resulting model aims to deliver accurate, context-aware legal text generatio
 
 ### Experiment Context
 
-This model was trained as part of our study for comparing **FSDP2 with bfloat16 precision** against **FSDP2 with FP8 mixed precision bfp16-fp8**. 
-We used `meta-llama/Llama-3.1-8B-Instruct`. The model has been loaded using `torch_dtype = bfloat16` and wrapped at once, also during forward/backward passes `bfloat16` has been used for computations.
-
-
+This model was trained with the Float8 modified rowwise scaling method that preserves the gradient weight computation in high precision (bfloat16) to enhance accuracy recipe. In this recipe during the forward pass, both the input and weight granularities are set to AXISWISE with a default E4M3 data type, while in the backward pass for gradient input, the input granularity remains AXISWISE and the weight granularity is TENSORWISE, both using E4M3; finally, in the backward pass for gradient weight, scaling is disabled for both input and grad output—keeping them in high precision and E4M3 respectively—and the configuration further enables round_scales_to_power_of_2=True to ensure numerical stability. 
 ```python
-
 from torchao.float8 import (
     convert_to_float8_training,
-    Float8LinearConfig
-)
-from dataclasses import replace
-
+    Float8LinearConfig)
 config = Float8LinearConfig.from_recipe_name("rowwise_with_gw_hp")
 model = convert_to_float8_training(model, config=config)
-if use_fp8:
-    for i, layer in enumerate(model.model.layers):
-        fully_shard(layer, **fsdp_kwargs)
-    fully_shard(model.model.embed_tokens, **fsdp_kwargs)
-    fully_shard(model.lm_head, **fsdp_kwargs)
-    config = replace(config, pad_inner_dim=True)
 ````
-
-
 ## Base Model Technical Specifications
 - **Parameters**: 8 Billion
 - **Architecture Family**: Llama 3.1
@@ -76,7 +60,7 @@ if use_fp8:
 
 ## Training Methodology
 
-### Training Configuration
+### *Training Configuration*
 - **Model**: `meta-llama/Llama-3.1-8B-Instruct`
 - **Sequence Length**: 4,096 (`seq_len`)
 - **Epochs**: 1
@@ -101,7 +85,7 @@ if use_fp8:
 - **Distributed Training**: `torch.distributed.run` (single node, multi-GPU)
   - FSDP2 (Optimized Fully Sharded Data Parallel)
 
-### Dependencies
+### *Dependencies*
 | package | Version |
 |-------|--------|
 | Transformers  | 4.57.1 |
@@ -112,68 +96,62 @@ if use_fp8:
 | tensorboard | 2.20.0 |
 | tensorboard-data-server | 0.7.2 |
 | wandb  | 0.22.1 |
-## **Performance Evaluation**
-# *2-models trained on 1Node with both (bfp16-fp8) with tensorwise && rowwise configurations*
-| perplexity metric results for bfp16-fp8 tensorwise && rowwise configurations | Accuracy metric results for bfp16-fp8 tensorwise && rowwise configurations | Loss metric results for bfp16-fp8 tensorwise && rowwise configurations | Memory allocation for bfp16-fp8 tensorwise && rowwise configurations | Utilization for bfp16-fp8 tensorwise && rowwise configurations | 
-|---------|---------|---------|---------|---------|
-|![W&B Chart 11_6_2025, 1_28_06 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/DOvLRT82pszUOYStDOBRG.png)| ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/27loZAUgU-V0gQ0jAZz4o.png) | ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/NX_Ni-U6spKd4R7PzBi8i.png)|![W&B Chart 11_6_2025, 1_24_01 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/YezrVMnv1CoJchKNhDA1V.png)| ![W&B Chart 11_6_2025, 1_25_41 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/qPkBYPC9rNKeyz0ONHmlO.png)|
 
+## **Performance Evaluation**
+### *2-models trained on 1Node with fp8 recipes*
+| Loss metric results for w16a8 & tensorwise recipe| Memory allocation for w16a8 & tensorwise recipe | Utilization for w16a8 & tensorwise recipe | 
+|---------|---------|---------|
+| ![lossRWGWHP](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/VAPmLlCaZPaks9SCSiGnW.png) | ![memALRWGWHP](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/WHhSPl1n2BpDhqzGl_ljh.png) | ![gpuutilsRWGWHP](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/PdiR1e2SGyTOloURHy19G.png) |
+
+| Loss metric results for w16a8 recipes| Memory allocation for w16a8 recipes | Utilization for w16a8 recipes | 
+|---------|---------|---------|
+| ![recipeloss](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/fAY3V4xAJzNOWfQDj2x3N.png) | ![recipeMemAl](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/BefD9N6e1ITRfIMt-beSR.png) | ![recipeUtils](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/WtoxFHU264cyhirrGgPys.png) |
+
+### *Loss Analysision*
+| Model                                           | Max Loss (train) | Min Loss (train) | Avg Loss (train) | Final Loss (train) | ± Std (train) | Max Loss (val) | Min Loss (val) | Avg Loss (val) | Final Loss (val) | ± Std (val) |
+| ---------------------------------------------- | ---------------- | ---------------- | ---------------- | ------------------ | ------------- | -------------- | -------------- | -------------- | ---------------- | ----------- |
+| Llama-3.1-8B-Instruct_w16a16                   | 3.1462           | 0.5710           | 0.8048           | 0.6374             | 0.2716        | 1.0517         | 0.8335         | 0.8876         | 0.8335           | 0.0678      |
+| Llama-3.1-8B-Instruct-w16a8-tw                 | 3.1983           | 0.5759           | 0.8113           | 0.6419             | 0.2756        | 1.0566         | 0.8390         | 0.8925         | 0.8391           | 0.0675      |
+| Llama-3.1-8B-Instruct_w16a8_4nodes_rw          | 3.1682           | 0.5740           | 0.8118           | 0.6431             | 0.2746        | 1.0613         | 0.8394         | 0.8937         | 0.8394           | 0.0688      |
+| Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp | 3.1837           | 0.5763           | 0.8116           | 0.6420             | 0.2751        | 1.0599         | 0.8391         | 0.8933         | 0.8391           | 0.0685      |
+| Llama-3.1-8B-Instruct-w16a8-mxtw               | 3.1983           | 0.5747           | 0.8115           | 0.6446             | 0.2758        | 1.0562         | 0.8384         | 0.8923         | 0.8384           | 0.0677      |
+
+### *Training Time Analysision*
 | Model                                               | Training Time (mins) | Memory Allocated (avg %) | GPU Utilization (avg %) | Speed vs bf16 |
 | :-------------------------------------------------- | --------------------: | -----------------------: | -----------------------: | -------------: |
 | **Llama-3.1-8B-Instruct_w16a16**                    | 138.75267            | 74.4189                  | 56.6059%                 | _             |
 | **Llama-3.1-8B-Instruct-w16a8-tw**                  | 123.75267            | 68.8982                  | 97.5364%                 | 12.11%        |
 | **Llama-3.1-8B-Instruct_w16a8_rw**                  | 115.75364            | 69.6132                  | 97.7689%                 | 19.87%        |
-| **Llama-3.1-8B-Instruct_w16a8_rowwise_with_gw_hp**  | 109.00364            | 69.4806                  | 97.3312%                 | 27.33%        |
+| **Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp**  | 109.00364            | 69.4806                  | 97.3312%                 | 27.33%        |
 | **Llama-3.1-8B-Instruct-w16a8-mxtw**                | 64.00328             | 68.8982                  | 95.5661%                 | 116.82%       |
 
-
-## Implementation
-### Gpu && Memory usage Profiling
-To visualize the usage of the memory and gpu's vram/utilis and more you can profile them with **pytorch-profiler** tool or **nsight systems** profiler tool . 
-- follow the steps to visualize with **Pytorch-profiler** : 
+## **Implementation**
+### *Gpu && Memory usage Profiling*
+The training progress has been profiled using **pytorch-profiler** tool. 
+- follow the steps to visualize the profiles: 
   1. pip install the versions that mentioned in the dependencies section of these libs tensorboard and tensorboard-data-server.
   2. Visualize pytorch profiles by runing the command provided below. 
   ```python
-  tensorboard --logdir="/gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node" --port="6006" --bind_all
-  ````
-
-- follow the steps to visualize with **nsight systems**:
-  1. download the files on to you local machine.
-  2. download the full_version of the nsys profiler tool from here : https://developer.nvidia.com/nsight-systems/get-started.
-  3. open the nsys-rep profiles using the nsight systems gui .
-
-  also you can get a summary of the report using these command lines up to your porpuse :
-    ```python
-    nsys stats --report cuda_gpu_kern_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep
-    ````
-    ```python
-    nsys stats --report cuda_gpu_trace /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep
-    ````
-    ```python
-      nsys stats --report cuda_gpu_mem_size_sum /gpfs/scratch/ehpc317/esavirdi/startClean/profiles/bfp16_1node/lama3.1_fp16_baseline.nsys-rep
+  tensorboard --logdir="/gpfs/scratch/ehpc317/esavirdi/dual_precision_training/profiles/Llama-3.1-8B-Instruct_tensorwise" --port="6006"
   ````
 
 
-### Usage
+### *Usage*
 
 **Note**: the final model has been saved in bfloat16 format. For inference, load the model in bfloat16 or float16 as shown below:
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 import torch
-
 model_name = "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp"
 dtype = torch.bfloat16
-
 tok = AutoTokenizer.from_pretrained(model_name)
 model = AutoModelForCausalLM.from_pretrained(
     model_name,
     torch_dtype=dtype,
     device_map="auto"
 )
-
 prompt = "Soru: Kişisel Verilerin Korunması Kanunu uyarınca hangi durumlarda açık rıza aranmaz? Cevap:"
-
 inputs = tok(prompt, return_tensors="pt").to(model.device)
 with torch.no_grad():
     out = model.generate(
@@ -184,7 +162,6 @@ with torch.no_grad():
 
 print(tok.decode(out[0], skip_special_tokens=True))
 ````
-
 ## Ethical Considerations and Disclaimers
 * Research & development purposes only; not a substitute for professional legal counsel.
 * Users must ensure compliance with data protection and sector regulations.
@@ -200,7 +177,6 @@ print(tok.decode(out[0], skip_special_tokens=True))
 ## References and Citations
 
 ### Base Model
-
 ```bibtex
 @misc{meta_llama31_8b_instruct,
   title={Llama 3.1 8B Instruct},
@@ -209,7 +185,6 @@ print(tok.decode(out[0], skip_special_tokens=True))
   howpublished={\url{https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct}}
 }
 ```
-
 ### Training Dataset
 
 ```bibtex