Hugging Face's logo

cpheemagazine
/
04cd8694-8cd2-423e-bd43-6af0ba123327

PEFT
Safetensors
llama
axolotl
Generated from Trainer
Model card Files
xet
 Community
cpheemagazine commited on
Commit
ea47089
·
verified ·
1 Parent(s): 6b4a4da

End of training

Browse files
Files changed (1) hide show
  1. README.md +134 -0
README.md ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ license: llama3.1
4
+ base_model: unsloth/Meta-Llama-3.1-8B
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ model-index:
9
+ - name: 04cd8694-8cd2-423e-bd43-6af0ba123327
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
17
+ <details><summary>See axolotl config</summary>
18
+
19
+ axolotl version: `0.10.0.dev0`
20
+ ```yaml
21
+ adapter: lora
22
+ base_model: unsloth/Meta-Llama-3.1-8B
23
+ bf16: true
24
+ chat_template: llama3
25
+ datasets:
26
+ - data_files:
27
+ - 212a39e4fa44f2c8_train_data.json
28
+ ds_type: json
29
+ format: custom
30
+ path: /workspace/input_data/
31
+ type:
32
+ field_input: input
33
+ field_instruction: instruct
34
+ field_output: output
35
+ format: '{instruction} {input}'
36
+ no_input_format: '{instruction}'
37
+ system_format: '{system}'
38
+ system_prompt: ''
39
+ eval_max_new_tokens: 256
40
+ evals_per_epoch: 2
41
+ flash_attention: false
42
+ fp16: false
43
+ gradient_accumulation_steps: 1
44
+ gradient_checkpointing: true
45
+ group_by_length: true
46
+ hub_model_id: cpheemagazine/04cd8694-8cd2-423e-bd43-6af0ba123327
47
+ learning_rate: 0.0002
48
+ logging_steps: 10
49
+ lora_alpha: 16
50
+ lora_dropout: 0.05
51
+ lora_fan_in_fan_out: false
52
+ lora_r: 8
53
+ lora_target_linear: true
54
+ lr_scheduler: cosine
55
+ max_steps: 11
56
+ micro_batch_size: 4
57
+ mlflow_experiment_name: /tmp/212a39e4fa44f2c8_train_data.json
58
+ model_type: AutoModelForCausalLM
59
+ num_epochs: 3
60
+ optimizer: adamw_bnb_8bit
61
+ output_dir: miner_id_24
62
+ pad_to_sequence_len: true
63
+ sample_packing: false
64
+ save_steps: 108
65
+ sequence_len: 2048
66
+ tf32: true
67
+ tokenizer_type: AutoTokenizer
68
+ train_on_inputs: false
69
+ trust_remote_code: true
70
+ val_set_size: 0.05
71
+ wandb_entity: null
72
+ wandb_mode: online
73
+ wandb_name: b983b3b1-fba7-427a-bbe9-cacea876706d
74
+ wandb_project: Gradients-On-Demand
75
+ wandb_run: apriasmoro
76
+ wandb_runid: b983b3b1-fba7-427a-bbe9-cacea876706d
77
+ warmup_steps: 100
78
+ weight_decay: 0.01
79
+
80
+ ```
81
+
82
+ </details><br>
83
+
84
+ # 04cd8694-8cd2-423e-bd43-6af0ba123327
85
+
86
+ This model is a fine-tuned version of [unsloth/Meta-Llama-3.1-8B](https://huggingface.co/unsloth/Meta-Llama-3.1-8B) on an unknown dataset.
87
+ It achieves the following results on the evaluation set:
88
+ - Loss: 1.4561
89
+
90
+ ## Model description
91
+
92
+ More information needed
93
+
94
+ ## Intended uses & limitations
95
+
96
+ More information needed
97
+
98
+ ## Training and evaluation data
99
+
100
+ More information needed
101
+
102
+ ## Training procedure
103
+
104
+ ### Training hyperparameters
105
+
106
+ The following hyperparameters were used during training:
107
+ - learning_rate: 0.0002
108
+ - train_batch_size: 4
109
+ - eval_batch_size: 4
110
+ - seed: 42
111
+ - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
112
+ - lr_scheduler_type: cosine
113
+ - lr_scheduler_warmup_steps: 100
114
+ - training_steps: 11
115
+
116
+ ### Training results
117
+
118
+ | Training Loss | Epoch | Step | Validation Loss |
119
+ |:-------------:|:------:|:----:|:---------------:|
120
+ | No log | 0.0005 | 1 | 1.4249 |
121
+ | No log | 0.0010 | 2 | 1.4633 |
122
+ | No log | 0.0021 | 4 | 1.4482 |
123
+ | No log | 0.0031 | 6 | 1.4277 |
124
+ | No log | 0.0041 | 8 | 1.4339 |
125
+ | 0.9248 | 0.0051 | 10 | 1.4561 |
126
+
127
+
128
+ ### Framework versions
129
+
130
+ - PEFT 0.15.2
131
+ - Transformers 4.51.3
132
+ - Pytorch 2.5.1+cu124
133
+ - Datasets 3.5.1
134
+ - Tokenizers 0.21.1