Hugging Face's logo

error577
/
47615d62-b7f3-4efd-b567-61a937215b9f

PEFT
Safetensors
llama
axolotl
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files
xet
 Community
error577 commited on
Commit
ad9fd58
·
verified ·
1 Parent(s): 4fd4b47

End of training

Browse files
Files changed (3) hide show
  1. README.md +159 -0
  2. adapter_model.bin +3 -0
  3. adapter_model.safetensors +1 -1
README.md ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ base_model: samoline/5255b81c-a162-4f3a-9a96-f4d344cb99c8
4
+ tags:
5
+ - axolotl
6
+ - generated_from_trainer
7
+ model-index:
8
+ - name: 47615d62-b7f3-4efd-b567-61a937215b9f
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
16
+ <details><summary>See axolotl config</summary>
17
+
18
+ axolotl version: `0.4.1`
19
+ ```yaml
20
+ adapter: qlora
21
+ auto_resume_from_checkpoints: true
22
+ base_model: samoline/5255b81c-a162-4f3a-9a96-f4d344cb99c8
23
+ bf16: auto
24
+ chat_template: llama3
25
+ dataset_prepared_path: null
26
+ dataset_processes: 6
27
+ datasets:
28
+ - data_files:
29
+ - 6af671d7f222ce26_train_data.json
30
+ ds_type: json
31
+ format: custom
32
+ path: /workspace/input_data/6af671d7f222ce26_train_data.json
33
+ type:
34
+ field_input: input
35
+ field_instruction: instruction
36
+ field_output: output
37
+ format: '{instruction} {input}'
38
+ no_input_format: '{instruction}'
39
+ system_format: '{system}'
40
+ system_prompt: ''
41
+ debug: null
42
+ deepspeed: null
43
+ early_stopping_patience: 5
44
+ eval_max_new_tokens: 128
45
+ eval_steps: 200
46
+ eval_table_size: null
47
+ evals_per_epoch: null
48
+ flash_attention: true
49
+ fp16: false
50
+ fsdp: null
51
+ fsdp_config: null
52
+ gradient_accumulation_steps: 4
53
+ gradient_checkpointing: true
54
+ group_by_length: false
55
+ hub_model_id: error577/47615d62-b7f3-4efd-b567-61a937215b9f
56
+ hub_repo: null
57
+ hub_strategy: checkpoint
58
+ hub_token: null
59
+ learning_rate: 0.0002
60
+ load_in_4bit: true
61
+ load_in_8bit: false
62
+ local_rank: null
63
+ logging_steps: 1
64
+ lora_alpha: 64
65
+ lora_dropout: 0.1
66
+ lora_fan_in_fan_out: null
67
+ lora_model_dir: null
68
+ lora_r: 32
69
+ lora_target_linear: true
70
+ lr_scheduler: cosine
71
+ max_grad_norm: 1.0
72
+ max_steps: null
73
+ micro_batch_size: 4
74
+ mlflow_experiment_name: /tmp/6af671d7f222ce26_train_data.json
75
+ model_type: AutoModelForCausalLM
76
+ num_epochs: 3
77
+ optimizer: adamw_bnb_8bit
78
+ output_dir: miner_id_24
79
+ pad_to_sequence_len: true
80
+ resume_from_checkpoint: null
81
+ s2_attention: null
82
+ sample_packing: false
83
+ save_steps: 200
84
+ sequence_len: 256
85
+ strict: false
86
+ tf32: false
87
+ tokenizer_type: AutoTokenizer
88
+ train_on_inputs: false
89
+ trust_remote_code: true
90
+ val_set_size: 0.005
91
+ wandb_entity: null
92
+ wandb_mode: online
93
+ wandb_name: 5b76a1a8-f96b-44f2-9330-30f496d3f59a
94
+ wandb_project: Gradients-On-Demand
95
+ wandb_run: your_name
96
+ wandb_runid: 5b76a1a8-f96b-44f2-9330-30f496d3f59a
97
+ warmup_steps: 30
98
+ weight_decay: 0.0
99
+ xformers_attention: null
100
+
101
+ ```
102
+
103
+ </details><br>
104
+
105
+ # 47615d62-b7f3-4efd-b567-61a937215b9f
106
+
107
+ This model is a fine-tuned version of [samoline/5255b81c-a162-4f3a-9a96-f4d344cb99c8](https://huggingface.co/samoline/5255b81c-a162-4f3a-9a96-f4d344cb99c8) on the None dataset.
108
+ It achieves the following results on the evaluation set:
109
+ - Loss: 0.8338
110
+
111
+ ## Model description
112
+
113
+ More information needed
114
+
115
+ ## Intended uses & limitations
116
+
117
+ More information needed
118
+
119
+ ## Training and evaluation data
120
+
121
+ More information needed
122
+
123
+ ## Training procedure
124
+
125
+ ### Training hyperparameters
126
+
127
+ The following hyperparameters were used during training:
128
+ - learning_rate: 0.0002
129
+ - train_batch_size: 4
130
+ - eval_batch_size: 4
131
+ - seed: 42
132
+ - gradient_accumulation_steps: 4
133
+ - total_train_batch_size: 16
134
+ - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
135
+ - lr_scheduler_type: cosine
136
+ - lr_scheduler_warmup_steps: 30
137
+ - num_epochs: 3
138
+
139
+ ### Training results
140
+
141
+ | Training Loss | Epoch | Step | Validation Loss |
142
+ |:-------------:|:------:|:----:|:---------------:|
143
+ | 0.7552 | 0.0001 | 1 | 0.7747 |
144
+ | 1.0145 | 0.0219 | 200 | 0.8068 |
145
+ | 1.0147 | 0.0437 | 400 | 0.8067 |
146
+ | 0.7413 | 0.0656 | 600 | 0.8176 |
147
+ | 0.8655 | 0.0875 | 800 | 0.8182 |
148
+ | 0.5897 | 0.1094 | 1000 | 0.8338 |
149
+ | 0.866 | 0.1312 | 1200 | 0.8312 |
150
+ | 0.5207 | 0.1531 | 1400 | 0.8338 |
151
+
152
+
153
+ ### Framework versions
154
+
155
+ - PEFT 0.13.2
156
+ - Transformers 4.46.0
157
+ - Pytorch 2.5.0+cu124
158
+ - Datasets 3.0.1
159
+ - Tokenizers 0.20.1
adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c92597f644c4f17da534b83a857e08813fdfd0d6f70d3eebb536b3158d9d2b2
3
+ size 194652458
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2cc353f73298357243ffea91c070db1f0968f3697e44aff63111d24765ae1b70
3
  size 194563400
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e0de863a2ce1d0645d5a750ff6f9e4dc3e55f0a96df6b18b1dff9570545bc2c
3
  size 194563400