Update README.md
Browse files
README.md
CHANGED
|
@@ -57,6 +57,88 @@ The most important principle in science is a is a this are not for that the to o
|
|
| 57 |
|
| 58 |
We are currently researching knowledge distillation and this was the first step! Things will better up!
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
## Final Thought
|
| 61 |
|
| 62 |
Knowledge distillation is a promising thing for us, we believe that LLMs can be helpful even being so small!
|
|
|
|
| 57 |
|
| 58 |
We are currently researching knowledge distillation and this was the first step! Things will better up!
|
| 59 |
|
| 60 |
+
## Benchmarks
|
| 61 |
+
|
| 62 |
+
| Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr|
|
| 63 |
+
|------------------------------------------------------------|------:|------|-----:|---------------|---|---------:|---|------|
|
| 64 |
+
|arc_easy | 1|none | 0|acc |↑ | 0.2517|± |0.0089|
|
| 65 |
+
| | |none | 0|acc_norm |↑ | 0.2677|± |0.0091|
|
| 66 |
+
|blimp | 2|none | |acc |↑ | 0.5184|± |0.0017|
|
| 67 |
+
| - blimp_adjunct_island | 1|none | 0|acc |↑ | 0.7610|± |0.0135|
|
| 68 |
+
| - blimp_anaphor_gender_agreement | 1|none | 0|acc |↑ | 0.1920|± |0.0125|
|
| 69 |
+
| - blimp_anaphor_number_agreement | 1|none | 0|acc |↑ | 0.3650|± |0.0152|
|
| 70 |
+
| - blimp_animate_subject_passive | 1|none | 0|acc |↑ | 0.5790|± |0.0156|
|
| 71 |
+
| - blimp_animate_subject_trans | 1|none | 0|acc |↑ | 0.6450|± |0.0151|
|
| 72 |
+
| - blimp_causative | 1|none | 0|acc |↑ | 0.5210|± |0.0158|
|
| 73 |
+
| - blimp_complex_NP_island | 1|none | 0|acc |↑ | 0.5150|± |0.0158|
|
| 74 |
+
| - blimp_coordinate_structure_constraint_complex_left_branch| 1|none | 0|acc |↑ | 0.2970|± |0.0145|
|
| 75 |
+
| - blimp_coordinate_structure_constraint_object_extraction | 1|none | 0|acc |↑ | 0.3530|± |0.0151|
|
| 76 |
+
| - blimp_determiner_noun_agreement_1 | 1|none | 0|acc |↑ | 0.5080|± |0.0158|
|
| 77 |
+
| - blimp_determiner_noun_agreement_2 | 1|none | 0|acc |↑ | 0.4900|± |0.0158|
|
| 78 |
+
| - blimp_determiner_noun_agreement_irregular_1 | 1|none | 0|acc |↑ | 0.4510|± |0.0157|
|
| 79 |
+
| - blimp_determiner_noun_agreement_irregular_2 | 1|none | 0|acc |↑ | 0.5310|± |0.0158|
|
| 80 |
+
| - blimp_determiner_noun_agreement_with_adj_2 | 1|none | 0|acc |↑ | 0.4730|± |0.0158|
|
| 81 |
+
| - blimp_determiner_noun_agreement_with_adj_irregular_1 | 1|none | 0|acc |↑ | 0.4710|± |0.0158|
|
| 82 |
+
| - blimp_determiner_noun_agreement_with_adj_irregular_2 | 1|none | 0|acc |↑ | 0.5370|± |0.0158|
|
| 83 |
+
| - blimp_determiner_noun_agreement_with_adjective_1 | 1|none | 0|acc |↑ | 0.5220|± |0.0158|
|
| 84 |
+
| - blimp_distractor_agreement_relational_noun | 1|none | 0|acc |↑ | 0.4940|± |0.0158|
|
| 85 |
+
| - blimp_distractor_agreement_relative_clause | 1|none | 0|acc |↑ | 0.4800|± |0.0158|
|
| 86 |
+
| - blimp_drop_argument | 1|none | 0|acc |↑ | 0.6900|± |0.0146|
|
| 87 |
+
| - blimp_ellipsis_n_bar_1 | 1|none | 0|acc |↑ | 0.2580|± |0.0138|
|
| 88 |
+
| - blimp_ellipsis_n_bar_2 | 1|none | 0|acc |↑ | 0.2810|± |0.0142|
|
| 89 |
+
| - blimp_existential_there_object_raising | 1|none | 0|acc |↑ | 0.6180|± |0.0154|
|
| 90 |
+
| - blimp_existential_there_quantifiers_1 | 1|none | 0|acc |↑ | 0.5550|± |0.0157|
|
| 91 |
+
| - blimp_existential_there_quantifiers_2 | 1|none | 0|acc |↑ | 0.0750|± |0.0083|
|
| 92 |
+
| - blimp_existential_there_subject_raising | 1|none | 0|acc |↑ | 0.6750|± |0.0148|
|
| 93 |
+
| - blimp_expletive_it_object_raising | 1|none | 0|acc |↑ | 0.5820|± |0.0156|
|
| 94 |
+
| - blimp_inchoative | 1|none | 0|acc |↑ | 0.4010|± |0.0155|
|
| 95 |
+
| - blimp_intransitive | 1|none | 0|acc |↑ | 0.5550|± |0.0157|
|
| 96 |
+
| - blimp_irregular_past_participle_adjectives | 1|none | 0|acc |↑ | 0.2340|± |0.0134|
|
| 97 |
+
| - blimp_irregular_past_participle_verbs | 1|none | 0|acc |↑ | 0.7760|± |0.0132|
|
| 98 |
+
| - blimp_irregular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.4810|± |0.0158|
|
| 99 |
+
| - blimp_irregular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.4370|± |0.0157|
|
| 100 |
+
| - blimp_left_branch_island_echo_question | 1|none | 0|acc |↑ | 0.9310|± |0.0080|
|
| 101 |
+
| - blimp_left_branch_island_simple_question | 1|none | 0|acc |↑ | 0.3260|± |0.0148|
|
| 102 |
+
| - blimp_matrix_question_npi_licensor_present | 1|none | 0|acc |↑ | 0.0200|± |0.0044|
|
| 103 |
+
| - blimp_npi_present_1 | 1|none | 0|acc |↑ | 0.6120|± |0.0154|
|
| 104 |
+
| - blimp_npi_present_2 | 1|none | 0|acc |↑ | 0.5830|± |0.0156|
|
| 105 |
+
| - blimp_only_npi_licensor_present | 1|none | 0|acc |↑ | 0.0000|± | 0|
|
| 106 |
+
| - blimp_only_npi_scope | 1|none | 0|acc |↑ | 0.0270|± |0.0051|
|
| 107 |
+
| - blimp_passive_1 | 1|none | 0|acc |↑ | 0.5190|± |0.0158|
|
| 108 |
+
| - blimp_passive_2 | 1|none | 0|acc |↑ | 0.5210|± |0.0158|
|
| 109 |
+
| - blimp_principle_A_c_command | 1|none | 0|acc |↑ | 0.6470|± |0.0151|
|
| 110 |
+
| - blimp_principle_A_case_1 | 1|none | 0|acc |↑ | 1.0000|± | 0|
|
| 111 |
+
| - blimp_principle_A_case_2 | 1|none | 0|acc |↑ | 0.5290|± |0.0158|
|
| 112 |
+
| - blimp_principle_A_domain_1 | 1|none | 0|acc |↑ | 1.0000|± | 0|
|
| 113 |
+
| - blimp_principle_A_domain_2 | 1|none | 0|acc |↑ | 0.5880|± |0.0156|
|
| 114 |
+
| - blimp_principle_A_domain_3 | 1|none | 0|acc |↑ | 0.4790|± |0.0158|
|
| 115 |
+
| - blimp_principle_A_reconstruction | 1|none | 0|acc |↑ | 0.3470|± |0.0151|
|
| 116 |
+
| - blimp_regular_plural_subject_verb_agreement_1 | 1|none | 0|acc |↑ | 0.5750|± |0.0156|
|
| 117 |
+
| - blimp_regular_plural_subject_verb_agreement_2 | 1|none | 0|acc |↑ | 0.4710|± |0.0158|
|
| 118 |
+
| - blimp_sentential_negation_npi_licensor_present | 1|none | 0|acc |↑ | 1.0000|± | 0|
|
| 119 |
+
| - blimp_sentential_negation_npi_scope | 1|none | 0|acc |↑ | 0.4020|± |0.0155|
|
| 120 |
+
| - blimp_sentential_subject_island | 1|none | 0|acc |↑ | 0.3030|± |0.0145|
|
| 121 |
+
| - blimp_superlative_quantifiers_1 | 1|none | 0|acc |↑ | 0.5090|± |0.0158|
|
| 122 |
+
| - blimp_superlative_quantifiers_2 | 1|none | 0|acc |↑ | 0.6150|± |0.0154|
|
| 123 |
+
| - blimp_tough_vs_raising_1 | 1|none | 0|acc |↑ | 0.3080|± |0.0146|
|
| 124 |
+
| - blimp_tough_vs_raising_2 | 1|none | 0|acc |↑ | 0.6980|± |0.0145|
|
| 125 |
+
| - blimp_transitive | 1|none | 0|acc |↑ | 0.5210|± |0.0158|
|
| 126 |
+
| - blimp_wh_island | 1|none | 0|acc |↑ | 0.4050|± |0.0155|
|
| 127 |
+
| - blimp_wh_questions_object_gap | 1|none | 0|acc |↑ | 0.9970|± |0.0017|
|
| 128 |
+
| - blimp_wh_questions_subject_gap | 1|none | 0|acc |↑ | 0.9990|± |0.0010|
|
| 129 |
+
| - blimp_wh_questions_subject_gap_long_distance | 1|none | 0|acc |↑ | 0.9980|± |0.0014|
|
| 130 |
+
| - blimp_wh_vs_that_no_gap | 1|none | 0|acc |↑ | 1.0000|± | 0|
|
| 131 |
+
| - blimp_wh_vs_that_no_gap_long_distance | 1|none | 0|acc |↑ | 1.0000|± | 0|
|
| 132 |
+
| - blimp_wh_vs_that_with_gap | 1|none | 0|acc |↑ | 0.0000|± | 0|
|
| 133 |
+
| - blimp_wh_vs_that_with_gap_long_distance | 1|none | 0|acc |↑ | 0.0000|± | 0|
|
| 134 |
+
|wikitext | 2|none | 0|bits_per_byte |↓ | 3.0125|± | N/A|
|
| 135 |
+
| | |none | 0|byte_perplexity|↓ | 8.0696|± | N/A|
|
| 136 |
+
| | |none | 0|word_perplexity|↓ |70687.5638|± | N/A|
|
| 137 |
+
|
| 138 |
+
|Groups|Version|Filter|n-shot|Metric| |Value | |Stderr|
|
| 139 |
+
|------|------:|------|------|------|---|-----:|---|-----:|
|
| 140 |
+
|blimp | 2|none | |acc |↑ |0.5184|± |0.0017|
|
| 141 |
+
|
| 142 |
## Final Thought
|
| 143 |
|
| 144 |
Knowledge distillation is a promising thing for us, we believe that LLMs can be helpful even being so small!
|