Upload climb model for musique (trained on 50 queries)

README.md

@@ -6,7 +6,6 @@ tags:
 - generated_from_trainer
 - dataset_size:50
 - loss:BinaryCrossEntropyLoss
-base_model: BAAI/bge-reranker-v2-m3
 pipeline_tag: text-ranking
 library_name: sentence-transformers
 metrics:
@@ -18,7 +17,7 @@ metrics:
 - recall
 - average_precision
 model-index:
-- name: CrossEncoder based on BAAI/bge-reranker-v2-m3
+- name: CrossEncoder
   results:
   - task:
       type: cross-encoder-binary-classification
@@ -31,13 +30,13 @@ model-index:
       value: 0.964769647696477
       name: Accuracy
     - type: accuracy_threshold
-      value: 0.04581885784864426
+      value: 0.09269625693559647
       name: Accuracy Threshold
     - type: f1
       value: 0.9446808510638298
       name: F1
     - type: f1_threshold
-      value: 0.023046962916851044
+      value: 0.049161043018102646
       name: F1 Threshold
     - type: precision
       value: 0.9568965517241379
@@ -46,19 +45,19 @@ model-index:
       value: 0.9327731092436975
       name: Recall
     - type: average_precision
-      value: 0.9846420260039257
+      value: 0.9828499641745869
       name: Average Precision
 ---
 
-# CrossEncoder based on BAAI/bge-reranker-v2-m3
+# CrossEncoder
 
-This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.
+This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model trained using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.
 
 ## Model Details
 
 ### Model Description
 - **Model Type:** Cross Encoder
-- **Base model:** [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) <!-- at revision 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e -->
+<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
 - **Maximum Sequence Length:** 1024 tokens
 - **Number of Output Labels:** 1 label
 <!-- - **Training Dataset:** Unknown -->
@@ -90,11 +89,11 @@ from sentence_transformers import CrossEncoder
 model = CrossEncoder("cross_encoder_model_id")
 # Get scores for pairs of texts
 pairs = [
-    ['when did England colonize the country where Freetown is located?', 'Yvonne Brathwaite Burke. Yvonne Brathwaite Burke (born October 5, 1932) is a politician from Los Angeles, California, United States. She was the first African-American woman to represent the West Coast in Congress. She served in the U.S. Congress from 1973 until January 1979. She was the Los Angeles County Supervisor representing the 2nd District (1992–2008). She has served as the Chair three times (1993–94, 1997–98, 2002–03). Her husband is William Burke, a prominent philanthropist and creator of the Los Angeles Marathon.'],
-    ['Margraviate of the country of the Botanical Garden of the place Josef Victor Rohon was educated is an instance of?', 'Josef Victor Rohon. He studied medicine, zoology, and neuroanatomy at the University of Vienna, where he was influenced by Theodor Meynert (1833-1892), Carl Claus (1835-1899), Hans Kundrat (1845-1893), and Eduard Albert (1841-1900). In 1884 he graduated magna cum laude at Munich, where he spent the ensuing years conducting paleontological research. At Munich, he worked closely with Karl Alfred von Zittel (1839-1904), performing research that included anatomical studies of conodonts.'],
-    ["Who was the singer of I Can't Sleep at Night in Home and Away?", 'I Can\'t Sleep at Night. "I Can\'t Sleep at Night" was written by Dannii Minogue, Rob Davis and Jewels & Stone for Minogue\'s fifth studio album "Club Disco" and included on the greatest hits compilation, "The Hits & Beyond" (2006). On 8 January 2007, the song and its remixes were released as a digital download in Australia, the United Kingdom and North America. The Radio Edit of the song features minor mixing and production differences and is the version featured in the music video.'],
-    ['The country setting the story of the basis of Ever After is a type of what?', 'Egypt (Roman province). The Roman province of Egypt (, ; ) was established in 30 BC after Octavian (the future Roman emperor Augustus) defeated his rival Mark Antony, deposed Pharaoh Cleopatra, and annexed the Ptolemaic Kingdom to the Roman Empire. The province encompassed most of modern-day Egypt except for the Sinai Peninsula (which would later be conquered by Trajan). Aegyptus was bordered by the provinces of Crete and Cyrenaica to the west and Judea (later Arabia Petraea) to the East.'],
-    ['The country setting the story of the basis of Ever After is a type of what?', "Cinderella. The oldest known version of the Cinderella story is the ancient Greek story of Rhodopis, a Greek courtesan living in the colony of Naucratis in Egypt, whose name means ``Rosy - Cheeks ''. The story is first recorded by the Greek geographer Strabo in his Geographica (book 17, 33), probably written around 7 BC or thereabouts:"],
+    ['Khosrowabad, in the city where Pouya Tajik was born, is found in what county?', 'Pouya Tajik. Pouya Tajik, (born October 28, 1987 in Tehran) is a professional Iranian basketball player who currently plays for BEEM Mazandaran BC of the Iranian Super League and also for the Iranian national basketball team. He is a 6-foot-nine-inch power forward'],
+    ['Khosrowabad, in the city where Pouya Tajik was born, is found in what county?', 'Frédéric Chopin. Chopin also endowed popular dance forms with a greater range of melody and expression. Chopin\'s mazurkas, while originating in the traditional Polish dance (the mazurek), differed from the traditional variety in that they were written for the concert hall rather than the dance hall; "it was Chopin who put the mazurka on the European musical map." The series of seven polonaises published in his lifetime (another nine were published posthumously), beginning with the Op. 26 pair (published 1836), set a new standard for music in the form. His waltzes were also written specifically for the salon recital rather than the ballroom and are frequently at rather faster tempos than their dance-floor equivalents.'],
+    ['Margraviate of the country of the Botanical Garden of the place Josef Victor Rohon was educated is an instance of?', 'Pouya Tajik. Pouya Tajik, (born October 28, 1987 in Tehran) is a professional Iranian basketball player who currently plays for BEEM Mazandaran BC of the Iranian Super League and also for the Iranian national basketball team. He is a 6-foot-nine-inch power forward'],
+    ['Khosrowabad, in the city where Pouya Tajik was born, is found in what county?', "Frédéric Chopin. Numerous recordings of Chopin's works are available. On the occasion of the composer's bicentenary, the critics of The New York Times recommended performances by the following contemporary pianists (among many others): Martha Argerich, Vladimir Ashkenazy, Emanuel Ax, Evgeny Kissin, Murray Perahia, Maurizio Pollini and Krystian Zimerman. The Warsaw Chopin Society organizes the Grand prix du disque de F. Chopin for notable Chopin recordings, held every five years."],
+    ['Margraviate of the country of the Botanical Garden of the place Josef Victor Rohon was educated is an instance of?', 'Arignar Anna Zoological Park. The butterfly house, constructed at a cost of ₹6 million, has more than 25 host plants and landscaped habitats, such as bushes, lianas, streams, waterfall and rock - gardens, that attract many species of butterflies such as the common Mormon, crimson rose, mottled emigrant, blue tiger, evening brown and lime butterfly. A network of ponds interconnected by streams maintains humidity in the area. The park covers an area of 5 acres. The butterfly garden with an insect museum at the entrance is set up by the Tamil Nadu Agricultural University (TNAU), Coimbatore. The insect museum has been planned with an exhibit area comprising insect exhibits representing the most common Indian species of all orders of insects both in the form of preserved specimens and in the form of photographs.'],
 ]
 scores = model.predict(pairs)
 print(scores.shape)
@@ -102,13 +101,13 @@ print(scores.shape)
 
 # Or rank different texts based on similarity to a single text
 ranks = model.rank(
-    'when did England colonize the country where Freetown is located?',
+    'Khosrowabad, in the city where Pouya Tajik was born, is found in what county?',
     [
-        'Yvonne Brathwaite Burke. Yvonne Brathwaite Burke (born October 5, 1932) is a politician from Los Angeles, California, United States. She was the first African-American woman to represent the West Coast in Congress. She served in the U.S. Congress from 1973 until January 1979. She was the Los Angeles County Supervisor representing the 2nd District (1992–2008). She has served as the Chair three times (1993–94, 1997–98, 2002–03). Her husband is William Burke, a prominent philanthropist and creator of the Los Angeles Marathon.',
-        'Josef Victor Rohon. He studied medicine, zoology, and neuroanatomy at the University of Vienna, where he was influenced by Theodor Meynert (1833-1892), Carl Claus (1835-1899), Hans Kundrat (1845-1893), and Eduard Albert (1841-1900). In 1884 he graduated magna cum laude at Munich, where he spent the ensuing years conducting paleontological research. At Munich, he worked closely with Karl Alfred von Zittel (1839-1904), performing research that included anatomical studies of conodonts.',
-        'I Can\'t Sleep at Night. "I Can\'t Sleep at Night" was written by Dannii Minogue, Rob Davis and Jewels & Stone for Minogue\'s fifth studio album "Club Disco" and included on the greatest hits compilation, "The Hits & Beyond" (2006). On 8 January 2007, the song and its remixes were released as a digital download in Australia, the United Kingdom and North America. The Radio Edit of the song features minor mixing and production differences and is the version featured in the music video.',
-        'Egypt (Roman province). The Roman province of Egypt (, ; ) was established in 30 BC after Octavian (the future Roman emperor Augustus) defeated his rival Mark Antony, deposed Pharaoh Cleopatra, and annexed the Ptolemaic Kingdom to the Roman Empire. The province encompassed most of modern-day Egypt except for the Sinai Peninsula (which would later be conquered by Trajan). Aegyptus was bordered by the provinces of Crete and Cyrenaica to the west and Judea (later Arabia Petraea) to the East.',
-        "Cinderella. The oldest known version of the Cinderella story is the ancient Greek story of Rhodopis, a Greek courtesan living in the colony of Naucratis in Egypt, whose name means ``Rosy - Cheeks ''. The story is first recorded by the Greek geographer Strabo in his Geographica (book 17, 33), probably written around 7 BC or thereabouts:",
+        'Pouya Tajik. Pouya Tajik, (born October 28, 1987 in Tehran) is a professional Iranian basketball player who currently plays for BEEM Mazandaran BC of the Iranian Super League and also for the Iranian national basketball team. He is a 6-foot-nine-inch power forward',
+        'Frédéric Chopin. Chopin also endowed popular dance forms with a greater range of melody and expression. Chopin\'s mazurkas, while originating in the traditional Polish dance (the mazurek), differed from the traditional variety in that they were written for the concert hall rather than the dance hall; "it was Chopin who put the mazurka on the European musical map." The series of seven polonaises published in his lifetime (another nine were published posthumously), beginning with the Op. 26 pair (published 1836), set a new standard for music in the form. His waltzes were also written specifically for the salon recital rather than the ballroom and are frequently at rather faster tempos than their dance-floor equivalents.',
+        'Pouya Tajik. Pouya Tajik, (born October 28, 1987 in Tehran) is a professional Iranian basketball player who currently plays for BEEM Mazandaran BC of the Iranian Super League and also for the Iranian national basketball team. He is a 6-foot-nine-inch power forward',
+        "Frédéric Chopin. Numerous recordings of Chopin's works are available. On the occasion of the composer's bicentenary, the critics of The New York Times recommended performances by the following contemporary pianists (among many others): Martha Argerich, Vladimir Ashkenazy, Emanuel Ax, Evgeny Kissin, Murray Perahia, Maurizio Pollini and Krystian Zimerman. The Warsaw Chopin Society organizes the Grand prix du disque de F. Chopin for notable Chopin recordings, held every five years.",
+        'Arignar Anna Zoological Park. The butterfly house, constructed at a cost of ₹6 million, has more than 25 host plants and landscaped habitats, such as bushes, lianas, streams, waterfall and rock - gardens, that attract many species of butterflies such as the common Mormon, crimson rose, mottled emigrant, blue tiger, evening brown and lime butterfly. A network of ponds interconnected by streams maintains humidity in the area. The park covers an area of 5 acres. The butterfly garden with an insect museum at the entrance is set up by the Tamil Nadu Agricultural University (TNAU), Coimbatore. The insect museum has been planned with an exhibit area comprising insect exhibits representing the most common Indian species of all orders of insects both in the form of preserved specimens and in the form of photographs.',
     ]
 )
 # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
@@ -150,12 +149,12 @@ You can finetune this model on your own dataset.
 | Metric                | Value      |
 |:----------------------|:-----------|
 | accuracy              | 0.9648     |
-| accuracy_threshold    | 0.0458     |
+| accuracy_threshold    | 0.0927     |
 | f1                    | 0.9447     |
-| f1_threshold          | 0.023      |
+| f1_threshold          | 0.0492     |
 | precision             | 0.9569     |
 | recall                | 0.9328     |
-| **average_precision** | **0.9846** |
+| **average_precision** | **0.9828** |
 
 <!--
 ## Bias, Risks and Limitations
@@ -183,11 +182,11 @@ You can finetune this model on your own dataset.
   | type    | string                                                                                         | string                                                                                             | float                                                          |
   | details | <ul><li>min: 39 characters</li><li>mean: 76.6 characters</li><li>max: 114 characters</li></ul> | <ul><li>min: 148 characters</li><li>mean: 511.34 characters</li><li>max: 1394 characters</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.32</li><li>max: 1.0</li></ul> |
 * Samples:
-  | sentence_0                                                                                                                      | sentence_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | label            |
-  |:--------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
-  | <code>when did England colonize the country where Freetown is located?</code>                                                   | <code>Yvonne Brathwaite Burke. Yvonne Brathwaite Burke (born October 5, 1932) is a politician from Los Angeles, California, United States. She was the first African-American woman to represent the West Coast in Congress. She served in the U.S. Congress from 1973 until January 1979. She was the Los Angeles County Supervisor representing the 2nd District (1992–2008). She has served as the Chair three times (1993–94, 1997–98, 2002–03). Her husband is William Burke, a prominent philanthropist and creator of the Los Angeles Marathon.</code> | <code>0.0</code> |
-  | <code>Margraviate of the country of the Botanical Garden of the place Josef Victor Rohon was educated is an instance of?</code> | <code>Josef Victor Rohon. He studied medicine, zoology, and neuroanatomy at the University of Vienna, where he was influenced by Theodor Meynert (1833-1892), Carl Claus (1835-1899), Hans Kundrat (1845-1893), and Eduard Albert (1841-1900). In 1884 he graduated magna cum laude at Munich, where he spent the ensuing years conducting paleontological research. At Munich, he worked closely with Karl Alfred von Zittel (1839-1904), performing research that included anatomical studies of conodonts.</code>                                          | <code>1.0</code> |
-  | <code>Who was the singer of I Can't Sleep at Night in Home and Away?</code>                                                     | <code>I Can't Sleep at Night. "I Can't Sleep at Night" was written by Dannii Minogue, Rob Davis and Jewels & Stone for Minogue's fifth studio album "Club Disco" and included on the greatest hits compilation, "The Hits & Beyond" (2006). On 8 January 2007, the song and its remixes were released as a digital download in Australia, the United Kingdom and North America. The Radio Edit of the song features minor mixing and production differences and is the version featured in the music video.</code>                                            | <code>1.0</code> |
+  | sentence_0                                                                                                                      | sentence_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | label            |
+  |:--------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
+  | <code>Khosrowabad, in the city where Pouya Tajik was born, is found in what county?</code>                                      | <code>Pouya Tajik. Pouya Tajik, (born October 28, 1987 in Tehran) is a professional Iranian basketball player who currently plays for BEEM Mazandaran BC of the Iranian Super League and also for the Iranian national basketball team. He is a 6-foot-nine-inch power forward</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | <code>1.0</code> |
+  | <code>Khosrowabad, in the city where Pouya Tajik was born, is found in what county?</code>                                      | <code>Frédéric Chopin. Chopin also endowed popular dance forms with a greater range of melody and expression. Chopin's mazurkas, while originating in the traditional Polish dance (the mazurek), differed from the traditional variety in that they were written for the concert hall rather than the dance hall; "it was Chopin who put the mazurka on the European musical map." The series of seven polonaises published in his lifetime (another nine were published posthumously), beginning with the Op. 26 pair (published 1836), set a new standard for music in the form. His waltzes were also written specifically for the salon recital rather than the ballroom and are frequently at rather faster tempos than their dance-floor equivalents.</code> | <code>0.0</code> |
+  | <code>Margraviate of the country of the Botanical Garden of the place Josef Victor Rohon was educated is an instance of?</code> | <code>Pouya Tajik. Pouya Tajik, (born October 28, 1987 in Tehran) is a professional Iranian basketball player who currently plays for BEEM Mazandaran BC of the Iranian Super League and also for the Iranian national basketball team. He is a 6-foot-nine-inch power forward</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | <code>0.0</code> |
 * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters:
   ```json
   {
@@ -325,9 +324,9 @@ You can finetune this model on your own dataset.
 ### Training Logs
 | Epoch | Step | validation_average_precision |
 |:-----:|:----:|:----------------------------:|
-| 1.0   | 13   | 0.9700                       |
-| 2.0   | 26   | 0.9774                       |
-| 3.0   | 39   | 0.9846                       |
+| 1.0   | 13   | 0.9844                       |
+| 2.0   | 26   | 0.9831                       |
+| 3.0   | 39   | 0.9828                       |
 
 
 ### Framework Versions

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "BAAI/bge-reranker-v2-m3",
+  "_name_or_path": "models/finetuned/musique-climb",
   "architectures": [
     "XLMRobertaForSequenceClassification"
   ],

eval/CrossEncoderClassificationEvaluator_validation_results.csv

@@ -2,3 +2,6 @@ epoch,steps,Accuracy,Accuracy_Threshold,F1,F1_Threshold,Precision,Recall,Average
 1.0,13,0.94579945799458,0.00901582557708025,0.912280701754386,0.00901582557708025,0.9541284403669725,0.8739495798319328,0.9700491973968657
 2.0,26,0.9512195121951219,0.033014170825481415,0.9224137931034483,0.033014170825481415,0.9469026548672567,0.8991596638655462,0.9774329046561461
 3.0,39,0.964769647696477,0.04581885784864426,0.9446808510638298,0.023046962916851044,0.9568965517241379,0.9327731092436975,0.9846420260039257
+1.0,13,0.964769647696477,0.09815724194049835,0.9446808510638298,0.04552947357296944,0.9568965517241379,0.9327731092436975,0.9843958194458395
+2.0,26,0.964769647696477,0.2853778004646301,0.944206008583691,0.2853778004646301,0.9649122807017544,0.9243697478991597,0.9831305652439385
+3.0,39,0.964769647696477,0.09269625693559647,0.9446808510638298,0.049161043018102646,0.9568965517241379,0.9327731092436975,0.9828499641745869

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f07256ab995737c486a8e4a783886011cb8b1c941b98e56e0acbf986e39d761e
+oid sha256:17af818c93fe8000999e32360bc07a8e37c1c413c3c89c95d14b4c413275a57a
 size 2271071852

tokenizer_config.json

@@ -46,10 +46,17 @@
   "cls_token": "<s>",
   "eos_token": "</s>",
   "mask_token": "<mask>",
+  "max_length": 1024,
   "model_max_length": 1024,
+  "pad_to_multiple_of": null,
   "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
   "sep_token": "</s>",
   "sp_model_kwargs": {},
+  "stride": 0,
   "tokenizer_class": "XLMRobertaTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
   "unk_token": "<unk>"
 }