Mark LIBRA Mini datasets with asterisk on leaderboard; refresh docs and LiquidAI LongContext scores

datasets_config.json

@@ -22,7 +22,7 @@
     ]
   },
   "matreshka_names": {
-    "name": "MatreshkaNames",
+    "name": "MatreshkaNames *",
     "lengths": [
       "4k",
       "8k",
@@ -53,7 +53,7 @@
     ]
   },
   "ru_sci_passage_count": {
-    "name": "ruSciPassageCount",
+    "name": "ruSciPassageCount *",
     "lengths": [
       "4k",
       "8k",
@@ -64,7 +64,7 @@
     ]
   },
   "ru_2wikimultihopqa": {
-    "name": "ru2WikiMultihopQA",
+    "name": "ru2WikiMultihopQA *",
     "lengths": [
       "8k",
       "16k",
@@ -72,14 +72,13 @@
     ]
   },
   "long_context_multiq": {
-    "name": "LongContextMultiQ",
+    "name": "LongContextMultiQ *",
     "lengths": [
       "4k",
       "8k",
       "16k",
       "32k",
-      "64k",
-      "128k"
+      "64k"
     ]
   },
   "ru_sci_abstract_retrieval": {
@@ -101,7 +100,7 @@
     ]
   },
   "librusec_mhqa": {
-    "name": "LibrusecMHQA",
+    "name": "LibrusecMHQA *",
     "lengths": [
       "8k"
     ]
@@ -129,7 +128,7 @@
     ]
   },
   "ru_babilong_qa3": {
-    "name": "ruBABILongQA3",
+    "name": "ruBABILongQA3 *",
     "lengths": [
       "4k",
       "8k",

docs/description.md

@@ -1,71 +1,128 @@
 # LIBRA: Long Input Benchmark for Russian Analysis
+<p align="center">
+<img src="logo.png" width="500" />
+</p>
 
-<img src="https://i.imgur.com/BNleRrG.png" width="800" />
+## LIBRA
 
-## Dataset Summary
+LIBRA (Long Input Benchmark for Russian Analysis) is designed to evaluate the capabilities of large language models (LLMs) in understanding and processing long texts in Russian. This benchmark includes 18 datasets adapted for different tasks and complexities. The tasks are divided into four complexity groups and allow evaluation across various context lengths ranging from 4k up to 512k tokens.
 
-LIBRA (Long Input Benchmark for Russian Analysis) is designed to evaluate the capabilities of large language models (LLMs) in understanding and processing long texts in Russian. This benchmark includes 21 datasets adapted for different tasks and complexities. The tasks are divided into four complexity groups and allow evaluation across various context lengths ranging from 4k up to 128k tokens.
+For model comparison and results, see the [LIBRA Leaderboard](https://huggingface.co/spaces/ai-forever/LIBRA-Leaderboard). The benchmark is described in detail in our [paper](https://arxiv.org/abs/2408.02439).
 
-## Tasks and Complexity Groups
+**NOTE:** This is a new benchmark version released in May 2026. The original version (described in the paper) can be found [here](https://huggingface.co/datasets/ai-forever/LIBRA_old). We strongly encourage using the current version as it contains cleaned and extended datasets with ensured data quality.
 
-### Group I: Simple Information Retrieval
-- **Passkey**: Extract a relevant piece of code number from a long text fragment. Based on the original [PassKey test](https://github.com/CStanKonrad/long_llama/blob/main/examples/passkey.py) from the m LongLLaMA’s GitHub repo.
-- **PasskeyWithLibrusec**: Similar to Passkey but with added noise from Librusec texts.
+## LIBRA Mini
+
+Running a full LIBRA evaluation can be prohibitively expensive and time-consuming due to the large number of datasets and long context lengths involved. Moreover, some of the included tasks have become less informative as benchmarks, with modern models achieving near-saturated scores on them.
+
+To address this, we introduce **LIBRA Mini** — a compact, curated subset of 6 datasets selected from the full benchmark. These datasets represent the most challenging and diagnostically informative tasks in LIBRA, covering diverse task types and complexity levels (see [Task Description](#task-description) for more information):
+
+- **ruBABILongQA3** * — multi-fact reasoning over long contexts
+- **ruSciPassageCount** * — counting unique paragraphs in extended scientific texts
+- **LibrusecMHQA** * — multi-hop QA with information spread across multiple text parts
+- **LongContextMultiQ** * — multi-hop QA based on Wikidata and Wikipedia
+- **ru2WikiMultihopQA** * — multi-hop reasoning across multiple Wikipedia articles
+- **MatreshkaNames** * — identifying persons in dialogues based on discussed topics
+
+**Note:** The same asterisk (*) is used on the leaderboard to label LIBRA Mini datasets in the results tables.
+
+LIBRA Mini uses the same **Exact Match (EM)** metric and evaluation methodology as the full benchmark. Results for LIBRA Mini are reported in a dedicated section on the [leaderboard](https://huggingface.co/spaces/ai-forever/LIBRA-Leaderboard).
+
+We recommend **using LIBRA Mini as the primary evaluation suite for model comparisons**, while the full LIBRA benchmark remains available for comprehensive analysis.
+
+## Dataset Structure
 
+<p align="center">
+  <img src="libra_structure.svg" width="600" />
+</p>
+
+The datasets are divided into subsets based on context lengths. The table below shows the number of examples per context length for each dataset. Datasets included in **LIBRA Mini** are highlighted in bold. Note that not all datasets cover the full range of context lengths — some are designed for specific length ranges that best suit their task type.
+
+| **Task** | **4k** | **8k** | **16k** | **32k** | **64k** | **128k** | **256k** | **512k** | | **Total** |
+|:---|---:|---:|---:|---:|---:|---:|---:|---:|---|---:|
+| *— Group I —* | | | | | | | | | | |
+| passkey | 200 | 200 | 200 | 200 | 200 | 200 | 200 | 200 | | **1600** |
+| passkey_with_librusec | 200 | 200 | 200 | 200 | 200 | 200 | 200 | 200 | | **1600** |
+| *— Group II —* | | | | | | | | | | |
+| librusec_history | - | 32 | 32 | 32 | 32 | - | - | - | | **128** |
+| **matreshka_names** | 150 | 150 | 145 | 150 | 45 | - | - | - | | **640** |
+| matreshka_yes_no | 300 | 300 | 300 | 300 | 290 | 280 | - | - | | **1770** |
+| ru_quality | - | 18 | 184 | - | - | - | - | - | | **202** |
+| ru_sci_abstract_retrieval | 209 | 210 | 210 | 206 | 185 | 200 | 200 | - | | **1420** |
+| ru_sci_fi | - | - | - | 216 | 213 | - | - | - | | **429** |
+| ru_tpo | - | 900 | - | - | - | - | - | - | | **900** |
+| *— Group III —* | | | | | | | | | | |
+| **librusec_mhqa** | - | 384 | - | - | - | - | - | - | | **384** |
+| **long_context_multiq** | 158 | 121 | 83 | 109 | 41 | - | - | - | | **512** |
+| **ru_2wikimultihopqa** | - | 147 | 384 | 369 | - | - | - | - | | **900** |
+| ru_babilong_qa1 | 99 | 99 | 99 | 94 | 91 | 99 | 200 | 98 | | **879** |
+| ru_babilong_qa2 | 85 | 76 | 77 | 73 | 65 | 99 | 200 | 98 | | **773** |
+| **ru_babilong_qa3** | 60 | 68 | 69 | 65 | 65 | 100 | 198 | 98 | | **723** |
+| ru_babilong_qa4 | 78 | 85 | 83 | 87 | 75 | 99 | 200 | 98 | | **805** |
+| ru_babilong_qa5 | 99 | 99 | 98 | 96 | 96 | 99 | 200 | 98 | | **885** |
+| *— Group IV —* | | | | | | | | | | |
+| **ru_sci_passage_count** | 104 | 99 | 100 | 87 | 91 | 90 | 43 | 60 | | **674** |
+## Task Description
+The benchmark tasks are organized into four complexity groups, ranging from simple retrieval to complex reasoning. The grouping reflects both the cognitive difficulty of the task and the degree to which models must integrate information across the full context window. Group I serves as a basic sanity check, verifying that a model can process long inputs at all. Groups II and III progressively require deeper language understanding, multi-step reasoning, and the ability to locate and combine information from distant parts of the context. Group IV represents the most demanding tasks, requiring complex reasoning that goes beyond standard question answering formats. The total score on the leaderboard is computed across all four groups.
+### Group I: Simple Information Retrieval (sanity check)
+This group includes the most simple tasks which serve as a sanity check for models to work with such amount of tokens.
+- **Passkey**: Extract a relevant piece of code number from a long text fragment. Based on the original [PassKey test](https://github.com/CStanKonrad/long_llama/blob/main/examples/passkey.py) from the LongLLaMA's GitHub repo.
+- **PasskeyWithLibrusec**: Similar to Passkey but with added noise from Librusec texts.
 ### Group II: Question Answering and Multiple Choice
+This group consists of standard QA and multiple choice tasks adapted for the long-context setting.
 - **MatreshkaNames**: Identify the person in dialogues based on the discussed topic. We used [Matreshka](https://huggingface.co/datasets/zjkarina/matreshka) dataset and [Russian Names](https://www.kaggle.com/datasets/rai220/russian-cyrillic-names-and-sex/data) dataset to create this and the next task.
 - **MatreshkaYesNo**: Indicate whether a specific topic was mentioned in the dialog.
-- **LibrusecHistory**: Answer questions based on historical texts. Ideologically similiar to the [PassageRetrieval dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/passage_retrieval_en) from LongBench.
-- **ruTREC**: Few-shot in-context learning for topic classification. Created by translating the [TREC dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/trec_e) from LongBench.
+- **LibrusecHistory**: Answer questions based on historical texts. Ideologically similar to the [PassageRetrieval dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/passage_retrieval_en) from LongBench.
 - **ruSciFi**: Answer true/false based on context and general world knowledge. Translation of [SciFi dataset](https://huggingface.co/datasets/L4NLP/LEval/viewer/sci_f) from L-Eval which originally was based on [SF-Gram](https://github.com/nschaetti/SFGram-dataset).
 - **ruSciAbstractRetrieval**: Retrieve relevant paragraphs from scientific abstracts.
 - **ruTPO**: Multiple-choice questions similar to TOEFL exams. Translation of the [TPO dataset](https://huggingface.co/datasets/L4NLP/LEval/viewer/tpo) from L-Eval.
 - **ruQuALITY**: Multiple-choice QA tasks based on detailed texts. Created by translating the [QuALITY dataset](https://huggingface.co/datasets/L4NLP/LEval/viewer/quality) from L-Eval.
-
 ### Group III: Multi-hop Question Answering
-- **ruBABILongQA**: 5 long-context reasoning tasks for QA using facts hidden among irrelevant information.
+This group includes long-context multi-hop QA problems where the answer requires combining multiple pieces of information distributed across the context.
+- **ruBABILongQA (1-5)**: 5 long-context reasoning tasks for QA using facts hidden among irrelevant information.
 - **LongContextMultiQ**: Multi-hop QA based on Wikidata and Wikipedia.
 - **LibrusecMHQA**: Multi-hop QA requiring information distributed across several text parts.
 - **ru2WikiMultihopQA**: Translation of the [2WikiMultihopQA dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/2wikimqa_e) from LongBench.
-
 ### Group IV: Complex Reasoning and Mathematical Problems
+This group includes the most complex long-context tasks, which span beyond multiple-choice and multi-hop QA. At this point Group IV comprises only one task and we invite the community to contribute to it.
 - **ruSciPassageCount**: Count unique paragraphs in a long text. Uses the basic idea of the original [PassageCount dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/passage_count) from LongBench.
-- **ruQasper**: Question Answering over academic research papers. Created by translating the [Qasper dataset](https://huggingface.co/datasets/THUDM/LongBench/viewer/qasper_e) from LongBench.
-- **ruGSM100**: Solve math problems using Chain-of-Thought reasoning. Created by translating the [GSM100](https://huggingface.co/datasets/L4NLP/LEval/viewer/gsm100) dataset from L-Eval.
-
-## Dataset Structure
-
-The datasets are divided into subsets based on context lengths: 4k, 8k, 16k, 32k, 64k, and 128k tokens. Each subset contains a different number of samples depending on the task complexity.
-
-## Add your model
-
-For placing your model to leaderboard you need to:
-
-1. Score the model on our repository.
-2. Add the result in the format **"\<model_name\>.json"** in the "results" folder.
-3. Create a Pull Request.
-
-## *GPT-4o
-
-Because of limited resources, we assessed GPT-4o on just 10% of each dataset in our benchmark, including each context length. Consequently, the results might not be exact.
-
+## Metrics
+We use **Exact Match (EM)** as a primary metric for all tasks. **EM** is used to evaluate the accuracy of the model's responses by comparing the predicted answers to the ground truth.
+## Changes from the Original Version
+This version of LIBRA includes both automatic and manual improvements over the original release. All datasets underwent automatic quality filtering to ensure consistency and reliability of annotations. In addition, several datasets were manually revised and extended with the help of human annotators: **LibrusecMHQA**, **LongContextMultiQ**, **MatreshkaNames**, **MatreshkaYesNo**, **ru2WikiMultihopQA**, **ruSciFi**, and **ruTPO** received targeted corrections and additional examples. The datasets **ruGSM100** and **ruQasper** were removed from the benchmark as they did not meet the updated quality criteria.
+The maximum supported context length has been extended from 128k to 512k tokens. The following datasets now include examples at longer context lengths not present in the original version: **ruBABILongQA (1–5)**, **ruSciAbstractRetrieval**, **ruSciPassageCount**, **LongContextMultiQ**, **MatreshkaYesNo**, **Passkey**, and **PasskeyWithLibrusec**.
+## Evaluation
+Starting from this version, LIBRA supports evaluation via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) — a widely adopted framework for standardized LLM evaluation. Both full **LIBRA** and compact **LIBRA Mini** evaluations are supported.
+To get started:
+```bash
+pip install lm-eval[vllm]
+```
+Run evaluation on the full LIBRA benchmark:
+```bash
+lm_eval --model vllm \
+        --model_args pretrained=Qwen/Qwen3-30B-A3B,max_model_len=262144 \
+        --tasks libra \
+        --apply_chat_template \
+        --device cuda:0
+```
+Run evaluation on LIBRA Mini only:
+```bash
+lm_eval --model vllm \
+        --model_args pretrained=Qwen/Qwen3-30B-A3B,max_model_len=262144 \
+        --tasks libra_mini \
+        --apply_chat_template \
+        --device cuda:0
+```
+For the full list of configuration options and instructions on adding new models, please refer to the [lm-evaluation-harness documentation](https://github.com/EleutherAI/lm-evaluation-harness).
 ## Citation
-
-_TODO_
-
 ```
-@article{LIBRA2024,
-title={Long Input Benchmark for Russian Analysis},
-author={Anonymous},
-journal={ACL},
-year={2024}
+@misc{churin2024longinputbenchmarkrussian,
+      title={Long Input Benchmark for Russian Analysis}, 
+      author={Igor Churin and Murat Apishev and Maria Tikhonova and Denis Shevelev and Aydar Bulatov and Yuri Kuratov and Sergei Averkiev and Alena Fenogenova},
+      year={2024},
+      eprint={2408.02439},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2408.02439}, 
 }
-```
-
-## License
-
-The datasets are published under the MIT license.
-
-## Acknowledgments
-
-For more details and code, please visit our [GitHub repository](https://github.com/ai-forever/LIBRA/).
+```

results/LiquidAI_LFM2.5-1.2B-Instruct.json

@@ -1,5 +1,5 @@
 {
-  "total_score": 0.2860481481481481,
+  "total_score": 0.2880111111111111,
   "passkey": {
     "dataset_total_score": 0.6666666666666666,
     "4k": 1.0,
@@ -58,13 +58,12 @@
     "32k": 0.287
   },
   "long_context_multiq": {
-    "dataset_total_score": 0.17666666666666667,
+    "dataset_total_score": 0.21200000000000002,
     "4k": 0.418,
     "8k": 0.364,
     "16k": 0.241,
     "32k": 0.037,
-    "64k": 0.0,
-    "128k": 0.0
+    "64k": 0.0
   },
   "ru_sci_abstract_retrieval": {
     "dataset_total_score": 0.2318333333333333,