diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -27,7 +27,6 @@ language: - nl - nb - 'no' -- nn - oc - pl - pt @@ -40,6 +39,12 @@ language: - uk - ast - an +- ar +- ja +- hi +- ko +- zh +- is base_model: - BSC-LT/salamandra-7b --- @@ -48,17 +53,27 @@ base_model: ![](./images/salamandra_header.png) -> [!IMPORTANT] -> **DISCLAIMER:** We have released a new, extended version of this model. Check it out: https://huggingface.co/BSC-LT/salamandraTA-7b-instruct/tree/v2.0 +## Releases + +| Version | Ref | Description | +|---------|-----|-------------| +| [v2.0](https://huggingface.co/BSC-LT/salamandraTA-7b-instruct) | `main` | Current release | +| [v1.0](https://huggingface.co/BSC-LT/salamandraTA-7b-instruct/tree/v1.0) | `v1.0` | Initial release | + +> To load a specific version use the `revision` parameter: +> ```python +> model = AutoModelForCausalLM.from_pretrained( +> "BSC-LT/salamandraTA-7b-instruct", +> revision="v1.0" # omit revision for current release +> ) +> ``` # SalamandraTA Model Card SalamandraTA-7b-instruct is a translation LLM that has been instruction-tuned from SalamandraTA-7b-base. -The base model results from continually pre-training [Salamandra-7b](https://huggingface.co/BSC-LT/salamandra-7b) on parallel data and has not been published, -but is reserved for internal use. -SalamandraTA-7b-instruct is proficient in 35 European languages (plus 3 varieties) and supports translation-related tasks, namely: sentence-level-translation, -paragraph-level-translation, document-level-translation, automatic post-editing, grammar checking, machine translation evaluation, alternative translations, -named-entity-recognition and context-aware translation. +The base model results from continually pre-training [Salamandra-7b](https://huggingface.co/BSC-LT/salamandra-7b) on monolingual and parallel data and has not been published, but is reserved for internal use. +SalamandraTA-7b-instruct (v2) is proficient in **40 languages (+ 3 varieties)** and is mainly trained to perform general translation tasks at the sentence, paragraph, and document levels. +The language coverage for this version of SalamandraTA-7B-instruct has been expanded to include five additional non-European languages: Arabic, Japanese, Hindi, Korean, and Simplified Chinese, which complement the European languages supported in [our previous model's version](https://huggingface.co/BSC-LT/salamandraTA-7b-instruct/tree/v1.0) and broaden the model's multilingual reach. Additionally, translation performance in all pre-existing language pairs has been improved with respect to the previous model's version (see [Evaluation](#evaluation)). > [!WARNING] > **DISCLAIMER:** This version of Salamandra is tailored exclusively for translation tasks. It lacks chat capabilities and has not been trained with any chat instructions. @@ -70,7 +85,14 @@ named-entity-recognition and context-aware translation. ### Description -SalamandraTA-7b-base is a continual pre-training of [Salamandra-7b](https://huggingface.co/BSC-LT/salamandra-7b) using parallel data, resulting in a total of 424B tokens processed during training. +SalamandraTA-7b-base is a continual pre-training of [Salamandra-7b](https://huggingface.co/BSC-LT/salamandra-7b). + +The model was trained on a mixture of monolingual and parallel corpora. The continual pre-training was conducted in two stages: + +- Stage 1: Monolingual data (≈69B tokens) +- Stage 2: A mixture of monolingual and parallel data, including instruction prompts (≈262B tokens) + +In total, approximately **331B tokens** were processed during continual pre-training. ### Architecture @@ -99,7 +121,7 @@ SalamandraTA-7b-base is a continual pre-training of [Salamandra-7b](https://hugg ### Direct Use -The model is intended for both research and commercial use in any of the languages included in the training data for general machine translation tasks. +The model is intended for both research and commercial use in any of the languages included in the training data for **general machine translation tasks**. ### Out-of-scope Use @@ -114,7 +136,7 @@ Irresponsible usage in production environments without proper risk assessment an ### Training Framework -SalamandraTA-7b-base was continually pre-trained using NVIDIA’s [NeMo Framework](https://docs.nvidia.com/nemo-framework/index.html), +SalamandraTA-7b-base was continually pre-trained using NVIDIA's [NeMo Framework](https://docs.nvidia.com/nemo-framework/index.html), which leverages PyTorch Lightning for efficient model training in highly distributed settings. SalamandraTA-7b-instruct was produced with [FastChat](https://github.com/lm-sys/FastChat). @@ -136,11 +158,9 @@ The accelerated partition is composed of 1,120 nodes with the following specific ## How to use -You can translate between the following **35 languages** (and 3 varieties): +You can translate between the following **40 languages** (and 3 varieties): -Aragonese, Asturian, Basque, Bulgarian, Catalan (and Catalan-Valencian variety), Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, Galician, German, Greek, Hungarian, -Irish, Italian, Latvian, Lithuanian, Maltese, Norwegian (Bokmål and Nynorsk varieties), Occitan (and Aranese variety), Polish, Portuguese, Romanian, Russian, Serbian, Slovak, -Slovenian, Spanish, Swedish, Ukrainian, Welsh. +Arabic, Aragonese, Asturian, Basque, Bulgarian, Catalan (and Catalan-Valencian variety), Chinese (simplified), Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, Galician, German, Greek, Hindi, Hungarian, Irish, Italian, Japanese, Korean, Latvian, Lithuanian, Maltese, Norwegian (Bokmål and Nynorsk varieties), Occitan (and Aranese variety), Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Ukrainian, Welsh. The instruction-following model uses the commonly adopted ChatML template: @@ -226,196 +246,227 @@ text = f"Translate the following text from {source} into {target}.\n{source}: {s -### Post-editing - -For post-editing tasks, you can use the following prompt template: - -``` -Please fix any mistakes in the following {source}-{target} machine translation or keep it unedited if it's correct. -Source: {source_sentence} -MT: {machine_translation} -Corrected:" -``` - -
-Show an example - -```python -source = 'Catalan' -target = 'English' -source_sentence = 'Rafael Nadal i Maria Magdalena van inspirar a una generació sencera.' -machine_translation = 'Rafael Christmas and Maria the Muffin inspired an entire generation each in their own way.' - -text = f"Please fix any mistakes in the following {source}-{target} machine translation or keep it unedited if it's correct.\nSource: {source_sentence} \nMT: {machine_translation} \nCorrected:" - -# Rafael Nadal and Maria Magdalena inspired an entire generation. -``` - -
- -### Document-level translation - -For document-level translation tasks, you can use the following prompt template: - -``` -Please translate this text from {source} into {target}. -{source}: {1st paragraph of the document} -{2nd paragraph of the document} -{Nth paragraph of the document} -{target}: -``` - -
-Show an example - -```python -source = 'English' -target = 'Asturian' - -text = """Please translate this text from {} into {}.\n{}: President Donald Trump, who campaigned on promises to crack down on illegal immigration, has raised alarms in the U.S. dairy industry with his threat to impose 25% tariffs on Mexico and Canada by February 2025. This move is part of a broader strategy to declare a national emergency at the southern border to halt illegal migration completely. -However, the implications for the agriculture sector, particularly dairy, are significant. Approximately half of the U.S. dairy industry's workforce consists of immigrant labor, many of whom are undocumented. The National Milk Producers Federation estimates that removing immigrant workers could decimate the dairy herd by 2.1 million cows and slash milk production by nearly 50 billion pounds, leading to a dramatic 90.4% increase in milk prices. -The complex perspectives of Americans on undocumented workers were highlighted in a Pew Research Center study. While 64% of U.S. adults support legal pathways for undocumented immigrants, 35% oppose it—a gap that has been narrowing recently. Factors influencing public opinion include the belief that immigrants should have jobs and pass security checks, contrasted by concerns about lawbreakers being rewarded, fairness for legal migrants, and resource allocation. -According to Zach Rutledge, an agricultural economist at Michigan State University, as nations grow wealthier, their labor forces transition away from agriculture toward sectors like services and manufacturing. This shift has led to the U.S. relying heavily on immigrant labor for agricultural work. Domestic workers, even with employment taxes, may cost $15 to $25 an hour, while H-2A visa program workers might cost $25 to $30 an hour, accounting for additional housing expenses. -The National Milk Producers Federation has been vocal in advocating for changes to the H-2A visa program, which outside of its current seasonal limitations, does not support the dairy industry's year-round labor needs. Executive vice-president Jaime Castaneda reiterated the need for legislative clarity to address the undocumented workforce issues in dairy farming. -The Farm Workforce Modernization Act of 2023, which could grant legal status to certain undocumented farmworkers, has been stalled in Congress, despite acknowledgment of the sector's importance to feeding America. The need for coordinated legislative efforts to ensure both border security and labor market stability is imperative moving forward. -{}:""".format(source, target, source, target) -``` +--- -
+## Data -### Named-entity recognition +### Pretraining Data -For named-entity recognition tasks, you can use the following prompt template: +The model was trained through two stages of continual pre-training using a combination of monolingual and parallel corpora covering all the 40 supported languages and 3 varieties. -``` -Analyse the following tokenized text and mark the tokens containing named entities. -Use the following annotation guidelines with these tags for named entities: -- ORG (Refers to named groups or organizations) -- PER (Refers to individual people or named groups of people) -- LOC (Refers to physical places or natural landmarks) -- MISC (Refers to entities that don't fit into standard categories). -Prepend B- to the first token of a given entity and I- to the remaining ones if they exist. -If a token is not a named entity, label it as O. -Input: {list of words in a sentence} -Marked: -``` +In total, approximately 331B tokens were processed during continual pre-training. -
-Show an example - -```python -text = """Analyse the following tokenized text and mark the tokens containing named entities. -Use the following annotation guidelines with these tags for named entities: -- ORG (Refers to named groups or organizations) -- PER (Refers to individual people or named groups of people) -- LOC (Refers to physical places or natural landmarks) -- MISC (Refers to entities that don't fit into standard categories). -Prepend B- to the first token of a given entity and I- to the remaining ones if they exist. -If a token is not a named entity, label it as O. -Input: ['La', 'defensa', 'del', 'antiguo', 'responsable', 'de', 'la', 'RFEF', 'confirma', 'que', 'interpondrá', 'un', 'recurso.'] -Marked: """ - -# [('La', 'O'), ('defensa', 'O'), ('del', 'O'), ('antiguo', 'O'), ('responsable', 'O'), ('de', 'O'), ('la', 'O'), ('RFEF', 'B-ORG'), ('confirma', 'O'), ('que', 'O'), ('interpondrá', 'O'), ('un', 'O'), ('recurso.', 'O')] -``` -
-### Grammar checker +#### Continual Pre-training Stage 1 (CPT1) -For fixing any mistakes in grammar, you can use the following prompt template: +The first stage of continual pre-training used only monolingual data. -``` -Please fix any mistakes in the following {source} sentence or keep it unedited if it's correct. -Sentence: {sentence} -Corrected: -``` +The corpus is primarily composed of FineWeb2 collections and Wikipedia dumps, covering European languages as well as several additional languages. Total tokens processed in CPT1: ≈69B tokens. -
-Show an example +Primary data sources include: -```python -source = 'Catalan' -sentence = 'Entonses, el meu jefe m’ha dit que he de treballar els fins de setmana.' +- [FineWeb2 multilingual web corpora](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) +- [Wikipedia dumps](https://dumps.wikimedia.org) +- Additional internally curated monolingual corpora for low-resource languages such as Aranese. -text = f"Please fix any mistakes in the following {source} sentence or keep it unedited if it's correct.\nSentence: {sentence} \nCorrected:" -# Llavors, el meu cap m'ha dit que he de treballar els caps de setmana. -``` -
+#### Continual Pre-training Stage 2 (CPT2) -## Data +The second stage of continual pre-training used a mixture of monolingual and parallel data. The total number of tokens processed during CPT2 is 262B tokens. -### Pretraining Data +The training data consists of: -The pretraining corpus consists of 424 billion tokens of Catalan-centric, Spanish-centric, and English-centric parallel data, -including all of the official European languages plus Catalan, Basque, Galician, Asturian, Aragonese and Aranese. -It amounts to 6,574,251,526 parallel sentence pairs. +- Monolingual corpora (FineWeb2 and Wikipedia) +- Parallel corpora formatted with instruction-style prompts +The parallel data contains hundreds of language pairs centered around Catalan, Spanish and English, as well as several direct translation pairs between other languages. This highly multilingual corpus is predominantly composed of data sourced from [OPUS](https://opus.nlpl.eu/), with additional data taken from the [NTEU Project](https://nteu.eu/), [Aina Project](https://projecteaina.cat/), and other sources (see: [Data Sources](#pre-data-sources) and [References](#pre-references)). Where little parallel Catalan <-> xx data could be found, synthetic Catalan data was generated from the Spanish side of the collected Spanish <-> xx corpora using -[Projecte Aina’s Spanish-Catalan model](https://huggingface.co/projecte-aina/aina-translator-es-ca). The final distribution of languages was as below: +[Projecte Aina's Spanish-Catalan model](https://huggingface.co/projecte-aina/aina-translator-es-ca). + + +Click the expand button below to see the full list of corpora included in the monolingual training data. + +
+Monolingual Data Sources (CPT1 and CPT2) + +| Language | FineWeb2 Web Corpus | Wikipedia (Wikimedia Dumps) | Other Sources | +|----------|---------------------|------------------------------|---------------| +| Arabic | ✓ | | | +| Aragonese | ✓ | | | +| Asturian | ✓ | ✓ | | +| Bulgarian | ✓ | | | +| Catalan | ✓ | | | +| Czech | ✓ | | | +| Welsh | ✓ | ✓ | | +| Danish | ✓ | | | +| German | ✓ | | | +| Greek | ✓ | | | +| Spanish | ✓ | | | +| Estonian | ✓ | | | +| Basque | ✓ | | | +| Finnish | ✓ | | | +| French | ✓ | | | +| Irish | ✓ | ✓ | | +| Galician | ✓ | | | +| Hindi | ✓ | | | +| Croatian | ✓ | | | +| Hungarian | ✓ | | | +| Icelandic | ✓ | | | +| Italian | ✓ | | | +| Japanese | ✓ | | | +| Korean | ✓ | | | +| Lithuanian | ✓ | | | +| Latvian | ✓ | | | +| Maltese | ✓ | ✓ | | +| Norwegian Bokmål | ✓ | | | +| Dutch | ✓ | | | +| Norwegian Nynorsk | ✓ | | | +| Occitan | ✓ | ✓ | | +| Polish | ✓ | | | +| Portuguese | ✓ | | | +| Romanian | ✓ | | | +| Russian | ✓ | | | +| Slovak | ✓ | | | +| Slovenian | ✓ | | | +| Serbian | ✓ | | | +| Swedish | ✓ | | | +| Ukrainian | ✓ | | | +| Chinese | ✓ | | | +| English | ✓ | ✓ | | +| Aranese | | | [Catalan-Aranese parallel corpus](https://huggingface.co/datasets/BSC-LT/Catalan-Aranese_Parallel_Corpus) | -![](./images/treemap.png) +
-Click the expand button below to see the full list of corpora included in the training data. -
-Data Sources - -| Dataset | Ca-xx Languages | Es-xx Langugages | En-xx Languages | -|-----------------------------------------------|----------------------------------------------------------------|-----------------------------------------------|----------------------------------------------------------------| -|[AINA](https://huggingface.co/collections/projecte-aina/mt-datasets-655f33d9f4be8787c8e7486b) | en | | | -|ARANESE-SYNTH-CORPUS-BSC | arn | | | -|BOUA-SYNTH-BSC | | val | | -|[BOUMH](https://github.com/transducens/PILAR/tree/main/valencian/BOUMH) | | val | | -|[BOUA-PILAR](https://github.com/transducens/PILAR/tree/main/valencian/BOUA) | | val | | -|[CCMatrix](https://opus.nlpl.eu/CCMatrix/corpus/version/CCMatrix) |eu | | ga | -|[DGT](https://opus.nlpl.eu/DGT/corpus/version/DGT) | |bg,cs,da,de,el ,et,fi,fr,ga,hr,hu,lt,lv,mt,nl,pl,pt,ro,sk,sl,sv | da,et,ga,hr,hu,lt,lv,mt,sh,sl| -|DOGV-SYNTH-BSC | | val | | -|[DOGV-PILAR](https://github.com/transducens/PILAR/tree/main/valencian/DOGV-html) | | val | | -|[ELRC-EMEA](https://opus.nlpl.eu/ELRC-EMEA/corpus/version/ELRC-EMEA) | |bg,cs,da,hu,lt,lv,mt,pl,ro,sk,sl | et,hr,lv,ro,sk,sl | -|[EMEA](https://opus.nlpl.eu/EMEA/corpus/version/EMEA) | |bg,cs,da,el,fi,hu,lt,mt,nl,pl,ro,sk,sl,sv | et,mt | -|[EUBookshop](https://opus.nlpl.eu/EUbookshop/corpus/version/EUbookshop) |lt,pl,pt |cs,da,de,el,fi,fr,ga,it,lv,mt,nl,pl,pt,ro,sk,sl,sv |cy,ga| -|[Europarl](https://opus.nlpl.eu/Europarl/corpus/version/Europarl) | |bg,cs,da,el,en,fi,fr,hu,lt,lv,nl,pl,pt ,ro,sk,sl,sv | | -|[Europat](https://opus.nlpl.eu/EuroPat/corpus/version/EuroPat) | |en,hr | no | -|[GAITU Corpus](https://gaitu.eus/) | | | eu| -|[KDE4](https://opus.nlpl.eu/KDE4/corpus/version/KDE4) |bg,cs,da,de,el ,et,eu,fi,fr,ga,gl,hr,it,lt,lv,nl,pl,pt,ro,sk,sl,sv |bg,ga,hr |cy,ga,nn,oc | -|[GlobalVoices](https://opus.nlpl.eu/GlobalVoices/corpus/version/GlobalVoices) | bg,de,fr,it,nl,pl,pt |bg,de,fr,pt | | -|[GNOME](https://opus.nlpl.eu/GNOME/corpus/version/GNOME) |eu,fr,ga,gl,pt |ga |cy,ga,nn| -|[JRC-Arquis](https://opus.nlpl.eu/JRC-Acquis/corpus/version/JRC-Acquis) | |cs,da,et,fr,lt,lv,mt,nl,pl ,ro,sv| et | -|LES-CORTS-VALENCIANES-SYNTH-BSC | | val | | -|[MaCoCu](https://opus.nlpl.eu/MaCoCu/corpus/version/MaCoCu) | en | | hr,mt,uk | -|[MultiCCAligned](https://opus.nlpl.eu/JRC-Acquis/corpus/version/JRC-Acquis) |bg,cs,de,el,et,fi,fr,hr,hu,it,lt,lv,nl,pl,ro,sk,sv |bg,fi,fr,hr,it,lv,nl,pt |bg,cy,da,et,fi,hr,hu,lt,lv,no,sl,sr,uk| -|[MultiHPLT](https://opus.nlpl.eu/MultiHPLT/corpus/version/MultiHPLT) |en, et,fi,ga,hr,mt | |fi,ga,gl,hr,mt,nn,sr | -|[MultiParaCrawl](https://opus.nlpl.eu/MultiParaCrawl/corpus/version/MultiParaCrawl) |bg,da |de,en,fr,ga,hr,hu,it,mt,pt |bg,cs,da,de,el,et,fi,fr,ga,hr,hu,lt,lv,mt,nn,pl,ro,sk,sl,uk| -|[MultiUN](https://opus.nlpl.eu/MultiUN/corpus/version/MultiUN) | |fr | | -|[News-Commentary](https://opus.nlpl.eu/News-Commentary/corpus/version/News-Commentary) | |fr | | -|[NLLB](https://opus.nlpl.eu/NLLB/corpus/version/NLLB) |bg,da,el,en,et,fi,fr,gl,hu,it ,lt,lv,pt,ro,sk,sl |bg,cs,da,de,el ,et,fi,fr,hu,it,lt,lv,nl,pl,pt ,ro,sk,sl,sv| bg,cs,cy,da,de,el,et,fi,fr,ga,hr,hu,it,lt,lv,mt,nl,no,oc,pl,pt,ro,ru,sk,sl,sr,sv,uk| -|[NÓS Authentic Corpus](https://zenodo.org/records/7675110) | | | gl | -|[NÓS Synthetic Corpus](https://zenodo.org/records/7685180) | | | gl | -|[NTEU](https://www.elrc-share.eu/repository/search/?q=NTEU) | |bg,cs,da,de,el,en,et,fi,fr,ga,hr,hu,it,lt,lv,mt,nl,pl,pt,ro,sk,sl,sv | da,et,ga,hr,lt,lv,mt,ro,sk,sl,sv | -|[OpenSubtitles](https://opus.nlpl.eu/OpenSubtitles/corpus/version/OpenSubtitles) |bg,cs,da,de,el ,et,eu,fi,gl,hr,hu,lt,lv,nl,pl,pt,ro,sk,sl,sv |da,de,fi,fr,hr,hu,it,lv,nl | bg,cs,de,el,et,hr,fi,fr,hr,hu,no,sl,sr| -|[OPUS-100](https://opus.nlpl.eu/opus-100.php) | en | | gl | -|[StanfordNLP-NMT](https://opus.nlpl.eu/StanfordNLP-NMT/corpus/version/StanfordNLP-NMT) | | |cs | -|[Tatoeba](https://opus.nlpl.eu/Tatoeba/corpus/version/Tatoeba) |de,pt |pt | | -|[TildeModel](https://opus.nlpl.eu/TildeMODEL/corpus/version/TildeMODEL) | |bg | et,hr,lt,lv,mt | -|[UNPC](https://opus.nlpl.eu/UNPC/corpus/version/UNPC) | |en,fr | ru | -|[PILAR-VALENCIAN-AUTH](https://github.com/transducens/PILAR/tree/main/valencian/Generalitat) | | val | | -|[PILAR-VALENCIAN-SYNTH](https://github.com/transducens/PILAR/tree/main/valencian/Generalitat) | | val | | -|[WikiMatrix](https://opus.nlpl.eu/WikiMatrix/corpus/version/WikiMatrix) |bg,cs,da,de,el ,et,eu,fi,fr,gl,hr,hu,it,lt,nl,pl,pt,ro,sk,sl,sv |bg,en,fr,hr,it,pt | oc,sh | -|[Wikimedia](https://opus.nlpl.eu/wikimedia/corpus/version/wikimedia) | | |cy,nn | -|[XLENT](https://opus.nlpl.eu/XLEnt/corpus/version/XLEnt) |eu,ga,gl |ga |cy,et,ga,gl,hr,oc,sh| +Click the expand button below to see the full list of corpora included in the parallel training data. + +
+Data Sources + Datasets with "-BSC" in their names (e.g., BOUA-SYNTH-BSC, DOGV-SYNTH-BSC) are synthetic datasets obtained by machine translating pre-existing monolingual corpora with our own seq-to-seq models. These datasets were generated internally for model training and are not published. -To consult the data summary document with the respective licences, please send an e-mail to ipr@bsc.es. - +#### Catalan pairs (Ca-xx) + +| Dataset | Languages | +|---|---| +| [AINA](https://huggingface.co/collections/projecte-aina/mt-datasets-655f33d9f4be8787c8e7486b) | en | +| ARANESE-SYNTH-CORPUS-BSC | arn | +| [CCMatrix](https://opus.nlpl.eu/datasets/CCMatrix) | eu | +| [EUBookshop](https://opus.nlpl.eu/datasets/EUbookshop) | lt, pl, pt | +| [KDE4](https://opus.nlpl.eu/datasets/KDE4) | bg, cs, da, de, el, et, eu, fi, fr, ga, gl, hr, it, ja, lt, lv, nl, pl, pt, ro, sk, sl, sv | +| [GlobalVoices](https://opus.nlpl.eu/datasets/GlobalVoices) | bg, de, fr, it, nl, pl, pt | +| [GNOME](https://opus.nlpl.eu/datasets/GNOME) | eu, fr, ga, gl, pt | +| [MaCoCu](https://opus.nlpl.eu/datasets/MaCoCu) | en | +| [MultiCCAligned](https://opus.nlpl.eu/datasets/MultiCCAligned) | bg, cs, de, el, et, fi, fr, hr, hu, it, ja, lt, lv, nl, pl, ro, sk, sv | +| [MultiHPLT](https://opus.nlpl.eu/datasets/MultiHPLT) | en, et, fi, ga, hr, mt | +| [MultiParaCrawl](https://opus.nlpl.eu/datasets/MultiParaCrawl) | bg, da | +| [NLLB](https://opus.nlpl.eu/datasets/NLLB) | bg, da, el, en, et, fi, fr, gl, hu, it, lt, lv, pt, ro, sk, sl | +| [OpenSubtitles](https://opus.nlpl.eu/datasets/OpenSubtitles) | bg, cs, da, de, el, et, eu, fi, gl, hr, hu, ja, lt, lv, nl, pl, pt, ro, sk, sl, sv | +| [OPUS-100](https://opus.nlpl.eu/opus-100.php) | en | +| [Tatoeba](https://opus.nlpl.eu/datasets/Tatoeba) | de, pt | +| [WikiMatrix](https://opus.nlpl.eu/datasets/WikiMatrix) | bg, cs, da, de, el, et, eu, fi, fr, gl, hr, hu, it, lt, nl, pl, pt, ro, sk, sl, sv | +| [XLENT](https://opus.nlpl.eu/datasets/XLEnt) | eu, ga, gl, ja | +| [Catalan-Aranese Parallel Corpus](https://huggingface.co/datasets/BSC-LT/Catalan-Aranese_Parallel_Corpus) | arn | + +#### Spanish pairs (Es-xx) + +| Dataset | Languages | +|---|---| +| BOUA-SYNTH-BSC | val | +| [BOUMH](https://github.com/transducens/PILAR/tree/main/valencian/BOUMH) | val | +| [BOUA-PILAR](https://github.com/transducens/PILAR/tree/main/valencian/BOUA) | val | +| [DGT](https://opus.nlpl.eu/datasets/DGT) | bg, cs, da, de, el, et, fi, fr, ga, hr, hu, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv | +| DOGV-SYNTH-BSC | val | +| [DOGV-PILAR](https://github.com/transducens/PILAR/tree/main/valencian/DOGV-html) | val | +| [EMEA](https://live.european-language-grid.eu/catalogue/corpus/3444) | bg, cs, da, el, fi, hu, lt, lv, mt, nl, pl, ro, sk, sl, sv | +| [EUBookshop](https://opus.nlpl.eu/datasets/EUbookshop) | cs, da, de, el, fi, fr, ga, it, lv, mt, nl, pl, pt, ro, sk, sl, sv | +| [Europarl](https://opus.nlpl.eu/datasets/Europarl) | bg, cs, da, el, en, fi, fr, hu, lt, lv, nl, pl, pt, ro, sk, sl, sv | +| [Europat](https://opus.nlpl.eu/datasets/EuroPat) | en, hr | +| [GlobalVoices](https://opus.nlpl.eu/datasets/GlobalVoices) | bg, de, fr, pt | +| [JRC-Acquis](https://opus.nlpl.eu/datasets/JRC-Acquis) | cs, da, et, fr, lt, lv, mt, nl, pl, ro, sv | +| [KDE4](https://opus.nlpl.eu/datasets/KDE4) | bg, ga, hr | +| LES-CORTS-VALENCIANES-SYNTH-BSC | val | +| [MultiCCAligned](https://opus.nlpl.eu/datasets/MultiCCAligned) | bg, fi, fr, hi, hr, it, ja, lv, nl, pt, zh | +| [MultiParaCrawl](https://opus.nlpl.eu/datasets/MultiParaCrawl) | de, en, fr, ga, hr, hu, it, mt, pt, zh | +| [MultiUN](https://opus.nlpl.eu/datasets/MultiUN) | fr, zh | +| [News-Commentary](https://opus.nlpl.eu/datasets/News-Commentary) | fr, zh | +| [NLLB](https://opus.nlpl.eu/datasets/NLLB) | ar, bg, cs, da, de, el, et, fi, fr, hi, hu, it, ja, lt, lv, nl, pl, pt, ro, sk, sl, sv, zh | +| [NTEU](https://live.european-language-grid.eu/) | bg, cs, da, de, el, en, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv | +| [OpenSubtitles](https://opus.nlpl.eu/datasets/OpenSubtitles) | ar, da, de, fi, fr, hi, hr, hu, it, ja, lv, nl, zh | +| [PILAR-VALENCIAN-AUTH](https://github.com/transducens/PILAR/tree/main/valencian/Generalitat) | val | +| [PILAR-VALENCIAN-SYNTH](https://github.com/transducens/PILAR/tree/main/valencian/Generalitat) | val | +| [Tatoeba](https://opus.nlpl.eu/datasets/Tatoeba) | pt, zh | +| [TildeModel](https://opus.nlpl.eu/datasets/TildeMODEL) | bg | +| [UNPC](https://opus.nlpl.eu/datasets/UNPC) | ar, en, fr, zh | +| [WikiMatrix](https://opus.nlpl.eu/datasets/WikiMatrix) | bg, en, fr, hr, it, pt, zh | + +#### English pairs (En-xx) + +| Dataset | Languages | +|---|---| +| [CCMatrix](https://opus.nlpl.eu/datasets/CCMatrix) | ga | +| [DGT](https://opus.nlpl.eu/datasets/DGT) | cs, da, et, ga, hr, hu, lt, lv, mt, sh, sl | +| [EMEA](https://live.european-language-grid.eu/catalogue/corpus/3444) | et, hr, lv, mt, ro, sk, sl | +| [EUBookshop](https://opus.nlpl.eu/datasets/EUbookshop) | ar, cy, ga, is, ja, ru, sh, uk, zh | +| [Europarl](https://opus.nlpl.eu/datasets/Europarl) | cs, et | +| [Europat](https://opus.nlpl.eu/datasets/EuroPat) | no | +| [GNOME](https://opus.nlpl.eu/datasets/GNOME) | cy, ga, nn | +| [HPLT](https://opus.nlpl.eu) | hi | +| [KDE4](https://opus.nlpl.eu/datasets/KDE4) | ar, cy, ga, is, ja, ko, nn, oc, ru, sh, uk, zh | +| [GlobalVoices](https://opus.nlpl.eu/datasets/GlobalVoices) | ar, ja, ko, ru, sh | +| [MaCoCu](https://opus.nlpl.eu/datasets/MaCoCu) | hr, mt, uk | +| [MultiCCAligned](https://opus.nlpl.eu/datasets/MultiCCAligned) | ar, bg, cy, da, et, fi, hr, hu, is, ja, ko, lt, lv, no, ru, sh, sl, sr, uk, zh | +| [MultiHPLT](https://opus.nlpl.eu/datasets/MultiHPLT) | ar, fi, ga, gl, hr, is, ja, ko, mt, nn, sh, sr, uk | +| [MultiParaCrawl](https://opus.nlpl.eu/datasets/MultiParaCrawl) | bg, cs, da, de, el, et, fi, fr, ga, hr, hu, is, lt, lv, mt, nn, pl, ro, ru, sk, sl, uk | +| [MultiUN](https://opus.nlpl.eu/datasets/MultiUN) | ar, ru, zh | +| [News-Commentary](https://opus.nlpl.eu/datasets/News-Commentary) | ar, cs, ja, ru, zh | +| [NLLB](https://opus.nlpl.eu/datasets/NLLB) | ar, bg, cs, cy, da, de, el, et, fi, fr, ga, hi, hr, hu, it, ja, ko, lt, lv, mt, nl, no, oc, pl, pt, ro, ru, sh, sk, sl, sr, sv, uk, zh | +| [NÓS Authentic Corpus](https://zenodo.org/records/7675110) | gl | +| [NÓS Synthetic Corpus](https://zenodo.org/records/7685180) | gl | +| [NTEU](https://live.european-language-grid.eu/) | da, et, ga, hr, lt, lv, mt, ro, sk, sl, sv | +| [Anuvaad](https://opus.nlpl.eu) | hi | +| [OpenSubtitles](https://opus.nlpl.eu/datasets/OpenSubtitles) | ar, bg, cs, de, el, et, fi, fr, hi, hr, hu, is, ja, ko, no, ru, sh, sl, sr, uk, zh | +| [OPUS-100](https://opus.nlpl.eu/opus-100.php) | gl | +| [ParaCrawl](https://opus.nlpl.eu) | cs, et, is, ko, ru, uk, zh | +| [ParIce](https://opus.nlpl.eu) | is | +| [Samanantar](https://opus.nlpl.eu) | hi | +| [StanfordNLP-NMT](https://opus.nlpl.eu/datasets/StanfordNLP-NMT) | cs | +| [Tatoeba](https://opus.nlpl.eu/datasets/Tatoeba) | ar, cs, et, is, ja, ko, ru, sh, uk | +| [TildeModel](https://opus.nlpl.eu/datasets/TildeMODEL) | cs, et, hr, is, lt, lv, mt, ru, sh, uk | +| [UNPC](https://opus.nlpl.eu/datasets/UNPC) | ar, ru, zh | +| [Wikimedia](https://opus.nlpl.eu/datasets/wikimedia) | cy, nn | +| [XLENT](https://opus.nlpl.eu/datasets/XLEnt) | ar, cs, cy, et, ga, gl, hr, is, ja, ko, oc, ru, sh, uk, zh | + +#### Other pairs + +| Dataset | Language pairs | +|---|---| +| [DGT](https://opus.nlpl.eu/datasets/DGT) | cs-de | +| [EUBookshop](https://opus.nlpl.eu/datasets/EUbookshop) | cs-de, cs-uk | +| [Europarl](https://opus.nlpl.eu/datasets/Europarl) | cs-de | +| [GlobalVoices](https://opus.nlpl.eu/datasets/GlobalVoices) | cs-de | +| [KDE4](https://opus.nlpl.eu/datasets/KDE4) | cs-de, cs-uk, ja-zh | +| [MultiCCAligned](https://opus.nlpl.eu/datasets/MultiCCAligned) | cs-de, cs-uk, ja-zh | +| [MultiParaCrawl](https://opus.nlpl.eu/datasets/MultiParaCrawl) | cs-de, cs-uk | +| [News-Commentary](https://opus.nlpl.eu/datasets/News-Commentary) | cs-de, ja-zh | +| [NLLB](https://opus.nlpl.eu/datasets/NLLB) | cs-de | +| [OpenSubtitles](https://opus.nlpl.eu/datasets/OpenSubtitles) | cs-de, cs-uk, ja-zh | +| [ParaCrawl](https://opus.nlpl.eu) | ja-zh | +| [Tatoeba](https://opus.nlpl.eu/datasets/Tatoeba) | cs-de, cs-uk | +| [TildeModel](https://opus.nlpl.eu/datasets/TildeMODEL) | cs-de | +| [WikiMatrix](https://opus.nlpl.eu/datasets/WikiMatrix) | ja-zh | +| [Wikimedia](https://opus.nlpl.eu/datasets/wikimedia) | ja-zh | +| [XLENT](https://opus.nlpl.eu/datasets/XLEnt) | cs-de, cs-uk, ja-zh | +To consult the data summary document with the respective licences, please send an e-mail to ipr@bsc.es.
@@ -423,122 +474,92 @@ To consult the data summary document with the respective licences, please send a
References - -- Aulamo, M., Sulubacak, U., Virpioja, S., & Tiedemann, J. (2020). OpusTools and Parallel Corpus Diagnostics. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 3782–3789). European Language Resources Association. https://aclanthology.org/2020.lrec-1.467 -- Chaudhary, V., Tang, Y., Guzmán, F., Schwenk, H., & Koehn, P. (2019). Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings. In O. Bojar, R. Chatterjee, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, A. Martins, C. Monz, M. Negri, A. Névéol, M. Neves, M. Post, M. Turchi, & K. Verspoor (Eds.), Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2) (pp. 261–266). Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-5435 + +- Bañón, M., Chen, P., Haddow, B., Heafield, K., Hoang, H., Esplà-Gomis, M., Forcada, M. L., Kamran, A., Kirefu, F., Koehn, P., Ortiz Rojas, S., Pla Sempere, L., Ramírez-Sánchez, G., Sarrías, E., Strelec, M., Thompson, B., Waites, W., Wiggins, D., & Zaragoza, J. (2020). ParaCrawl: Web-Scale Acquisition of Parallel Corpora. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics* (pp. 4555–4567). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.417 +- de-Dios-Flores, I., Paniagua Suárez, S., Carbajal Pérez, C., Bardanca Outeiriño, D., Garcia, M., & Gamallo, P. (2024). CorpusNÓS: A massive Galician corpus for training large language models. In *Proceedings of the 16th International Conference on Computational Processing of Portuguese* (Vol. 1, pp. 593–599). https://aclanthology.org/2024.propor-1.66/ +- Penedo, G., Kydlíček, H., Sabolčec, V., Messmer, B., Foroutan, N., Kargaran, A. H., Raffel, C., Jaggi, M., Von Werra, L., & Wolf, T. (2025). FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language. arXiv. https://arxiv.org/abs/2506.20920 +- Wikimedia Foundation. (2025). Wikimedia Downloads. https://dumps.wikimedia.org +- Aulamo, M., Sulubacak, U., Virpioja, S., & Tiedemann, J. (2020). OpusTools and Parallel Corpus Diagnostics. In *Proceedings of the Twelfth Language Resources and Evaluation Conference* (pp. 3782–3789). European Language Resources Association. https://aclanthology.org/2020.lrec-1.467 +- Chaudhary, V., Tang, Y., Guzmán, F., Schwenk, H., & Koehn, P. (2019). Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings. In *Proceedings of the Fourth Conference on Machine Translation* (Volume 3: Shared Task Papers, Day 2) (pp. 261–266). Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-5435 - DGT-Translation Memory—European Commission. (n.d.). Retrieved November 4, 2024, from https://joint-research-centre.ec.europa.eu/language-technology-resources/dgt-translation-memory_en -- Eisele, A., & Chen, Y. (2010). MultiUN: A Multilingual Corpus from United Nation Documents. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, & D. Tapias (Eds.), Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2010/pdf/686_Paper.pdf -- El-Kishky, A., Chaudhary, V., Guzmán, F., & Koehn, P. (2020). CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5960–5969. https://doi.org/10.18653/v1/2020.emnlp-main.480 -- El-Kishky, A., Renduchintala, A., Cross, J., Guzmán, F., & Koehn, P. (2021). XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 10424–10430. https://doi.org/10.18653/v1/2021.emnlp-main.814 +- Eisele, A., & Chen, Y. (2010). MultiUN: A Multilingual Corpus from United Nation Documents. In *Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)*. European Language Resources Association. http://www.lrec-conf.org/proceedings/lrec2010/pdf/686_Paper.pdf +- El-Kishky, A., Chaudhary, V., Guzmán, F., & Koehn, P. (2020). CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs. *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 5960–5969. https://doi.org/10.18653/v1/2020.emnlp-main.480 +- El-Kishky, A., Renduchintala, A., Cross, J., Guzmán, F., & Koehn, P. (2021). XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment. *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, 10424–10430. https://doi.org/10.18653/v1/2021.emnlp-main.814 - Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Celebi, O., Wenzek, G., Chaudhary, V., Goyal, N., Birch, T., Liptchinsky, V., Edunov, S., Grave, E., Auli, M., & Joulin, A. (2020). Beyond English-Centric Multilingual Machine Translation (No. arXiv:2010.11125). arXiv. https://doi.org/10.48550/arXiv.2010.11125 -- García-Martínez, M., Bié, L., Cerdà, A., Estela, A., Herranz, M., Krišlauks, R., Melero, M., O’Dowd, T., O’Gorman, S., Pinnis, M., Stafanovič, A., Superbo, R., & Vasiļevskis, A. (2021). Neural Translation for European Union (NTEU). 316–334. https://aclanthology.org/2021.mtsummit-up.23 -- Gibert, O. de, Nail, G., Arefyev, N., Bañón, M., Linde, J. van der, Ji, S., Zaragoza-Bernabeu, J., Aulamo, M., Ramírez-Sánchez, G., Kutuzov, A., Pyysalo, S., Oepen, S., & Tiedemann, J. (2024). A New Massive Multilingual Dataset for High-Performance Language Technologies (No. arXiv:2403.14009). arXiv. http://arxiv.org/abs/2403.14009 -- Koehn, P. (2005). Europarl: A Parallel Corpus for Statistical Machine Translation. Proceedings of Machine Translation Summit X: Papers, 79–86. https://aclanthology.org/2005.mtsummit-papers.11 -- Kreutzer, J., Caswell, I., Wang, L., Wahab, A., Van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A., Papadimitriou, I., Osei, S., Suarez, P. O., … Adeyemi, M. (2022). Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 10, 50–72. https://doi.org/10.1162/tacl_a_00447 -- Rozis, R.,Skadiņš, R (2017). Tilde MODEL - Multilingual Open Data for EU Languages. https://aclanthology.org/W17-0235 +- Galiano-Jiménez, A., Sánchez-Martínez, F., & Pérez-Ortiz, J. A. (Eds.). (2024). PILAR. GitHub. https://github.com/transducens/PILAR +- García-Martínez, M., Bié, L., Cerdà, A., Estela, A., Herranz, M., Krišlauks, R., Melero, M., O'Dowd, T., O'Gorman, S., Pinnis, M., Stafanovič, A., Superbo, R., & Vasiļevskis, A. (2021). Neural Translation for European Union (NTEU). 316–334. https://aclanthology.org/2021.mtsummit-up.23 +- de Gibert, O., Nail, G., Arefyev, N., Bañón, M., van der Linde, J., Ji, S., Zaragoza-Bernabeu, J., Aulamo, M., Ramírez-Sánchez, G., Kutuzov, A., Pyysalo, S., Oepen, S., & Tiedemann, J. (2024). A New Massive Multilingual Dataset for High-Performance Language Technologies (No. arXiv:2403.14009). arXiv. http://arxiv.org/abs/2403.14009 +- Koehn, P. (2005). Europarl: A Parallel Corpus for Statistical Machine Translation. *Proceedings of Machine Translation Summit X: Papers*, 79–86. https://aclanthology.org/2005.mtsummit-papers.11 +- Kreutzer, J., Caswell, I., Wang, L., Wahab, A., Van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A., Papadimitriou, I., Osei, S., Suarez, P. O., … Adeyemi, M. (2022). Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. *Transactions of the Association for Computational Linguistics*, 10, 50–72. https://doi.org/10.1162/tacl_a_00447 +- Lison, P., & Tiedemann, J. (2016). OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)* (pp. 923–929). European Language Resources Association. https://aclanthology.org/L16-1147/ +- Ramesh, G., Doddapaneni, S., Bheemaraj, A., Jobanputra, M., AK, R., Sharma, A., Sahoo, S., Diddee, H., J, M., Kakwani, D., Kumar, N., Pradeep, A., Nagaraj, S., Deepak, K., Raghavan, V., Kunchukuttan, A., Kumar, P., & Khapra, M. S. (2022). Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages. *Transactions of the Association for Computational Linguistics*, 10, 145–162. https://doi.org/10.1162/tacl_a_00452 +- Rozis, R., & Skadiņš, R. (2017). Tilde MODEL - Multilingual Open Data for EU Languages. https://aclanthology.org/W17-0235 - Schwenk, H., Chaudhary, V., Sun, S., Gong, H., & Guzmán, F. (2019). WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (No. arXiv:1907.05791). arXiv. https://doi.org/10.48550/arXiv.1907.05791 - Schwenk, H., Wenzek, G., Edunov, S., Grave, E., & Joulin, A. (2020). CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB (No. arXiv:1911.04944). arXiv. https://doi.org/10.48550/arXiv.1911.04944 +- Skadiņš, R., Tiedemann, J., Rozis, R., & Deksne, D. (2014). Billions of Parallel Words for Free: Building and Using the EU Bookshop Corpus. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)* (pp. 1850–1855). European Language Resources Association. https://aclanthology.org/L14-1652/ - Steinberger, R., Pouliquen, B., Widiger, A., Ignat, C., Erjavec, T., Tufiş, D., & Varga, D. (n.d.). The JRC-Acquis: A Multilingual Aligned Parallel Corpus with 20+ Languages. http://www.lrec-conf.org/proceedings/lrec2006/pdf/340_pdf -- Subramani, N., Luccioni, S., Dodge, J., & Mitchell, M. (2023). Detecting Personal Information in Training Corpora: An Analysis. In A. Ovalle, K.-W. Chang, N. Mehrabi, Y. Pruksachatkun, A. Galystan, J. Dhamala, A. Verma, T. Cao, A. Kumar, & R. Gupta (Eds.), Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) (pp. 208–220). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.trustnlp-1.18 -- Tiedemann, J. (23-25). Parallel Data, Tools and Interfaces in OPUS. In N. C. (Conference Chair), K. Choukri, T. Declerck, M. U. Doğan, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12). European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper +- Subramani, N., Luccioni, S., Dodge, J., & Mitchell, M. (2023). Detecting Personal Information in Training Corpora: An Analysis. In *Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)* (pp. 208–220). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.trustnlp-1.18 +- Tiedemann, J. (2012). Parallel Data, Tools and Interfaces in OPUS. In *Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)*. European Language Resources Association. http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper - Ziemski, M., Junczys-Dowmunt, M., & Pouliquen, B. (n.d.). The United Nations Parallel Corpus v1.0. https://aclanthology.org/L16-1561 - -
### Instruction Tuning Data -This model has been fine-tuned on ~135k instructions, primarily targeting machine translation performance for Catalan, English, and Spanish. -Additional instruction data for other European and closely related Iberian languages was also included, as it yielded a positive impact on the languages of interest. -That said, the performance in these additional languages is not guaranteed due to the limited amount of available data and the lack of resources for thorough testing. - -A portion of our fine-tuning data comes directly from, or is sampled from [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2). -We also created additional datasets for our main languages of interest. -While tasks relating to machine translation are included, it’s important to note that no chat data was used in the fine-tuning process. -The final distribution of tasks was as below: +This model has been fine-tuned on ~683k instructions, primarily targeting general machine translation tasks. It's important to note that no chat data was used in the fine-tuning process. +We created instructions using the following datasets: -![](./images/chart.png) - - -Click the expand button below to see the full list of tasks included in the finetuning data. - -
-Data Sources +- [AnCora-Ca-NER](https://huggingface.co/datasets/projecte-aina/ancora-ca-ner) +- [EIEC (Basque Named Entities Corpus)](https://www.ixa.eus/node/4486?language=en) +- [SLI NERC Galician Gold Corpus](https://github.com/xavier-gz/SLI_Galician_Corpora) +- [ACAData](https://huggingface.co/datasets/BSC-LT/ACAData) +- [NTEU](https://huggingface.co/datasets/BSC-LT/NTEU_Multilingual_Evaluation_Dataset) +- [News-Commentary](https://huggingface.co/datasets/Helsinki-NLP/news_commentary) +- [NewsPalm](https://github.com/google-research/google-research/tree/master/newspalm) +- [Tatoeba](https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt) +- [WMT++](https://huggingface.co/datasets/google/wmt24pp) +- [FLEURS](https://huggingface.co/datasets/google/fleurs) +- [LangMark](https://zenodo.org/records/15553365) +- [Q21](https://lindat.mff.cuni.cz/repository/items/f22bfcdc-413d-459a-a550-cf8851245cac) +- [ApeQuest](https://apequest.wordpress.com/downloads/) +- [Europarl](https://huggingface.co/datasets/Helsinki-NLP/europarl) +- [Project Gutenberg](https://www.gutenberg.org/) +- [GLITTER](https://huggingface.co/datasets/mbvslowride/glitter) +- [GeNTE](https://huggingface.co/datasets/FBK-MT/GeNTE) +- [EuroGEST](https://huggingface.co/datasets/utter-project/EuroGEST) +- Eupress (Internally created dataset, in course of publication) -| Task | Source | Languages | Count | -|----------------------------------|------------------------------------------------------------------------------------------|----------------------------------------------------------------|--------| -| Multi-reference Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [Tatoeba Dev (filtered)](https://github.com/Helsinki-NLP/Tatoeba-Challenge) | mixed | 10000 | -| Paraphrase | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [PAWS-X Dev](https://github.com/google-research-datasets/paws) | mixed | 3521 | -| Named-entity Recognition | [AnCora-Ca-NER](https://huggingface.co/datasets/projecte-aina/ancora-ca-ner) | ca | 12059 | -| Named-entity Recognition | [BasqueGLUE](https://huggingface.co/datasets/orai-nlp/basqueGLUE), [EusIE](https://huggingface.co/datasets/HiTZ/EusIE) | eu | 4304 | -| Named-entity Recognition | [SLI NERC Galician Gold Corpus](https://github.com/xavier-gz/SLI_Galician_Corpora) | gl | 6483 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | pt | 854 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | nl | 800 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | es | 1654 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | en | 1671 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | ru | 800 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | it | 858 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | fr | 857 | -| Named-entity Recognition | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MultiCoNER 2022 and 2023 Dev](https://registry.opendata.aws/multiconer/) | de | 1312 | -| Terminology-aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [WMT21 Terminology Dev (filtered)](https://www.statmt.org/wmt21/terminology-task.html) | en-ru | 50 | -| Terminology-aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [WMT21 Terminology Dev (filtered)](https://www.statmt.org/wmt21/terminology-task.html) | en-fr | 29 | -| Automatic Post Editing | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [QT21](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-2390), [ApeQuest](https://apequest.wordpress.com/) | en-fr | 6133 | -| Automatic Post Editing | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [QT21](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-2390), [ApeQuest](https://apequest.wordpress.com/) | en-nl | 9077 | -| Automatic Post Editing | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [QT21](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-2390), [ApeQuest](https://apequest.wordpress.com/) | en-pt | 5762 | -| Automatic Post Editing | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [QT21](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-2390), [ApeQuest](https://apequest.wordpress.com/) | de-en | 10000 | -| Automatic Post Editing | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [QT21](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-2390), [ApeQuest](https://apequest.wordpress.com/) | en-de | 10000 | -| Machine Translation Evaluation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2)-sample: [WMT20 to WMT22 Metrics MQM](https://www.statmt.org/wmt22/results.html), [WMT17 to WMT22 Metrics Direct Assessments](https://www.statmt.org/wmt22/results.html) | en-ru, en-pl, ru-en, en-de, en-ru, de-fr, de-en, en-de | 353 | -| Machine Translation Evaluation | Non-public | four pivot languages (eu, es, ca, gl) paired with European languages (bg, cs, da, de, el, en, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv) | 9700 | -| General Machine Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [WMT14 to WMT21](https://www.statmt.org/wmt22/results.html), [NTREX](https://github.com/MicrosoftTranslator/NTREX), [Flores Dev](https://github.com/facebookresearch/flores), [FRMT](https://github.com/google-research/google-research/tree/master/frmt), [QT21](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-2390), [ApeQuest](https://apequest.wordpress.com/), [OPUS (Quality Filtered)](https://opus.nlpl.eu/), [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval) | nl-en, en-ru, it-en, fr-en, es-en, en-fr, ru-en, fr-de, en-nl, de-fr | 500 | -| General Machine Translation | Non-public | three pivot languages (es, ca, en) paired with European languages (ast, arn, arg, bg, cs, cy, da, de, el, et, fi, ga, gl, hr, it, lt, lv, mt, nb, nn, nl, oc, pl, pt, ro, ru, sk, sl, sr, sv, uk, eu) | 9350 | -| Fill-in-the-Blank | Non-public | five pivot languages (ca, es, eu, gl, en) paired with European languages (cs, da, de, el, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv) | 11500 | -| Document-level Translation | Non-public | two pivot languages (es, en) paired with European languages (bg, cs, da, de, el, et, fi, fr, hu, it, lt, lv, nl, pl, pt, ro, ru, sk, sv) | 7600 | -| Paragraph-level Translation | Non-public | two pivot languages (es, en) paired with European languages (bg, cs, da, de, el, et, fi, fr, hu, it, lt, lv, nl, pl, pt, ro, ru, sk, sv) | 7600 | -| Context-Aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval) | en-it | 348 | -| Context-Aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval) | en-ru | 454 | -| Context-Aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval) | en-fr | 369 | -| Context-Aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval) | en-nl | 417 | -| Context-Aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval) | en-es | 431 | -| Context-Aware Translation | [TowerBlocks](https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2): [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval) | en-de | 558 | -|**Total** | | | **135,404** | - -The non-public portion of this dataset was jointly created by the [ILENIA](https://proyectoilenia.es/) partners: BSC-LT, [HiTZ](http://hitz.ehu.eus/es), -and [CiTIUS](https://citius.gal/es/). For further information regarding the instruction-tuning data, -please contact . - -
References -- Alves, D. M., Pombal, J., Guerreiro, N. M., Martins, P. H., Alves, J., Farajian, A., Peters, B., Rei, R., Fernandes, P., Agrawal, S., Colombo, P., de Souza, J. G. C., & Martins, A. F. T. (2024). Tower: An open multilingual large language model for translation-related tasks (No. arXiv: 2402.17733). arXiv. https://arxiv.org/abs/2402.17733 -- Armengol-Estapé, J., Carrino, C. P., Rodriguez-Penagos, C., de Gibert Bonet, O., Armentano-Oller, C., Gonzalez-Agirre, A., Melero, M., & Villegas, M. (2021). Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 4933–4946. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.findings-acl.437 -- Currey, A., Nadejde, M., Pappagari, R. R., Mayer, M., Lauly, S., Niu, X., Hsu, B., & Dinu, G. (2022). MT-GenEval: A counterfactual and contextual dataset for evaluating gender accuracy in machine translation. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 4287–4299). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.emnlp-main.288 -- Federmann, C., Kocmi, T., & Xin, Y. (2022). NTREX-128 – News test references for MT evaluation of 128 languages. Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, 21–24. Association for Computational Linguistics. https://aclanthology.org/2022.sumeval-1.4 -- Ive, J., Specia, L., Szoc, S., Vanallemeersch, T., Van den Bogaert, J., Farah, E., Maroti, C., Ventura, A., & Khalilov, M. (2020). A post-editing dataset in the legal domain: Do we underestimate neural machine translation quality? In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 3692–3697). European Language Resources Association. https://aclanthology.org/2020.lrec-1.455/ -- Malmasi, S., Fang, A., Fetahu, B., Kar, S., & Rokhlenko, O. (2022). MultiCoNER: A large-scale multilingual dataset for complex named entity recognition. Proceedings of the 29th International Conference on Computational Linguistics, 3798–3809. International Committee on Computational Linguistics. https://aclanthology.org/2022.coling-1.334/ -- NLLB Team, Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Mejia Gonzalez, G., Hansanti, P., Hoffman, J., Jarrett, S., Sadagopan, K. R., Rowe, D., Spruit, S., Tran, C., Andrews, P., Ayan, N. F., Bhosale, S., Edunov, S., Fan, A., Gao, C., Goswami, V., Guzmán, F., Koehn, P., Mourachko, A., Ropers, C., Saleem, S., Schwenk, H., & Wang, J. (2022). No language left behind: Scaling human-centered machine translation (No. arXiv: 2207.04672). arXiv. https://arxiv.org/abs/2207.04672 -- Riley, P., Dozat, T., Botha, J. A., Garcia, X., Garrette, D., Riesa, J., Firat, O., & Constant, N. (2022). FRMT: A benchmark for few-shot region-aware machine translation (No. arXiv: 2210.00193). arXiv. https://doi.org/10.48550/ARXIV.2210.00193 -- Specia, L., Harris, K., Blain, F., Burchardt, A., Macketanz, V., Skadiņa, I., Negri, M., & Turchi, M. (2017). Translation quality and productivity: A study on rich morphology languages. Proceedings of Machine Translation Summit XVI, 55–71. Nagoya, Japan. -- Tiedemann, J. (2020). The Tatoeba translation challenge – Realistic data sets for low-resource and multilingual MT. Proceedings of the Fifth Conference on Machine Translation, 1174–1182. Association for Computational Linguistics. https://www.aclweb.org/anthology/2020.wmt-1.139 -- Urbizu, G., San Vicente, I., Saralegi, X., Agerri, R., & Soroa, A. (2022). BasqueGLUE: A natural language understanding benchmark for Basque. Proceedings of the Language Resources and Evaluation Conference, 1603–1612. European Language Resources Association. https://aclanthology.org/2022.lrec-1.172 -- Yang, Y., Zhang, Y., Tar, C., & Baldridge, J. (2019). PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3687–3692). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1382 -- Zubillaga, M., Sainz, O., Estarrona, A., Lopez de Lacalle, O., & Agirre, E. (2024). Event extraction in Basque: Typologically motivated cross-lingual transfer-learning analysis (No. arXiv: 2404.06392). arXiv. https://arxiv.org/abs/2404.06392 - - +### References + +- Conneau, A., Ma, M., Khanuja, S., Zhang, Y., Axelrod, V., Dalmia, S., Riesa, J., Rivera, C., & Bapna, A. (2022). FLEURS: Few-shot learning evaluation of universal representations of speech. *2022 IEEE Spoken Language Technology Workshop (SLT)*, 798–805. https://arxiv.org/abs/2205.12446 +- Deutsch, D., Briakou, E., Caswell, I., Finkelstein, M., Galor, R., Juraska, J., Kovacs, G., Lui, A., Rei, R., Riesa, J., Rijhwani, S., Riley, P., Salesky, E., Trabelsi, F., Winkler, S., Zhang, B., & Freitag, M. (2025). WMT24++: Expanding the language coverage of WMT24 to 55 languages & dialects (No. arXiv:2502.12404). arXiv. https://arxiv.org/abs/2502.12404 +- Finkelstein, M., Juraska, J., & Freitag, M. (2024). Introducing the NewsPaLM MBR and QE dataset: LLM-generated high-quality parallel data outperforms traditional web-crawled data (No. arXiv:2408.06537). arXiv. https://arxiv.org/abs/2408.06537 +- Ive, J., Specia, L., Szoc, S., Vanallemeersch, T., Van den Bogaert, J., Farah, E., Maroti, C., Ventura, A., & Khalilov, M. (2020). A post-editing dataset in the legal domain: Do we underestimate neural machine translation quality? In N. Calzolari et al. (Eds.), *Proceedings of the Twelfth Language Resources and Evaluation Conference* (pp. 3692–3697). European Language Resources Association. https://aclanthology.org/2020.lrec-1.455/ +- Lacunza, I., Garcia Gilabert, J., De Luca Fornaciari, F., Aula-Blasco, J., Gonzalez-Agirre, A., Melero, M., & Villegas, M. (2025). ACAData: Parallel dataset of academic data for machine translation (No. arXiv:2510.12621). arXiv. https://arxiv.org/abs/2510.12621 +- Piergentili, A., Savoldi, B., Fucci, D., Negri, M., & Bentivogli, L. (2023). Hi guys or hi folks? Benchmarking gender-neutral machine translation with the GeNTE corpus. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing* (pp. 7892–7907). Association for Computational Linguistics. https://arxiv.org/abs/2310.05294 +- Project Gutenberg. (n.d.). Project Gutenberg. https://www.gutenberg.org/ +- Rowe, J., Klimaszewski, M., Guillou, L., Vallor, S., & Birch, A. (2025). EuroGEST: Investigating gender stereotypes in multilingual language models. In *Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*. https://arxiv.org/abs/2506.03867 +- Specia, L., Harris, K., Blain, F., Burchardt, A., Macketanz, V., Skadiņa, I., Negri, M., & Turchi, M. (2017). Translation quality and productivity: A study on rich morphology languages. *Proceedings of Machine Translation Summit XVI*, 55–71. +- Tiedemann, J. (2020). The Tatoeba translation challenge – Realistic data sets for low-resource and multilingual MT. *Proceedings of the Fifth Conference on Machine Translation*, 1174–1182. Association for Computational Linguistics. https://aclanthology.org/2020.wmt-1.139 +- Urbizu, G., San Vicente, I., Saralegi, X., Agerri, R., & Soroa, A. (2022). BasqueGLUE: A natural language understanding benchmark for Basque. *Proceedings of the Language Resources and Evaluation Conference*, 1603–1612. European Language Resources Association. https://aclanthology.org/2022.lrec-1.172 +- Velazquez, D., Grace, M., Karageorgos, K., Carin, L., Schliem, A., Zaikis, D., & Wechsler, R. (2025). LangMark: A multilingual dataset for automatic post-editing. In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics* (Vol. 1, pp. 32653–32667). Association for Computational Linguistics. https://aclanthology.org/2025.acl-long.1569/
## Evaluation -Below are the evaluation results on the [Flores+200 devtest set](https://huggingface.co/datasets/openlanguagedata/flores_plus), -compared against the state-of-the-art [MADLAD400-7B-mt model](https://huggingface.co/google/madlad400-7b-mt) ([Kudugunta, S., et al.](https://arxiv.org/abs/2309.04662)) and SalamandraTA-7b-base model. +Below are the evaluation results on the [Flores+200 devtest set](https://huggingface.co/datasets/openlanguagedata/flores_plus) and [BOUQuET test subset (sentence-levl)](https://huggingface.co/datasets/facebook/bouquet), +compared against the state-of-the-art [MADLAD400-7B-mt model](https://huggingface.co/google/madlad400-7b-mt) ([Kudugunta, S., et al.](https://arxiv.org/abs/2309.04662)), and our previously released translation LLM. These results cover the translation directions CA-XX, ES-XX, EN-XX, as well as XX-CA, XX-ES, and XX-EN. -The metrics have been computed excluding Asturian, Aranese, and Aragonese, as we report them separately. +The metrics have been computed excluding Asturian, Aranese, and Aragonese, as we report them separately. +We additionally evaluate on the newly added Asian languages (Chinese, Japanese, Korean, Hindi, and Arabic), reported separately with comparisons against [MADLAD400-7B-mt model](https://huggingface.co/google/madlad400-7b-mt) ([Kudugunta, S., et al.](https://arxiv.org/abs/2309.04662)) and [NLLB-200-3.3B](https://huggingface.co/facebook/nllb-200-3.3B) ([Costa-Jussà, M. R. et al.](https://arxiv.org/abs/2207.04672)). The evaluation was conducted using [MT-Lens](https://github.com/langtech-bsc/mt-evaluation), following the standard setting (beam search with beam size 5, limiting the translation length to 500 tokens). We report the following metrics:
@@ -563,23 +584,35 @@ The evaluation was conducted using [MT-Lens](https://github.com/langtech-bsc/mt- This section presents the evaluation metrics for English translation tasks. +#### Flores+200 devtest set -| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | -|:---------------------------------|-------:|------:|-------:|--------:|-------------:|---------:|----------:|-------------:| -| **EN-XX** | | | | | | | | | -| SalamandraTA-7b-instruct | 35.20 | 53.40 | 61.58 | **0.89** | **0.86** | 0.78 | **0.96** | **0.81** | -| MADLAD400-7B | **35.73** | **51.87** | **63.46** | 0.88 | 0.85 | **0.79** | 1.16 | 1.10 | -| SalamandraTA-7b-base | 34.99 | 52.64 | 62.58 | 0.87 | 0.84 | 0.77 | 1.45 | 1.23 | -| **XX-EN** | | | | | | | | | -| SalamandraTA-7b-instruct | **44.37** | **42.49** | 68.29 | **0.89** | **0.86** | **0.80** | **1.05** | **0.99** | -| MADLAD400-7B | 43.20 | 43.33 | 67.98 | **0.89** | **0.86** | **0.80** | 1.13 | 1.15 | -| SalamandraTA-7b-base | 44.12 | 43.00 | **68.43** | **0.89** | 0.85 | **0.80** | 1.13 | 1.22 | - +| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | +|:---|-------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:| +| **EN-XX** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **37.71** | **49.91** | 64.21 | **0.8919** | 0.8556 | **0.8014** | **0.8491** | **0.7264** | +| SalamandraTA-7b-instruct (v1) | 37.41 | 50.78 | **64.23** | 0.8901 | **0.8561** | 0.7972 | 0.9377 | 0.8045 | +| MADLAD400-7B | 34.69 | 56.78 | 62.14 | 0.8751 | 0.8341 | 0.7790 | 1.5296 | 1.5076 | +| **XX-EN** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **45.87** | **40.98** | **69.24** | **0.8897** | 0.8553 | **0.8036** | **1.0281** | **0.9888** | +| SalamandraTA-7b-instruct (v1) | 44.96 | 41.94 | 68.65 | 0.8884 | 0.8546 | 0.8012 | 1.0419 | 0.9934 | +| MADLAD400-7B | 43.81 | 42.76 | 68.35 | 0.8863 | **0.8554** | 0.7970 | 1.1264 | 1.1430 | +English FLORES +#### BOUQuET test subset (sentence-levl) +| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | +|:---|-------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:| +| **EN-XX** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **45.06** | **44.38** | **66.75** | **0.91** | **0.85** | **0.82** | **0.72** | 0.91 | +| SalamandraTA-7b-instruct (v1) | 44.05 | 45.84 | 66.31 | 0.90 | 0.85 | 0.81 | 0.75 | 0.97 | +| MADLAD400-7B | 43.78 | 46.43 | 66.64 | 0.90 | 0.85 | 0.81 | 0.83 | **0.88** | +| **XX-EN** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **49.97** | **38.42** | 68.51 | 0.90 | 0.85 | 0.81 | 0.82 | 0.87 | +| SalamandraTA-7b-instruct (v1) | 48.53 | 39.79 | 67.48 | 0.90 | 0.85 | 0.80 | 0.82 | 0.84 | +| MADLAD400-7B | 49.95 | 38.66 | **68.68** | **0.90** | **0.85** | **0.81** | **0.76** | **0.69** | -English +English BOUQuET
@@ -590,23 +623,38 @@ This section presents the evaluation metrics for English translation tasks. This section presents the evaluation metrics for Spanish translation tasks. +#### Flores+200 devtest set -| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | -|:---------------------------------|-------:|------:|-------:|--------:|-------------:|---------:|----------:|-------------:| -| **ES-XX** | | | | | | | | | -| SalamandraTA-7b-instruct | **23.68** | **67.31** | **53.98** | **0.87** | **0.83** | **0.76** | **0.93** | **0.80** | -| MADLAD400-7B | 22.48 | 68.91 | 53.93 | 0.86 | **0.83** | 0.75 | 1.09 | 1.14 | -| SalamandraTA-7b-base | 21.63 | 70.08 | 52.98 | 0.86 | **0.83** | 0.74 | 1.24 | 1.12 | -| **XX-ES** | | | | | | | | | -| SalamandraTA-7b-instruct | **26.40** | 62.27 | **53.54** | **0.85** | **0.84** | **0.74** | **0.80** | **1.07** | -| MADLAD400-7B | 24.85 | **61.82** | 53.00 | **0.85** | **0.84** | **0.74** | 1.05 | 1.50 | -| SalamandraTA-7b-base | 24.71 | 62.33 | 52.96 | **0.85** | **0.84** | 0.73 | 1.06 | 1.37 | +| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | +|:---|-------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:| +| **ES-XX** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **24.57** | **65.43** | **54.51** | **0.8687** | 0.8212 | **0.7615** | **0.9042** | **0.7890** | +| SalamandraTA-7b-instruct (v1) | 24.04 | 66.92 | 54.31 | 0.8682 | **0.8267** | 0.7599 | 0.9244 | 0.7910 | +| MADLAD400-7B | 21.83 | 72.41 | 52.96 | 0.8578 | 0.8263 | 0.7396 | 1.2570 | 1.3107 | +| **XX-ES** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | 25.97 | **60.53** | 53.44 | 0.8529 | 0.8389 | **0.7455** | 0.8861 | 1.1598 | +| SalamandraTA-7b-instruct (v1) | **26.48** | 62.12 | **53.62** | **0.8533** | 0.8378 | 0.7443 | **0.7958** | **1.0700** | +| MADLAD400-7B | 24.90 | 61.75 | 53.04 | 0.8488 | **0.8396** | 0.7390 | 1.0484 | 1.5098 | +Spanish FLORES -English -ESXX +#### BOUQuET test subset (sentence-levl) + +| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | +|:---|-------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:| +| **ES-XX** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **34.67** | 56.05 | 58.58 | **0.89** | 0.82 | **0.79** | **0.74** | 0.85 | +| SalamandraTA-7b-instruct (v1) | 33.15 | 58.43 | 57.74 | 0.89 | **0.82** | 0.79 | 0.75 | 0.85 | +| MADLAD400-7B | 34.60 | **55.99** | **59.07** | 0.88 | 0.82 | 0.79 | 0.79 | **0.82** | +| **XX-ES** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | 38.12 | **51.40** | 60.73 | 0.88 | 0.82 | 0.79 | 0.71 | 0.95 | +| SalamandraTA-7b-instruct (v1) | 32.64 | 57.98 | 57.51 | 0.87 | 0.82 | 0.78 | 0.66 | 0.90 | +| MADLAD400-7B | **38.86** | 51.46 | **61.48** | **0.88** | **0.83** | **0.79** | **0.64** | **0.78** | + +Spanish BOUQuET +
@@ -617,120 +665,95 @@ This section presents the evaluation metrics for Spanish translation tasks. This section presents the evaluation metrics for Catalan translation tasks. - -| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | -|:---------------------------------|-------:|------:|-------:|--------:|-------------:|---------:|----------:|-------------:| -| **CA-XX** | | | | | | | | | -| SalamandraTA-7b-instruct | **29.50** | 59.26 | 58.21 | **0.88** | **0.81** | **0.77** | **0.97** | **0.98** | -| MADLAD400-7B | 29.37 | **59.01** | **58.47** | 0.87 | **0.81** | **0.77** | 1.08 | 1.31 | -| SalamandraTA-7b-base | 29.06 | 59.32 | 58.00 | 0.87 | **0.81** | 0.76 | 1.23 | 1.28 | -| **XX-CA** | | | | | | | | | -| SalamandraTA-7b-instruct | **34.51** | **54.21** | **60.10** | **0.86** | **0.81** | **0.76** | **0.90** | **1.29** | -| MADLAD400-7B | 33.02 | 55.01 | 59.38 | **0.86** | **0.81** | 0.75 | 1.18 | 1.79 | -| SalamandraTA-7b-base | 32.75 | 55.78 | 59.42 | **0.86** | **0.81** | 0.75 | 1.17 | 1.63 | +#### Flores+200 devtest set +| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | +|:---|-------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:| +| **CA-XX** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **30.32** | **57.69** | **58.67** | **0.8763** | 0.8092 | **0.7777** | **0.9215** | **0.9640** | +| SalamandraTA-7b-instruct (v1) | 30.10 | 58.64 | 58.65 | 0.8760 | **0.8150** | 0.7750 | 0.9631 | 0.9779 | +| MADLAD400-7B | 28.52 | 67.29 | 57.20 | 0.8645 | 0.8111 | 0.7592 | 1.3280 | 1.6205 | +| **XX-CA** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **35.28** | **52.92** | **60.58** | **0.8671** | 0.8129 | **0.7641** | **0.8870** | 1.2876 | +| SalamandraTA-7b-instruct (v1) | 34.72 | 53.96 | 60.25 | 0.8640 | 0.8114 | 0.7577 | 0.9004 | **1.2875** | +| MADLAD400-7B | 33.18 | 54.77 | 59.49 | 0.8605 | **0.8130** | 0.7506 | 1.1917 | 1.7937 | -English +Catalan FLORES - - -
-Galician evaluation - -### Galician - -This section presents the evaluation metrics for Galician translation tasks. +#### BOUQuET test subset (sentence-levl) -| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | -|:---------------------------------|-------:|------:|-------:|--------:|-------------:|---------:|----------:|-------------:| -| **GL-XX** | | | | | | | | | -| SalamandraTA-7b-instruct | **36.95** | **50.12** | **62.55** | **0.88** | **0.85** | **0.77** | **0.86** | **0.98** | -| MADLAD400-7B | 26.43 | 64.30 | 55.99 | 0.86 | **0.85** | 0.76 | 1.35 | 2.06 | -| SalamandraTA-7b-base | 27.47 | 61.39 | 56.96 | 0.87 | 0.82 | 0.76 | 1.23 | 1.29 | -| **XX-GL** | | | | | | | | | -| SalamandraTA-7b-instruct | **34.37** | **52.49** | **60.99** | **0.88** | **0.85** | **0.73** | **0.75** | **0.92** | -| MADLAD400-7B | 27.77 | 59.46 | 54.92 | 0.84 | **0.85** | 0.67 | 1.42 | 2.72 | -| SalamandraTA-7b-base | 28.22 | 59.52 | 56.28 | 0.85 | 0.82 | 0.69 | 1.27 | 1.78 | +| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | +|:---|-------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:| +| **CA-XX** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | 31.56 | 61.98 | 56.81 | **0.88** | 0.79 | 0.78 | **0.78** | **1.05** | +| SalamandraTA-7b-instruct (v1) | 31.45 | 60.77 | 56.71 | 0.88 | **0.79** | **0.78** | 0.80 | 1.06 | +| MADLAD400-7B | **32.30** | **58.65** | **56.96** | 0.87 | 0.79 | 0.78 | 0.85 | 1.17 | +| **XX-CA** | | | | | | | | | +| SalamandraTA-7b-instruct (v2) | **34.50** | **54.62** | **57.69** | **0.86** | 0.77 | **0.76** | **0.90** | 1.43 | +| SalamandraTA-7b-instruct (v1) | 32.99 | 57.49 | 56.26 | 0.86 | **0.78** | 0.76 | 0.91 | **1.39** | +| MADLAD400-7B | 32.19 | 57.27 | 55.85 | 0.85 | 0.77 | 0.75 | 1.00 | 1.56 | -English +Catalan BOUQuET
-
-Basque evaluation - -### Basque - -This section presents the evaluation metrics for Basque translation tasks. - - - -| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ | -|:---------------------------------|-------:|------:|-------:|--------:|-------------:|---------:|----------:|-------------:| -| **EU-XX** | | | | | | | | | -| SalamandraTA-7b-instruct | **29.89** | **58.54** | **56.66** | **0.87** | **0.85** | **0.76** | **0.90** | **0.89** | -| MADLAD400-7B | 21.26 | 69.75 | 49.80 | 0.85 | 0.82 | 0.72 | 1.54 | 2.71 | -| SalamandraTA-7b-base | 22.87 | 67.38 | 52.19 | 0.86 | 0.79 | 0.74 | 1.19 | 1.61 | -| **XX-EU** | | | | | | | | | -| SalamandraTA-7b-instruct | **18.89** | **71.74** | **57.16** | **0.87** | **0.84** | **0.82** | **0.58** | **0.44** | -| MADLAD400-7B | 13.64 | 85.01 | 50.96 | 0.82 | 0.80 | 0.78 | 2.09 | 3.58 | -| SalamandraTA-7b-base | 17.01 | 75.92 | 55.22 | 0.85 | 0.77 | 0.80 | 1.04 | 1.17 | - - -English - -
### Low-Resource Languages of Spain -The tables below summarize the performance metrics for English, Spanish, and Catalan to Asturian, Aranese and Aragonese compared -against [Transducens/IbRo-nllb](https://huggingface.co/Transducens/IbRo-nllb) [(Galiano Jimenez, et al.)](https://aclanthology.org/2024.wmt-1.85/), -[NLLB-200-3.3B](https://huggingface.co/facebook/nllb-200-3.3B) ([Costa-jussà et al., 2022](https://arxiv.org/abs/2207.04672)) and [SalamandraTA-2B](https://huggingface.co/BSC-LT/salamandraTA-2B). +The tables below report performance metrics on the FLORES-200 devtest set for translation from English, Spanish, and Catalan into Asturian, Aranese, and Aragonese. Results are compared +to [Transducens/IbRo-nllb](https://huggingface.co/Transducens/IbRo-nllb) [(Galiano Jimenez, et al.)](https://aclanthology.org/2024.wmt-1.85/). +
English evaluation - + #### English-XX -| | source | target | Bleu ↑ | Ter ↓ | ChrF ↑ | -|:-------------------------|:---------|:---------|:----------|:----------|:----------| -| SalamandraTA-7b-instruct | en | ast | **31.79** | **54.07** | **61.78** | -| SalamandraTA-7b-base | en | ast | 26.40 | 64.02 | 57.35 | -| Transducens/IbRo-nllb | en | ast | 20.56 | 63.92 | 53.32 | +| Model | source | target | Bleu ↑ | Ter ↓ | ChrF ↑ | +|:--------------------------|:-------|:-------|:-----------|:-----------|:-----------| +| SalamandraTA-7b-instruct (v2) | en | ast | **34.22** | **51.47** | **62.02** | +| SalamandraTA-7b-instruct (v1) | | | 31.79 | 54.07 | 61.78 | +| Transducens/IbRo-nllb | | | 20.56 | 63.92 | 53.32 | | | | | | | | -| SalamandraTA-7b-instruct | en | arn | **22.77** | **66.06** | **52.61** | -| SalamandraTA-7b-base | en | arn | 14.13 | 74.05 | 46.17 | -| Transducens/IbRo-nllb | en | arn | 12.81 | 73.21 | 45.76 | +| SalamandraTA-7b-instruct (v2) | en | arn | **26.20** | **60.90** | **55.21** | +| SalamandraTA-7b-instruct (v1) | | | 22.77 | 66.06 | 52.61 | +| Transducens/IbRo-nllb | | | 12.81 | 73.21 | 45.76 | | | | | | | | -| SalamandraTA-7b-instruct | en | arg | **19.74** | 71.58 | **51.08** | -| Transducens/IbRo-nllb | en | arg | 14.07 | **70.37** | 46.89 | -| SalamandraTA-7b-base | en | arg | 12.24 | 73.48 | 44.75 | +| SalamandraTA-7b-instruct (v2) | en | arg | **25.40** | **60.11** | **54.42** | +| SalamandraTA-7b-instruct (v1) | | | 19.74 | 71.58 | 51.08 | +| Transducens/IbRo-nllb | | | 14.07 | 70.37 | 46.89 | + +
+ +
Spanish evaluation - + #### Spanish-XX -| | source | target | Bleu ↑ | Ter ↓ | ChrF ↑ | -|:-------------------------|:---------|:---------|:----------|:----------|:----------| -| SalamandraTA-7b-instruct | es | ast | **20.66** | **71.81** | **53.14** | -| SalamandraTA-7b-base | es | ast | 17.65 | 75.78 | 51.05 | -| Transducens/IbRo-nllb | es | ast | 16.79 | 76.36 | 50.89 | +| Model | source | target | Bleu ↑ | Ter ↓ | ChrF ↑ | +|:--------------------------|:-------|:-------|:-----------|:-----------|:-----------| +| SalamandraTA-7b-instruct (v2) | es | ast | **23.56** | **65.25** | **54.27** | +| SalamandraTA-7b-instruct (v1) | | | 20.66 | 71.81 | 53.14 | +| Transducens/IbRo-nllb | | | 16.79 | 76.36 | 50.89 | | | | | | | | -| SalamandraTA-7b-base | es | arn | **51.59** | **35.51** | **73.50** | -| Transducens/IbRo-nllb | es | arn | 50.20 | 36.60 | 73.16 | -| SalamandraTA-7b-instruct | es | arn | 47.37 | 39.29 | 70.65 | +| SalamandraTA-7b-instruct (v2) | es | arn | **53.33** | **34.71** | **74.43** | +| SalamandraTA-7b-instruct (v1) | | | 47.37 | 39.29 | 70.65 | +| Transducens/IbRo-nllb | | | 50.20 | 36.60 | 73.16 | | | | | | | | -| Transducens/IbRo-nllb | es | arg | **59.75** | **28.01** | **78.73** | -| SalamandraTA-7b-base | es | arg | 53.96 | 31.51 | 76.08 | -| SalamandraTA-7b-instruct | es | arg | 44.10 | 39.98 | 71.12 | +| SalamandraTA-7b-instruct (v2) | es | arg | 57.79 | 29.54 | 77.30 | +| SalamandraTA-7b-instruct (v1) | | | 44.10 | 39.98 | 71.12 | +| Transducens/IbRo-nllb | | | **59.75** | **28.01** | **78.73** | + +
@@ -741,80 +764,203 @@ against [Transducens/IbRo-nllb](https://huggingface.co/Transducens/IbRo-nllb) [( #### Catalan-XX -| | source | target | Bleu ↑ | Ter ↓ | ChrF ↑ | -|:-------------------------|:---------|:---------|:----------|:----------|:----------| -| SalamandraTA-7b-instruct | ca | ast | **28.13** | **58.84** | **58.98** | -| SalamandraTA-7b-base | ca | ast | 26.11 | 63.63 | 58.08 | -| Transducens/IbRo-nllb | ca | ast | 24.77 | 61.60 | 57.49 | +| Model | source | target | Bleu ↑ | Ter ↓ | ChrF ↑ | +|:--------------------------|:-------|:-------|:-----------|:-----------|:-----------| +| SalamandraTA-7b-instruct (v2) | ca | ast | **30.41** | **55.92** | **59.29** | +| SalamandraTA-7b-instruct (v1) | | | 28.13 | 58.84 | 58.98 | +| Transducens/IbRo-nllb | | | 24.77 | 61.60 | 57.49 | | | | | | | | -| SalamandraTA-7b-base | ca | arn | **31.76** | **53.71** | **60.71** | -| Transducens/IbRo-nllb | ca | arn | 31.22 | 54.30 | 60.30 | -| SalamandraTA-7b-instruct | ca | arn | 30.89 | 54.70 | 59.78 | +| SalamandraTA-7b-instruct (v2) | ca | arn | **31.66** | 55.10 | 59.95 | +| SalamandraTA-7b-instruct (v1) | | | 30.92 | 54.69 | 59.82 | +| Transducens/IbRo-nllb | | | 31.22 | **54.30** | **60.30** | | | | | | | | -| Transducens/IbRo-nllb | ca | arg | **24.44** | **60.79** | **55.51** | -| SalamandraTA-7b-base | ca | arg | 22.53 | 62.37 | 54.32 | -| SalamandraTA-7b-instruct | ca | arg | 20.96 | 65.64 | 52.41 | +| SalamandraTA-7b-instruct (v2) | ca | arg | **25.12** | **60.31** | 54.81 | +| SalamandraTA-7b-instruct (v1) | | | 20.96 | 65.64 | 52.41 | +| Transducens/IbRo-nllb | | | 24.44 | 60.79 | **55.51** | -### Gender Aware Translation +### Chinese, Korean, Japanese, Hindi, Arabic -Below are the evaluation results for gender aware translation evaluated on the [MT-GenEval](https://github.com/amazon-science/machine-translation-gender-eval?tab=readme-ov-file#mt-geneval) -dataset ([Currey, A. et al.](https://github.com/amazon-science/machine-translation-gender-eval?tab=readme-ov-file#mt-geneval)). -These have been calculated for translation from English into German, Spanish, French, Italian, Portuguese and Russian and are compared -against [MADLAD400-7B-mt](https://huggingface.co/google/madlad400-7b-mt), [TowerInstruct-7B-v0.2](https://huggingface.co/Unbabel/TowerInstruct-7B-v0.2) -and the SalamandraTA-7b-base model. -Evaluation was conducted using [MT-Lens](https://github.com/langtech-bsc/mt-evaluation) and is reported as accuracy computed using the accuracy metric -provided with MT-GenEval.
+English evaluation -| | Source | Target | Masc | Fem | Pair | -|:--|:--|:--|:--|:--|:--| -| MADLAD400-7B | en | de | **0.877** | 0.823 | 0.713 | -| SalamandraTA-7b-base | en | de | 0.857 | 0.770 | 0.660 | -| SalamandraTA-7b-instruct | en | de | 0.863 | **0.867** | **0.740** | -| TowerInstruct-7B-v0.2 | en | de | 0.863 | 0.840 | 0.727 | -| | | | | | | -| MADLAD400-7B | en | es | 0.887 | 0.780 | 0.687 | -| SalamandraTA-7b-base | en | es | **0.890** | 0.733 | 0.643 | -| SalamandraTA-7b-instruct | en | es | 0.860 | **0.837** | **0.710** | -| TowerInstruct-7B-v0.2 | en | es | 0.850 | 0.823 | 0.693 | -| | | | | | | -| MADLAD400-7B | en | fr | 0.873 | 0.777 | 0.663 | -| SalamandraTA-7b-base | en | fr | 0.887 | 0.710 | 0.617 | -| SalamandraTA-7b-instruct | en | fr | **0.900** | 0.813 | **0.730** | -| TowerInstruct-7B-v0.2 | en | fr | 0.880 | **0.823** | 0.717 | -| | | | | | | -| MADLAD400-7B | en | it | 0.907 | 0.663 | 0.597 | -| SalamandraTA-7b-base | en | it | 0.893 | 0.593 | 0.513 | -| SalamandraTA-7b-instruct | en | it | 0.913 | **0.780** | 0.707 | -| TowerInstruct-7B-v0.2 | en | it | **0.947** | 0.747 | **0.713** | -| | | | | | | -| MADLAD400-7B | en | pt | 0.923 | 0.687 | 0.627 | -| SalamandraTA-7b-base | en | pt | 0.923 | 0.650 | 0.597 | -| SalamandraTA-7b-instruct | en | pt | **0.933** | **0.797** | **0.747** | -| TowerInstruct-7B-v0.2 | en | pt | 0.907 | 0.730 | 0.670 | -| | | | | | | -| MADLAD400-7B | en | ru | 0.940 | 0.797 | 0.740 | -| SalamandraTA-7b-base | en | ru | 0.933 | 0.713 | 0.653 | -| SalamandraTA-7b-instruct | en | ru | **0.950** | **0.830** | **0.783** | -| TowerInstruct-7B-v0.2 | en | ru | 0.933 | 0.797 | 0.733 | -| | | | | | | +#### English → Asian languages + +| Model | source | target | BLEU ↑ | ChrF ↑ | COMET ↑ | COMET-kiwi ↑ | BLEURT ↑ | MetricX ↓ | MetricX-QE ↓ | +|:-------------------------------|:-----|:-----|:---------|:---------|:---------|:---------|:---------|:---------|:---------| +| SalamandraTA-7b-instruct (v2) | en | zh | **44.32** | **40.08** | **0.88** | **0.84** | **0.73** | **1.64** | **1.56** | +| MADLAD-400-7B | | | 41.48 | 36.96 | 0.86 | 0.82 | 0.71 | 2.12 | 1.87 | +| NLLB-200-3.3B | | | 27.01 | 25.42 | 0.78 | 0.71 | 0.58 | 5.23 | 7.99 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | en | ko | 30.18 | 36.91 | **0.89** | **0.86** | **0.68** | **1.82** | 1.33 | +| MADLAD-400-7B | | | **30.74** | **37.34** | 0.88 | 0.85 | 0.68 | 1.90 | 1.07 | +| NLLB-200-3.3B | | | 28.64 | 35.18 | 0.88 | 0.86 | 0.67 | 1.89 | **0.80** | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | en | ja | **35.23** | **42.11** | **0.91** | **0.87** | **0.71** | **1.37** | **1.12** | +| MADLAD-400-7B | | | 27.78 | 36.81 | 0.90 | 0.86 | 0.68 | 1.77 | 1.71 | +| NLLB-200-3.3B | | | 20.08 | 30.37 | 0.86 | 0.83 | 0.59 | 2.70 | 2.30 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | en | hi | **34.06** | **58.17** | **0.81** | **0.85** | **0.73** | **2.00** | 1.23 | +| MADLAD-400-7B | | | 31.14 | 55.86 | 0.79 | 0.82 | 0.70 | 2.99 | 2.01 | +| NLLB-200-3.3B | | | 33.19 | 58.04 | 0.81 | 0.84 | 0.72 | 2.07 | **1.03** | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | en | ar | **22.89** | **54.69** | **0.86** | **0.81** | **0.72** | **2.06** | **1.70** | +| MADLAD-400-7B | | | 17.93 | 50.83 | 0.83 | 0.78 | 0.64 | 2.79 | 2.11 | +| NLLB-200-3.3B | | | 19.17 | 51.95 | 0.83 | 0.78 | 0.64 | 2.96 | 1.91 | +| | | | | | | + +#### Asian languages → English + +| Model | source | target | BLEU ↑ | ChrF ↑ | COMET ↑ | COMET-kiwi ↑ | BLEURT ↑ | MetricX ↓ | MetricX-QE ↓ | +|:-------------------------------|:-----|:-----|:---------|:---------|:---------|:---------|:---------|:---------|:---------| +| SalamandraTA-7b-instruct (v2) | zh | en | **32.98** | **60.68** | **0.88** | 0.84 | 0.77 | 1.27 | 1.33 | +| MADLAD-400-7B | | | 30.64 | 59.62 | 0.88 | **0.85** | **0.77** | **1.26** | **1.27** | +| NLLB-200-3.3B | | | 29.49 | 57.63 | 0.86 | 0.84 | 0.74 | 1.70 | 1.60 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ko | en | **34.08** | **61.29** | 0.89 | 0.85 | 0.77 | 1.33 | 1.35 | +| MADLAD-400-7B | | | 33.35 | 60.72 | **0.89** | **0.86** | **0.77** | **1.23** | **1.24** | +| NLLB-200-3.3B | | | 29.38 | 57.72 | 0.87 | 0.85 | 0.74 | 1.66 | 1.57 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ja | en | **31.50** | **59.87** | **0.88** | 0.86 | 0.76 | **1.38** | **1.20** | +| MADLAD-400-7B | | | 29.21 | 58.30 | 0.88 | **0.86** | **0.76** | 1.45 | 1.27 | +| NLLB-200-3.3B | | | 28.16 | 56.85 | 0.87 | 0.85 | 0.74 | 1.76 | 1.45 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | hi | en | **44.95** | **68.29** | **0.90** | 0.85 | **0.78** | **1.31** | **1.42** | +| MADLAD-400-7B | | | 42.62 | 66.98 | 0.90 | **0.86** | 0.78 | 1.35 | 1.49 | +| NLLB-200-3.3B | | | 42.75 | 67.28 | 0.90 | 0.85 | 0.78 | 1.46 | 1.52 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ar | en | **44.89** | 68.32 | 0.88 | 0.81 | 0.78 | 1.37 | 1.66 | +| MADLAD-400-7B | | | 44.44 | **68.50** | **0.88** | **0.83** | **0.79** | **1.33** | **1.33** | +| NLLB-200-3.3B | | | 42.09 | 66.37 | 0.88 | 0.82 | 0.78 | 1.48 | 1.44 | +| | | | | | | + +
+ +
+Spanish evaluation + +#### Spanish → Asian languages + +| Model | source | target | BLEU ↑ | ChrF ↑ | COMET ↑ | COMET-kiwi ↑ | BLEURT ↑ | MetricX ↓ | MetricX-QE ↓ | +|:-------------------------------|:-----|:-----|:---------|:---------|:---------|:---------|:---------|:---------|:---------| +| SalamandraTA-7b-instruct (v2) | es | zh | **37.02** | **33.75** | 0.87 | 0.80 | **0.71** | 1.77 | 1.58 | +| MADLAD-400-7B | | | 35.87 | 32.02 | **0.87** | **0.81** | 0.71 | **1.74** | **1.52** | +| NLLB-200-3.3B | | | 20.81 | 20.44 | 0.77 | 0.69 | 0.56 | 5.04 | 8.90 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | es | ko | **22.42** | **30.27** | **0.87** | **0.82** | **0.66** | **1.81** | 1.32 | +| MADLAD-400-7B | | | 21.66 | 28.95 | 0.86 | 0.82 | 0.66 | 2.14 | 1.90 | +| NLLB-200-3.3B | | | 20.00 | 27.35 | 0.85 | 0.81 | 0.65 | 2.12 | **0.96** | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | es | ja | **27.63** | **35.35** | **0.89** | **0.83** | **0.68** | **1.57** | **1.55** | +| MADLAD-400-7B | | | 18.20 | 28.64 | 0.85 | 0.79 | 0.65 | 3.02 | 4.05 | +| NLLB-200-3.3B | | | 15.65 | 25.43 | 0.85 | 0.79 | 0.57 | 2.76 | 2.02 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | es | hi | **21.22** | **47.37** | **0.76** | 0.79 | **0.68** | **2.24** | 1.40 | +| MADLAD-400-7B | | | 19.42 | 46.48 | 0.75 | **0.80** | 0.68 | 2.28 | 1.63 | +| NLLB-200-3.3B | | | 18.90 | 45.76 | 0.75 | 0.80 | 0.67 | 2.44 | **1.13** | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | es | ar | **16.34** | **48.55** | **0.85** | **0.79** | **0.69** | **1.71** | **1.39** | +| MADLAD-400-7B | | | 5.22 | 37.46 | 0.75 | 0.70 | 0.54 | 5.06 | 5.42 | +| NLLB-200-3.3B | | | 8.46 | 40.74 | 0.78 | 0.73 | 0.55 | 3.67 | 2.37 | +| | | | | | | + +#### Asian languages → Spanish + +| Model | source | target | BLEU ↑ | ChrF ↑ | COMET ↑ | COMET-kiwi ↑ | BLEURT ↑ | MetricX ↓ | MetricX-QE ↓ | +|:-------------------------------|:-----|:-----|:---------|:---------|:---------|:---------|:---------|:---------|:---------| +| SalamandraTA-7b-instruct (v2) | zh | es | **21.59** | 49.80 | 0.85 | 0.82 | 0.71 | 1.15 | 1.62 | +| MADLAD-400-7B | | | 21.27 | **50.00** | **0.86** | **0.84** | **0.72** | **1.00** | **1.49** | +| NLLB-200-3.3B | | | 18.04 | 46.61 | 0.83 | 0.80 | 0.67 | 1.92 | 2.18 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ko | es | **20.92** | **49.41** | **0.85** | 0.84 | **0.71** | **1.17** | **1.59** | +| MADLAD-400-7B | | | 20.58 | 48.69 | 0.85 | **0.84** | 0.71 | 1.25 | 1.82 | +| NLLB-200-3.3B | | | 17.32 | 45.53 | 0.83 | 0.82 | 0.67 | 1.91 | 2.08 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ja | es | **21.00** | **49.09** | **0.85** | **0.85** | **0.71** | **1.20** | **1.46** | +| MADLAD-400-7B | | | 20.05 | 48.13 | 0.85 | 0.84 | 0.70 | 1.32 | 1.55 | +| NLLB-200-3.3B | | | 17.39 | 45.76 | 0.83 | 0.83 | 0.67 | 1.96 | 1.98 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | hi | es | **24.46** | **51.68** | **0.86** | 0.82 | **0.72** | **1.10** | **1.73** | +| MADLAD-400-7B | | | 22.57 | 50.49 | 0.86 | **0.82** | 0.72 | 1.40 | 2.20 | +| NLLB-200-3.3B | | | 20.95 | 48.87 | 0.84 | 0.82 | 0.69 | 1.73 | 2.10 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ar | es | **24.59** | 51.75 | 0.84 | 0.80 | 0.72 | 1.33 | 2.08 | +| MADLAD-400-7B | | | 24.22 | **52.44** | **0.85** | **0.82** | **0.73** | **1.22** | **1.64** | +| NLLB-200-3.3B | | | 21.30 | 50.02 | 0.83 | 0.80 | 0.70 | 1.69 | 1.92 | +| | | | | | | + +
+ +
+Catalan evaluation + +#### Catalan → Asian languages + +| Model | source | target | BLEU ↑ | ChrF ↑ | COMET ↑ | COMET-kiwi ↑ | BLEURT ↑ | MetricX ↓ | MetricX-QE ↓ | +|:-------------------------------|:-----|:-----|:---------|:---------|:---------|:---------|:---------|:---------|:---------| +| SalamandraTA-7b-instruct (v2) | ca | zh | 38.20 | **35.25** | 0.87 | 0.80 | 0.71 | 1.90 | 2.19 | +| MADLAD-400-7B | | | **39.58** | 35.14 | **0.87** | **0.81** | **0.71** | **1.75** | **1.92** | +| NLLB-200-3.3B | | | 25.42 | 23.57 | 0.80 | 0.72 | 0.60 | 4.06 | 6.20 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ca | ko | 26.08 | **33.42** | 0.87 | 0.82 | 0.67 | 1.80 | 1.27 | +| MADLAD-400-7B | | | **27.15** | 33.15 | **0.88** | **0.83** | **0.69** | **1.62** | 1.14 | +| NLLB-200-3.3B | | | 23.06 | 29.71 | 0.85 | 0.81 | 0.65 | 2.24 | **0.99** | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ca | ja | **29.20** | **37.48** | **0.90** | **0.82** | **0.69** | **1.43** | **1.52** | +| MADLAD-400-7B | | | 23.88 | 31.56 | 0.86 | 0.79 | 0.65 | 2.78 | 3.62 | +| NLLB-200-3.3B | | | 16.58 | 26.62 | 0.85 | 0.78 | 0.57 | 2.80 | 2.31 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ca | hi | **26.79** | **51.72** | **0.77** | 0.81 | **0.70** | **2.23** | 1.67 | +| MADLAD-400-7B | | | 23.76 | 50.48 | 0.77 | **0.81** | 0.69 | 2.25 | 1.69 | +| NLLB-200-3.3B | | | 24.11 | 49.92 | 0.75 | 0.80 | 0.68 | 2.58 | **1.58** | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ca | ar | **18.90** | **50.90** | **0.85** | **0.79** | **0.71** | **1.76** | **1.95** | +| MADLAD-400-7B | | | 6.03 | 37.68 | 0.74 | 0.67 | 0.53 | 5.98 | 7.53 | +| NLLB-200-3.3B | | | 11.50 | 43.77 | 0.79 | 0.72 | 0.57 | 3.81 | 3.29 | +| | | | | | | + +#### Asian languages → Catalan + +| Model | source | target | BLEU ↑ | ChrF ↑ | COMET ↑ | COMET-kiwi ↑ | BLEURT ↑ | MetricX ↓ | MetricX-QE ↓ | +|:-------------------------------|:-----|:-----|:---------|:---------|:---------|:---------|:---------|:---------|:---------| +| SalamandraTA-7b-instruct (v2) | zh | ca | **28.35** | **55.25** | 0.86 | 0.80 | 0.72 | **1.41** | 1.68 | +| MADLAD-400-7B | | | 26.45 | 54.19 | **0.86** | **0.82** | **0.72** | 1.47 | **1.58** | +| NLLB-200-3.3B | | | 20.85 | 48.92 | 0.83 | 0.78 | 0.64 | 2.73 | 2.66 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ko | ca | **27.29** | **54.80** | 0.86 | 0.81 | 0.72 | 1.32 | 2.03 | +| MADLAD-400-7B | | | 26.01 | 54.07 | **0.86** | **0.82** | **0.72** | **1.30** | **1.95** | +| NLLB-200-3.3B | | | 21.37 | 49.03 | 0.83 | 0.79 | 0.65 | 2.45 | 2.74 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ja | ca | **26.30** | **54.21** | **0.86** | 0.81 | **0.71** | **1.50** | 1.87 | +| MADLAD-400-7B | | | 23.96 | 51.73 | 0.85 | **0.82** | 0.70 | 1.54 | **1.85** | +| NLLB-200-3.3B | | | 20.64 | 48.63 | 0.83 | 0.80 | 0.65 | 2.47 | 2.42 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | hi | ca | **31.76** | 57.56 | 0.87 | 0.81 | 0.73 | 1.40 | 2.18 | +| MADLAD-400-7B | | | 31.63 | **57.69** | **0.87** | **0.82** | **0.73** | **1.38** | **1.78** | +| NLLB-200-3.3B | | | 26.33 | 53.33 | 0.85 | 0.81 | 0.68 | 2.22 | 2.47 | +| | | | | | | +| SalamandraTA-7b-instruct (v2) | ar | ca | **33.88** | **59.29** | 0.86 | 0.78 | 0.74 | **1.45** | 1.98 | +| MADLAD-400-7B | | | 32.92 | 59.28 | **0.86** | **0.80** | **0.74** | 1.53 | **1.70** | +| NLLB-200-3.3B | | | 27.69 | 55.66 | 0.84 | 0.78 | 0.69 | 2.25 | 2.02 | +| | | | | | |
+ + + + + + ## Ethical Considerations and Limitations Detailed information on the work done to examine the presence of unwanted social and cognitive biases in the base model can be found at [Salamandra-7B model card](https://huggingface.co/BSC-LT/salamandra-7b). -With regard to MT models, the only analysis related to bias which we have conducted is the MT-GenEval evaluation. -No specific analysis has yet been carried out in order to evaluate potential biases or limitations in translation -accuracy across different languages, dialects, or domains. However, we recognize the importance of identifying and addressing any harmful stereotypes, -cultural inaccuracies, or systematic performance discrepancies that may arise in Machine Translation. As such, we plan to continue performing more analyses -as we implement the necessary metrics and methods within our evaluation framework [MT-Lens](https://github.com/langtech-bsc/mt-evaluation). +With regard to MT models, no specific analysis has yet been carried out in order to evaluate potential biases or limitations in translation accuracy across different languages, dialects, or domains. However, we recognize the importance of identifying and addressing any harmful stereotypes, cultural inaccuracies, or systematic performance discrepancies that may arise in Machine Translation. As such, we plan to continue performing more analyses as we implement the necessary metrics and methods within our evaluation framework [MT-Lens](https://github.com/langtech-bsc/mt-evaluation). Note that the model has only undergone preliminary instruction tuning. We urge developers to consider potential limitations and conduct safety testing and tuning tailored to their specific applications. @@ -830,27 +976,19 @@ For further information, please send an email to . Copyright(c) 2025 by Language Technologies Unit, Barcelona Supercomputing Center. ### Funding -This work has been promoted and financed by the Government of Catalonia through the [Aina Project](https://projecteaina.cat/). -This work is funded by the _Ministerio para la Transformación Digital y de la Función Pública_ - Funded by EU – NextGenerationEU -within the framework of [ILENIA Project](https://proyectoilenia.es/) with reference 2022/TL22/00215337. - -### Acknowledgements - -The success of this project has been made possible thanks to the invaluable contributions of our partners in the [ILENIA Project](https://proyectoilenia.es/): -[HiTZ](http://hitz.ehu.eus/es), and [CiTIUS](https://citius.gal/es/). -Their efforts have been instrumental in advancing our work, and we sincerely appreciate their help and support. +This work is funded by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project Desarrollo Modelos ALIA. +This work has been promoted and financed by the Government of Catalonia through the [Aina Project](https://projecteaina.cat/). +### Acknowledgements -### Disclaimer ### Disclaimer Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, -they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, -including those governing the use of Artificial Intelligence. +they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence. The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use. @@ -859,17 +997,3 @@ The Barcelona Supercomputing Center, as the owner and creator of the model, shal ### Citation - -If you find our model useful, we would appreciate if you could cite our work as follows: - -``` -@misc{gilabert2025salamandrasalamandratabscsubmission, - title={From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Task}, - author={Javier Garcia Gilabert and Xixian Liao and Severino Da Dalt and Ella Bohman and Audrey Mash and Francesca De Luca Fornaciari and Irene Baucells and Joan Llop and Miguel Claramunt Argote and Carlos Escolano and Maite Melero}, - year={2025}, - eprint={2508.12774}, - archivePrefix={arXiv}, - primaryClass={cs.CL}, - url={https://arxiv.org/abs/2508.12774}, -} -```